There is an interesting discussion SnowMirror vs. ODBC driver on the ServiceNow Community. This is a topic touching the original motivation for creating SnowMirror. We have been developing SnowMirror since 2012 and the need for such a tool is still there, the ServiceNow technology has simply not changed. If you want to access ServiceNow data then there is either SOAP or REST API. Unlike other cloud platforms, there is no bulk API to extract high data volumes. The REST and SOAP APIs are designed as transactional APIs to work with a single record or smaller record lists. There is also no direct access to the underlying ServiceNow database. Neither customers nor consultants are able to query the database directly.
That’s why the ServiceNow ODBC driver is not a proper database driver as the name might suggest. It is just a wrapper around the SOAP API sending web service calls over HTTP for every SQL query. This results in limited features because you cannot send complicated database queries over simple SOAP API. Also, the queries are sent sequentially, using one HTTP connection and one thread. By contrast, SnowMirror is designed as a data replication tool with high data throughput in mind. It uses the same API as the ODBC driver but the key thing is to use the API and the whole ServiceNow instance in the most efficient way. Here are the most important areas making SnowMirror faster than other solutions:
- Incremental load – SnowMirror downloads just the deltas; only the new and updated records and it removes the deleted items. This enables it to work even with the largest tables.
- Smart pagination. SnowMirror uses special algorithms for handling batches. It downloads a batch of primary keys (sys_ids) first and then it downloads smaller chunks of actual data.
- Concurrency. SnowMirror is able to download a single table in multiple threads which are increasing performance significantly.
- Instance utilization – and this is the most important point for bigger instances. SnowMirror is able to maintain sessions with individual ServiceNow nodes which boosts performance and data throughput dramatically.
The last point is making the biggest difference when comparing just the data throughput. Let’s explain it by the following picture.
SnowMirror is able to identify individual ServiceNow nodes and maintain two sessions (configurable) with each node separately. This increases the number of replication threads, especially for bigger ServiceNow instances. Most of the instances have only two nodes, but the bigger the instance the more nodes it has. There are ServiceNow customers with 16 or even more nodes. Then a single table synchronization can utilize all of the sessions/connection to download data concurrently. This is a major difference comparing to an ODBC driver. Imagine a bigger ServiceNow instance with 8 nodes. In this case, SnowMirror is 16 times faster compared to ODBC. And this is taking into account only a theoretical data throughput and not the effects of smart pagination and other features.
