As the ServiceNow platform is getting more and more popular, the size of ServiceNow databases is increasing over time. Individual tables are becoming so large that the initial loads performed by SnowMirror are sometimes taking very long. That’s why we have spent the last six months re-designing the SnowMirror replication core. We are proud to announce SnowMirror 4.0 which significantly improves the replication performance.
Apply for the SnowMirror Beta Program. Please fill in the form below to participate! We are constantly developing new exciting features for SnowMirror. To see if we are going in the right direction, we really need to get real feedback from you!
Beta Program Application
Performance Improvement
ServiceNow does not provide a way how to access its data in bulk, so SnowMirror has to use the out-of-the-box API which is designed for a transactional access only. That is the reason why the performance bottleneck in the whole replication architecture is the ServiceNow instance itself and not the other parts such as the SnowMirror agent or the mirror database.
The original SnowMirror till the 3.9.x series connects to the ServiceNow instance using 3 concurrent HTTP connections. This is a documented approach, to create and maintain a session with ServiceNow and reuse it for connections to the instance. It completely relies on the ServiceNow infrastructure and routing of the requests to individual instance nodes is out of SnowMirror’s control. That’s why the default was 3 connections in parallel because typically all the requests were routed to the same node, so we wanted to use maximum 3 out of 4 integration semaphores, not to overload a single node.
Another important design feature was that a single synchronization was running in a single thread and using only a single HTTP connection. So, in order to utilize the whole available throughput there had to be at least 3 synchronizations running in parallel.
The SnowMirror 4.0 replication core is completely redesigned. It uses a lower-level approach of connecting to ServiceNow, it better utilizes the ServiceNow cluster and the algorithms are fully parallel. All of this results in a significant performance boost.
The major difference is that SnowMirror is able to identify individual ServiceNow nodes and maintain two sessions with each node separately. This increases the number of replication threads, especially for bigger ServiceNow instances. Most of the instances have only two nodes, but the bigger the instance the more nodes it has.
The second leap forward is the fact that even a single synchronization can utilize the whole thread pool (i.e. all connections) which makes a single-running table several times faster than before. Of course, the general throughput improvement (i.e. if multiple synchronizations are running) is tightly related to the number of nodes. But still, even if your instance has only two nodes then the overall data throughput is 4/3 times higher and an individual table replicates up to 4 times faster.
Example
The SnowMirror 3.x series used only one thread per synchronization and the max number of threads was 3. SnowMirror 4.x can utilize all threads for a single synchronization and the number of threads (i.e. sessions, concurrent connections) increases significantly for bigger instances making the whole throughput a lot higher.
Here are the performance improvements for sample ServiceNow instances (using the default of 2 sessions per node).
- 2 nodes (default) – single table 4 times faster, throughput 1,33 times higher
- 4 nodes – single table 8 times faster, throughput 2,67 times higher
- 8 nodes – single table 16 times faster, throughput 5,33 times higher
- 12 nodes – single table 24 times faster, throughput 8 times higher
