The SnowMirror team proudly announces the general availability of SnowMirror 3.0. We started developing SnowMirror 3 in cooperation with ServiceNow database experts several months ago. The cooperation resulted in a completely re-worked replication algorithm together with several new features improving SnowMirror’s performance and reliability. We do recommend upgrading the test SnowMirror installation first and then, if successful, upgrading the production mirrors afterward. By introducing indexes on the ServiceNow side the performance improves even more.

What Is New?
The replication algorithm no longer uses the sys_created_on column but only the sys_updated_on column. ServiceNow confirms that using this single column is better for overall performance. The disadvantage is that SnowMirror is not able to count the number of new records before the synchronization begins. It can only count created and updated records together. The pagination algorithm has been improved as well to be more reliable especially for large tables and tables under a load during the synchronization run. Some of the users were experiencing data losses and inconsistencies. These issues should be resolved by this improvement.

More features and fixes:
Synchronization validation – including the check of existing indexes
Consistency check – at the end of each run
Encoded queries and views replication improved
Synchronization filter and sorting – UI improvement
ServiceNow server time usage – requires Aggregate Web Service plugin
Clean & Synchronize using temporary tables
SnowMirror Monitor – Fuji-compatible ServiceNow application, the old update set is deprecated

More detailed information in the Release Notes and in the post covering the beta-testing.

IMPORTANT – Indexes
By using the new algorithm ServiceNow strongly recommends creating indexes on the ServiceNow side for larger tables to avoid full-table scans during the incremental loads. So please, create indexes on the sys_updated_on column for tables larger than 100k records. Either raise a ticket in the HI instance or, starting with the Fuji release, you can create an index on your own. Please read this article on how to introduce an index in Fuji.

ServiceNow announced general availability of their latest release Geneva today. SnowMirror is always keeping pace with ServiceNow releases and patches, so welcome SnowMirror 3.1, Geneva-compatible version which was released the same day as ServiceNow Geneva. This new version contains several new features and improvements. Mainly the Display Values support has been re-worked and there was a large optimization in the meta-data handling.

Geneva Support
SnowMirror 3.1 fully supports the Geneva release Patch 1. The data type mapping has been adjusted, all connection testers and validation functions improved and the whole application has been tested against a number of Geneva instances. If you find any issues replicating the data from Geneva, please let us know at service@www.snow-mirror.com.

Display Values Improvements
In the older SnowMirror releases the data types of all the display value columns were unified to strings with 1000 characters. Simply, SnowMirror was not analyzing all the meta-data in the System Dictionary to figure out which field is going to be a display value for a certain record. Starting with SnowMirror 3.1 this analysis is happening so all the display value columns get correct data types. These new rules apply only to synchronizations created in the new version. Older tables with the 1000-char strings are still supported for data downloads.

Please note that this new algorithm requires access to one more table: sys_dictionary_override. Existing users shall adjust privileges of their SnowMirror ServiceNow users.

Meta-Data Handling
SnowMirror queries ServiceNow dictionary and other tables to get the info about tables, columns, views, etc. Most of the queries were being cached so the instance was not queried over and over again with the same requests. However, there was still some space to make it better. SnowMirror team completely re-designed the meta-data handling so it is sending significantly fewer requests for ServiceNow meta-data. Both during configuration and data downloads. The most visible effect is that the bulk creation process executes more than 10 times faster in the new version.

The new release SnowMirror 3.2 brings several improvements and fixes as requested by our growing user base. The most important feature is the Consistency Check which compares the number of records on both sides and raises a warning if necessary. Another feature is the log retention job improving SnowMirror operations in long-term. The release contains many other small improvements and bug fixes as well.

Consistency Check
If enabled, SnowMirror compares the number of records in ServiceNow with the number of records in the mirror DB after each synchronization run. If there is an inconsistency in the numbers then the run finishes with a new warning status. New notification types are available so SnowMirror is able to send emails on warnings.

This feature is enabled by default when installing SnowMirror 3.2 from scratch. Users upgrading to this new version would have to enable the consistency check manually in Settings -> General Settings.

If any synchronization finishes in a warning status then the troubleshooting comes into play. The first step would be to check the activity log. There is a new section at the end of each run where SnowMirror logs messages about the consistency check. It is easy to figure out if there are records missing in the mirror DB or the other way round. See the log snapshot:

To fix the consistency issue there is the good old Clean & Synchronize command to perform a full load. Which might take some time obviously and which might cause an outage in reporting. Since 3.2 there is a new feature called Differential Synchronization. This type of synchronization compares the sys_ids on both sides and downloads the missing records and removes the orphans. The benefit is that it is significantly faster than the full load and there is no outage of the mirror table.

Please note that these features are available in the Enterprise Edition only.

Activity Log Retention
If you run SnowMirror for a long time or if the synchronization intervals are short then the amount of activity logs grows quickly. The new Activity Log Retention feature purges older logs optionally including the records in the configuration database.

This feature is enabled for all new installations. By default all the replication history including the activity logs is removed after 30 days. Upgrading users have to enable this feature manually again.

Other Changes

Temporary Tables for Full Loads
There is a new option which allows to perform all full loads using temporary tables. So the load is inserting a data into the temp table and the original table is available for reporting. After the load is successful the tables are swapped and the original table then removed. Enable this feature in Settings -> General Settings.

Improved Validate Button
The Validate button on the synchronization edit screen has been improved a lot. It performs more checks and the messages are more detailed. We strongly recommend running the validation before creating every single new synchronization

No More Edition-Specific Releases
SnowMirror is now distributed as a single package regardless on the edition. All the features available in the editions are driven by the license only.