SnowMirror Release Notes

Updated dependencies (Apache Tomcat 10.1.33, jQuery 3.7.1)

Fixed: Synchronizations could have deleted data that should not have been deleted. It also caused slowness on Snowflake. It happened only if:

Fixed: Failed to synchronize attachments that are not compressed in ServiceNow.

Fixed: If a clean synchronization to a temporary table failed while counting records in a database table, any subsequent synchronization attempts would fail with "snowmirror_backup_n table already exists" error.

Fixed: Dashboard page did not show up because of "error converting expression to data type int" error.

Updated dependencies (Java 22.0.2+9, Spring 6.1.14)

Fixed: failed to synchronize attachments due to new line in the folder name.

Fixed: SnowMirror 5.9.0 failed to upgrade on Windows.

Fixed: Synchronizations with option "Table Rotation" set to "Replicate and merge shards" and a delete strategy set to "Audit" may have failed to delete rotated data.

Fixed: Failed synchronization did not log the root cause of an error. That made it harder to debug the issue.

Fixed: Azure Synapse did not log problematic columns of data truncation error.

Fixed: "Validate" button failed on Settings → Performance page.

Fixed: "mirror.jdbc.c3p0.properties" property is configurable per database.

Fixed: Copy from tmp table to target table failed on Synapse and SQL Server when batching was disabled.

Supporting a new ServiceNow release - Xanadu. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Java upgrade - Upgraded JRE to the latest version 22.0.2+9. 32-bit systems are no longer supported.

Fixed: Full History page did not load.

Fixed: SnowMirror Monitor did not work with Performance API. Update sets of SnowMirror Monitor and Performance API did not change.

Fixed: Synchronization to Azure Synapse could have failed with incorrect column name.

Decreased the number of requests sent to Azure Storage while synchronizing to Azure Synapse.

Fixed: Test Connection button on "Settings → Database" page did not work with AWS Aurora MySQL when it was defined as a second database.

Fixed "Only Display Value" option:

Fixed: SnowMirror regularly deletes old activity logs from configuration database. We have improved performance of this process to prevent failures.

Added ability to recover from failover on Aurora MySQL.

Fixed: After upgrade from previous version to 5.8.0 or 5.8.1, SnowMirror failed to persist ServiceNow indexes into SnowMirror’s configuration database.

Fixed: Performance API

Fixed: Synchronization to CSV files on Azure Storage failed with "java.lang.NumberFormatException: Cannot parse null string".

Debug option: Added an option to skip failing SOAP GetRecords operation using "snowMirror.debug.skipFailingGetRecordsSyncIds" property.

Updated dependencies to their latest versions.

Fixed: updated Release Notes of previous release.

Tags - added tags configuration option that allows you to group synchronizations according to your preferences for greater clarity.

Performance API - With this release, we announce general availability of Performance API.

Fixed: Consistency Check was slow when "Soft-delete" option was enabled. To fix this issue, we added an index on "mirror_deleted_on" column.

Improved "Test Connection" button on "Settings → ServiceNow" and "Settings → Database" pages.

Fixed: Minor user interface glitches on Backup and Backup Attachment pages.

Updated dependencies to their latest versions.

Fixed: Shutdown script did not wait for SnowMirror to stop.

Fixed: Performance API failed to synchronize "sys_template" table. To make this change work, please install the latest update set into your ServiceNow instance.

Improved detailed logging (snowMirror.debug.deleteNonMatching).

Fixed: SnowMirror did not stop. From now one, it waits 60 seconds to gracefully close all connections, treads, etc. If they don’t close in 60 seconds, SnowMirror shuts them down forcefully.

Fixed: Performance API failed to synchronize tables with names starting with "x_awns"

Fixed: Improved validation of import file on Settings → Import/Export page.

Fixed: Improved input parameter validation to prevent minor security glitches (parameter pollution, open redirection).

Fixed: Display value columns - fixed null pointer exception and improved logging (snowMirror.debug.displayValueColumns).

Fixed: C3P0 connection pool logged "apparent deadlock" to server log. SnowMirror tests idle database connections regularly. When the test
failed to establish a connection, it hung for 10 minutes. C3P0 mistakenly considered that as a deadlock. We have fixed the issue by defining a timeout so that the test always finishes fast.

Fixed: None delete strategy could have deleted records that did not match the filter (Encoded Query) anymore.

Fixed: Failed to authenticate to SQL Server using Azure AD. ("java.lang.NoSuchMethodError: 'com.microsoft.aad.msal4j.AbstractClientApplicationBase$Builder").

Fixed: Failed to store Advanced Settings page when using cloud file storage (Azure Storage, S3, Google Storage).

Fixed: Upgrade failed on Azure Synapse, Azure Storage, S3 and Redshift due to an empty 'authenticationChoice' property.

Fixed: Successful upgrade could have taken longer time to finish than necessary.

Fixed: "Quote Identifiers" field not loaded properly on Settings → Database page.

Updated dependencies to their latest versions, including Java (17.0.11+9) and Apache Tomcat (10.1.20).

Fixed: Windows installer failed to update Windows service ("access denied" error when calling "sc" command).

Windows installer: improved logging.

Fixed: Performance API failed to synchronize a database view.

Fixed: Initial Wizard configuration failed with Oracle as a configuration database.

Fixed: Validate button failed with "Unknown Mirror Database ID provided." error.

Fixed: Failed to upgrade to 5.7.0 on PostgreSQL.

Fixed: failed to deselect days from an incremental weekly scheduler.

Fixed: Auto Schema Update failed to delete a column from configuration DB when index replication was disabled but was enabled in the past.

Fixed: Validation of "Mirror Table" did not allow multiple synchronizations with the same table name prefix.

Supporting a new ServiceNow release - Washington. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Added an option to configure multiple connections to the mirror databases simultaneously and assign synchronizations to them.
Only available if your license supports it.

New authentication options for Azure - Added an option to authenticate into the Azure Storage and Azure Synapse.
If SnowMirror is running in an Azure Container, Instance Managed Identity can be used for authentication. Otherwise, you can use Access Key or SAS Token.

Added new columns into activityLog table in configuration database to provide more information about consistency check and scheduler.

Updated dependencies: including Microsoft’s JDBC driver.

Data Truncation - in some cases, truncation failed ("Multiple entries with same key" error).

Fixed: "Count records" button did not work on a new synchronization.

Fixed: do not wrap date and duration fields on the list of synchronizations.

Fixed: Failed to upgrade internal Tomcat webserver on Windows installations.

Fixed: When Performance API was much faster than mirror database, data piled up in SnowMirror and consumed too much memory.

Fixed: Auto Schema Update tried to add a column which already existed.

Fixed: Attachments with the same names (where the only difference was the case of letter) were not synchronized properly. On Windows, only one of such attachments was synchronized.

Fixed: "Test Connection" button did not work for Azure Synapse.

Fixed: "Remember Me" did not work.

Fixed: Improved performance of small synchronizations (under 10,000 records) of REST and Performance API by splitting it into multiple parallel threads.

Performance API: added an option to count records using GlideRecord because the default option does not skip business rules. Please, don’t to install the latest update set into your ServiceNow instance.

Fixed Data Archiving: In some cases, we failed to check for records restored using Data Archiving plugin. The check also took too long to finish.

Fixed: "Validate" button failed when using Performance API.

Fixed: Minor user interface glitches

Fixed: Failed to load the latest SnowMirror version in the footer.

Fixed: SSO setup failed with ConcurrentModificationException.

Fixed Data Archiving: When "snowMirror.replication.dataArchiving.algorithm" was set to "ar", a synchronization could have failed on "Invalid table ar_tableName" error.

Fixed: SnowMirror Monitor - a warning during installation.

Performance API - SnowMirror Performance API provides a better performance compared to out-of-the-box REST or SOAP APIs. Performance API is a global-scoped application in ServiceNow. It achieves better download times by turning off ACL checks, turning off business rules and streaming the results. It is primarily targeted to the largest customers with the largest amounts of data. See Performance API documentation for details on how to get started.

Fixed: SnowMirror failed to upload data from "Edit Settings" page to SnowMirror Monitor. Error message: "Cannot read property "type" from null". Affected versions: 5.5.0 - 5.5.8.

Fixed: From now on, SnowMirror limits the size of activity logs being uploaded to SnowMirror Monitor in ServiceNow. There are 2 reasons for that: performance and reliability. Because of that, we have decided to limit the size of activity log to upload to 2 MB. Larger logs will be truncated. The behavior is configurable via "serviceNow.sncPlugin.activityLogsUpload.maxSizeBytes" property. The truncation can be turned off by setting the property to 0.

Fixed: SnowMirror Monitor - a large activity log may have not been processed and attached to its synchronization run.

Fixed: Some of the Settings pages did not contain autocomplete="off" parameter to prevent autocomplete of sensitive information.

Fixed: Migration from H2 database failed. If you use SnowMirror 5.5.0 - 5.5.6, please upgrade your SnowMirror to >= 5.5.7, and then you can migrate from H2.

Updated Apache Tomcat to the latest version 10.1.17.

Schedulers: Added an option to define time zone of schedulers.

Schedulers: Added a debugging option that allows you to see when exactly the sync is going to be executed. Open "Edit Scheduler" page and append "?showScheduleFrom=2024-03-30" parameter to the URL. SnowMirror will print next 100 execution time from March 30, 2024. You can choose your own date.

Fixed: Silent Windows installer ignored "ConfigDbSsl" option.

Upgraded JRE to the latest version 17.0.9+9 and Apache Tomcat 10.1.16.

Fixed: failed to upgrade SnowMirror on PostgreSQL configuration database.

Azure Synapse Table Index: added an option to choose a table index type on a synchronization level. Go to Detail of your synchronization → arrow on Edit Settings → Advanced Settings → See "Azure Synapse" section and a field "Index Type".
Do not change the index type on existing synchronizations. Instead, create a new synchronization and change the type before you run it for the first time.
By default, SnowMirror uses clustered columnstore index. This feature allows you to switch to "heap" or "clustered index" so that you can utilize "varchar(max)" data type. Also, heap may perform better on smaller tables.

Azure Synapse: added a new configuration option (mirror.synapse.useHeapTemporaryTables). When storing data to Synapse, SnowMirror first stores data to a temporary table. Then, it merges the changes to the target table. This new option allows you to create the temporary table with a HEAP table index. That could improve performance of merge operation (no guarantees given).

Updated dependencies to their latest versions.

Fixed: Do not create indexes in a temporary table during incremental synchronization on Azure Synapse and Amazon Redshift. This improves performance of the incremental synchronization.

Fixed: Schedule on Detail page was shown in an incorrect time zone.

Updated dependencies to their latest versions.

Added optional logging of data type conversion. It can be enabled using "snowMirror.activityLog.debugTypeConversion" advanced property.

Improved re-try mechanism for intermittent authentication issues.

Updated dependencies to their latest versions.

Supporting a new ServiceNow release - Vancouver. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

User interface look and feel: Changed look and feel of user interface so that SnowMirror looks better and provides a moder user experience.

Multiple schedulers: Added an option to define multiple different schedulers on a single synchronization. Your current schedulers are not affected by this change and the keep working as they did in previous versions.

IAM Roles support - Added an option to authenticate into the Amazon S3 and Redshift database using IAM Roles if SnowMirror is running in the Amazon EC2 cloud.

Improved security:

Updated dependencies to their latest versions.

Fixed: Count the number of records during Auto Schema Update only when necessary.

Upgraded JRE to the latest version 17.0.8.1+1

Fixed: Differential delete strategy consumed more memory than necessary. In case a target table was missing millions of records, it could have led to out of memory issue (depending on how much memory your SM uses, how many such differential syncs were running at the same time, how many records were missing in the target table).

Fixed a memory issue: SnowMirror held data in memory longer than necessary. It always cleaned up the data in the end. However, the application could have run out memory sooner than the data was cleaned up.

Upgraded JRE to the latest version 17.0.8+7

Fixed: Failed to soft-delete records from Snowflake.

OAuth2 authentication:

Improved logging of incoming values (snowMirror.debug.logUnmarshallingColumns)

Fixed: Failed to authenticate against ServiceNow using OAuth2 due to a race condition.

Data Archiving Plugin - Added an option to check for archived/restored records in "ar_" table. By default, we look for archived/restored records in "sys_archive_log" table. In case the table is large and times-out, we can use "ar_" table instead (snowMirror.replication.dataArchiving.algorithm). On the "ar_" table, you should have DB indexes in ServiceNow on "sys_id", "sys_archived" and "sys_archive_restored" columns.

Fixed: Failed to authenticate against ServiceNow using OAuth.

Fixed: Mirror DB metrics not captured on Settings → Stats page.

Updated dependencies to their latest versions (Apache Tomcat 10.1.10, Spring 6.0.10, Oracle driver 21.10.0.0)

Fast Difference delete strategy could have deleted a record incorrectly.

Difference and Fast Difference delete strategy could have missed a deleted record.

Fixed: Filling a default value into a newly added column failed.

Improved logging. Added a new advanced property "snowMirror.activityLog.debugDeleteStrategy" which allows us to debug deleting strategy.

Fixed: Incorrectly called API when there was no attachment to download.

Upgraded JRE to the latest version 17.0.7+7 and Apache Tomcat 10.1.8.

Fixed: Windows Installer - Oracle "Service Name" option was not visible.

Fixed: Synchronizations failed with ORA-01461 error on Oracle database.

Added a new optional property "snowMirror.mirrorTable.logAllRecordsIfOneByOneSucceeds" which helps us to debug problematic records.

Improved re-try mechanism - when an OAuth authenticated connection to ServiceNow fails, SnowMirror clears a session and OAuth token so that it can retrieve new ones.

Fixed: After an action on "Synchronizations" page was executed, a filter got cleared.

Fixed: "Test Connection" button on "Settings → ServiceNow" page could have failed with REST API.

Fixed: "Test Connection" button in Windows installer failed when data contained a quote (") character.

Fixed: A synchronization failed when:

Supporting a new ServiceNow release - Utah. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Cloud Storage Support - by default, CSV files are compressed with GZIP. We have added an option (mirror.cloudStorage.local.useCompression) not to compress the files.

Optional detailed logs - Every synchronization can be configured to log detailed information into activity logs.

Window Installer: Improved stopping of a service during upgrade or uninstall.

Fixed: import of "advanced settings" ignored some configuration options.

Fixed: SnowMirror failed to start on Windows.

Fixed: Failed to truncate data in columns with a custom name defined on Edit Mapping page.

Fixed: The "Find Solution" button was missing on warnings on Activity page.

Fixed: Windows installer failed to import certificates during upgrade process.

Fixed: Failed test a database connection from Windows installer.

Fixed: Failed to install SnowMirror on Windows 10.

Amazon Redshift support - added support for Amazon Redshift.

Cloud Storage Support - added an option to store data into CSV files into cloud storage.

Remember Me - Optionally, "Remember Me" can be disabled.

Upgraded libraries to their latest versions:

How to upgrade:

Fixed: Failed to truncate data in columns with a custom name defined on Edit Mapping page.

Fixed: UnknownFormatConversionException when synchronizing a view.

In Tokyo ServiceNow release, data type of "cmdb_ci.ip_address" column has changed from "string" to "ip_addr". In case you use "Auto Schema Update" option, SnowMirror found the change and triggered a clean synchronization. However, a clean synchronization is not necessary. Therefore, we modified SnowMirror so that it does not trigger a clean synchronization because of this column.

Updated dependencies to their latest versions. (Apache Tomcat: 9.0.71)

Fixed: Differential synchronization with "Check Updates" option enabled did not work properly when a value of "sys_updated_on" column was null.

Fixed: Activity Log Retention - Daily cleanup uses smaller database transactions. Therefore, it less likely to block other transactions.

Updated dependencies to the latest versions.

Fixed: Failed to insert a row on Azure Synapse when a display value column had a name changed on Edit Mapping page.

Fixed: Differential synchronization failed when ServiceNow returned an empty SysId column.

Fixed: Synchronization failed with "Type:-155 is not a valid Types.java value." error. Only users with SQL Server with MS driver and "Timezone Data Types"=Yes were affected.

Fixed: "Synchronize From" could have started from an incorrect date.

Fixed: PostgreSQL fails with "ERROR: invalid byte sequence for encoding "UTF8": 0x00" error.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version.

Fixed: An incorrect batch size may have been used.

Fixed: Full Load Scheduler with "Synchronize From" offset did not offset the synchronization.

Fixed: PostgreSQL failed with "ERROR: index "schema_version_vr_idx" does not exist" error.

Fixed: Retry request in case it ends up with 401 error.

Added a new Advanced property "serviceNow.httpMessageSender.preemptiveAuth.impl" to control how pre-emptive basic authentication works.

Java upgrade - Upgraded JRE to the latest version 11.0.17+8.

Fixed: A warning about missing columns reported an incorrect number of affected rows. When the warning showed up, there was at least one row with a missing column. However, the number in the log was incorrect.

Fixed: "Settings"→"Database"→"Test Connection" button ignored "mirror.jdbc.encryption" settings when set to "SSL_WITH_CERT_VERIFICATION". Only Test Connection button was affected.

Added a new advanced property which allows us to exclude records from certain scopes (sys_scope table).

If you don’t have enough permissions to read all records, ServiceNow "getKeys" API method may return the SysId of such records. However, "getRecords" API method may not have to return such records. From this release, SnowMirror ignores such records and instead of failing, it shows a warning about missing records.

Added an option to adjust the number of records to download in a single batch per synchronization. That is useful if the table contains large amounts of data (sys_email, sys_attachment_doc) and times-out. This option allows you to decrease the batch size without affecting other synchronizations.

Added support for AWS Aurora MySQL. Customers have to download Aurora database driver "jar" file themselves. Customers are free to use the driver, but we cannot distribute it with SnowMirror due to licensing reasons.

Added a new advanced property "snowmirror.mail.javaMailProperties". It allows you to provide additional SMTP server configuration using properties from Java Mail library.

Exposed Export/Import via REST API.

Updated dependencies to their latest versions. (Apache Tomcat: 9.0.68, Java: 11.0.16.1+1)

Supporting a new ServiceNow release - Tokyo. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Fixed: Synchronization failed with NullPointerException.

Optionally, SnowMirror can keep columns deleted in ServiceNow in your database.

Fixed: Failed to synchronize "Other Date" ServiceNow data type into "date" column on MySQL. Affected versions: 4.13.0-4.13.14

Fixed: Failed to synchronize data into "date" column on MySQL. Affected versions: 4.13.0-4.13.13

Java upgrade - Upgraded JRE to the latest version 11.0.16.1+1.

Updated Microsoft’s database driver to the latest version (11.2.0) because it fixes a memory leak. In previous versions, SnowMirror could have consumed too much memory. Only SQL Server users who use Microsoft’s driver were affected. Please note that most customers still use an alternative JTDS driver and those are not affected.

Azure Synapse: Added an alternative option how to authenticate from "COPY INTO" command to Azure Storage. By default, SnowMirror authenticates using Storage Account Key. From now on, you can authenticate using "Managed Identity". That is useful if your Synapse communicates with Storage Account using a private endpoint.

Fixed: Failed to update data in Azure Synapse due to a changes in "merge" command on Azure-side.

Fixed: "Discover Nodes" button on Settings → Performance page did not work properly.

Fixed: A change of "snowMirror.auth.sso.saml.role.fallbackRoles" property does not require a restart anymore.

Updated dependencies to their latest versions. (Apache Tomcat: 9.0.65, Java: 11.0.16+8)

Fixed: A synchronization could have been executed twice when two schedulers were defined and SnowMirror was running in an Active-Active High Availability Cluster.

Fixed: SQL Server < 2019 did not report problematic columns in case of data truncation error.

Fixed: Failed to upgrade SnowMirror and had to manually patch "schema_version" table in configuration database.

Fixed: SnowMirror tried to connect to MySQL mirror DB using encrypted connection despite "Settings → Database → Use SSL" being unchecked.

Updated dependencies to their latest versions. (Apache Tomcat: 9.0.64, Spring: 5.3.21, Log4j: 2.18.0)

Fixed: Deadlock errors on SQL Server configuration database.

Fixed: SnowMirror failed to configure SSO authentication due to "NoSuchFieldError: xmss_SHA256ph" error.

Fixed: SnowMirror was not able to recover REST API request when OAuth2 token was refused by ServiceNow (HTTP code 401).

Fixed: SnowMirror upgrade could have failed due to "No enum constant org.flywaydb.core.api.MigrationType.INIT" error.

Fixed: H2 Migration Tool failed to migrate into a different schema than the default one.

Fixed: H2 Migration Tool failed to start the migration process.

MySQL connection - When "config.jdbc.encryption" and "mirror.jdbc.encryption" options are set to "IGNORE", connections to MySQL are not encrypted. If you want to use encrypted connections:

PostgreSQL database is now available in Lite license.

Fixed: SnowMirror has updated its internal table "dataStorage" in configuration DB way too much. Also, the size of the data has decreased.

Fixed: An internal table "dataStorage" table may have not been cleaned up properly and grow too much. Therefore, upgrade to the latest SnowMirror version is generally recommended.

Fixed: Failed to change a data type of some columns on Edit Mappings page.

Fixed: Failed to delete a synchronization with rotation schedule.

Fixed: SnowMirror Monitor failed to upload data to ServiceNow.

Fixed: Data Archiving Plugin - synchronization of "ar_" table could have missed restored record.

Fixed: SnowMirror tried to connect to MySQL using encrypted connection despite "Require SSL" option being disabled.

Fixed: Synchronization failed with a "FromAddress must not be empty" error.

Fixed: With a certain configuration, SnowMirror failed to start on Oracle configuration database.

MySQL Database Driver - We have replaced Oracle’s MySQL driver with LGPL-licensed MariaDB Connector/J. The main reason for a switch are licensing terms of the drivers.

SQL Server Database Driver - We have added an optional support for Microsoft’s JDBC driver. We have been using open-source JTDS driver. However, JTDS driver is not actively developed and maintained anymore.

Fixed: Differential synchronization failed when "Diff Check Updated" option was enabled.

Fixed: "Soft Delete" failed with Snowflake database.

Fixed: PostgreSQL & Edit Mappings page - failed to change a data type to "json" and "jsonb".

PostgreSQL: Optionally, you can store ServiceNow "JSON" data type to "json" or "jsonb" in database. You can enable json data types using "snowMirror.mirrorTable.storeJsonAsJson" or "snowMirror.mirrorTable.storeJsonAsJsonb" advanced properties.

Fixed: Failed to filter synchronizations by "Include Inherited Columns" field.

Fixed: When multiple SnowMirror installations run on the same address with a different port, browsers allowed you to be logged in only in one of the installation at the same time.

Fixed: Improved performance of a deletion of synchronizations.

Fixed: When Soft Delete was enabled, "mirror_deleted_on" column could have been updated even if it was already filled in from previous synchronization.

Java upgrade - Upgraded JRE to the latest version 11.0.15+10.

Fixed: Failed to authenticate against SnowMirror’s REST API when SSO was configured.

Fixed: Synchronization failed when "Reference Fields" option was set to "Display Values" and column’s Dictionary record had an empty "sys_created_on" field.

Fixed: Failed to synchronize a column of "document_id" type when "Reference Fields" option was set to "Display Values".

Fixed: Upgraded spring-core library to the latest version which fixes CVE-2010-1622 issue.

Fixed: CVE-2010-1622 in spring-core dependency.

Updated dependencies to their latest versions.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version 9.0.60.

Single Sign-On (SSO): added support for SHA256 signing algorithm. New SnowMirror installations use SHA256, existing installations keep using SHA1.

Added an option to enable/disable index replication per synchronization on Advanced Settings page.

Fixed: the first run of a synchronization may have failed because SnowMirror did not create a unique name of an index

SnowMirror may have failed to properly convert value so that it can be inserted to database. Only REST API user were affected. Anyone using REST API and SnowMirror 4.12.0 should upgrade. Affected versions: 4.12.0.

Supporting a new ServiceNow release - San Diego. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

REST API performance: improved performance when downloading data using REST API. We have added "sysparm_no_count" parameter to all requests. Therefore, ServiceNow does not have to count total number of records every time we request a batch of data. That makes ServiceNow to respond faster. This is especially useful on large tables.

Schedule Date/Time - Added an optional support for Schedule Date/Time (schedule_date_time) data type. You can enable it on Settings → Advanced Settings page by setting snowMirror.mirrorTable.storeScheduleDateTimeAsDatetime property to true. Once enabled, values of schedule_date_time columns will be stored as date/time instead of varchar. If you enable this option, you should drop all tables containing "schedule_date_time" columns and synchronize them from scratch.

Java upgrade - Upgraded JRE to the latest version 11.0.14.1+1.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version 9.0.59.

task.comments_and_work_notes - If you download this column, it is and always was empty. You should download sys_journal table instead. Some customers download this column anyway. ServiceNow returns also its display value in a response. SnowMirror does not do anything with the display value. However, it slows down responses from ServiceNow. We don’t request this column from ServiceNow anymore to improve the performance.

Fixed: Test connection button on SMTP Settings page timed-out too soon.

Fixed "Validate" button: It failed when creating a new synchronization. It did not take into account "incremental column" setting.

Fixed: Rarely, SnowMirror could have failed to properly initialize and start. Because of that, some queries were executed outside a database transaction and failed with and error message "cannot reliably process 'merge' call".

Updated records during Differential Synchronization - Fixed: Differential synchronization could have missed updated records when executed by a Full Load scheduler. Only users who use "Check Updated" feature during Differential synchronization are affected.

Validate Button - added validation for database indexes of "sys_audit_delete" table. The validation is executed only if a synchronization uses Audit Delete strategy.

Fixed: Failed to start a new installation on Linux. Existing installation were not affected. Windows were not affected.

Updated Log4j to the latest version 2.17.1.

Validate Button - added validation for database indexes of "sys_archive_log" table. The validation is executed only if support for Archiving plugin is enabled.

Every request to ServiceNow contains a "User-Agent" header. From now on, it contains a value "SnowMirror".

Fixed: A name of a backup ZIP file now contains a month when it was created.

Documentation: Updated a chapter on how to increase REST API Quota in ServiceNow.

Updated Log4j to the latest version 2.17.0.

Fixed: Updated Log4j to the latest version 2.16.0. Customers with SnowMirror 4.11.8 don’t have to hurry to upgrade. Customers with SnowMirror 4.11.7 and older should upgrade immediately. If you cannot or don’t want to upgrade, please follow instructions from our knowledge base.

Java upgrade - Upgraded JRE to the latest version 11.0.13+8.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version 9.0.56.

Fixed: Updated Log4j logging library to the latest version. The older version contained vulnerability which could lead to a remote code execution. Everyone should upgrade their SnowMirror installations.

Fixed: Connection to Snowflake database may have failed. Affected versions: 4.11.2 - 4.11.6

Fixed: When connecting to SQL Server using a domain name, SnowMirror used NTLMv1 protocol instead of NTLMv2. Because of that, user was not able to authenticate. Affected versions: 4.11.2 - 4.11.5

Fixed: "Validate" button on _Settings → Performance" page could have reported an incorrect number of ServiceNow nodes.

Fixed: SnowMirror may have mapped data types of length > 8000 on Azure Synapse to and incorrect data type.

Fixed: Settings → ServiceNow → Test Connection button shown a misleading error message. In case sys_archive_log
table is empty, we now show an explanation why we cannot test ACLs of this table.

Fixed: Settings → ServiceNow → Test Connection button could have finished successfully, even if some columns were
not accessible due to ACLs.

Fixed: Synchronization to Azure Synapse could have failed. SnowMirror did not use storage access key to execute
"COPY INTO" command and which resulted in "404" error.

Fixed: Failed to connect to a built-in mirror DB.

Fixed: Settings → Database page failed to test a connection.

Fixed: SnowMirror failed to create a synchronization on MySQL configuration DB, because data did not fit into audit log.

Fixed: Windows installer failed to test database connection.

Fixed: WSTG-INPV-11 - administrator could have entered code into fields and make SnowMirror evaluate it.

Docs: How to secure SnowMirror after installation

Java upgrade - Upgraded JRE to the latest version 11.0.12+7.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version 9.0.54.

Fixed: A script for cleaning up SnowMirror Monitor.

Fixed: XSS vulnerability

Fixed: All URL (except for images and scripts) should not be cached. Therefore, we always send "Cache-Control: no-store" header.

Fixed: Google Chart library can be disabled using "snowMirror.security.useChartsExternalLibrary" advanced property.

Supporting a new ServiceNow release - Rome. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Read Replica: Added an option to download data from read-replica database. It is configurable globally on Settings → ServiceNow page or per synchronization on Advanced Settings page.

Docs: How to create a copy of your environment, Encryption, How to synchronize HTML fields, Download links in Installation chapter

SSO - Fixed: some customers were not able to login via SSO after upgrading SnowMirror to 4.10.0.

Notifications - Fixed: failed to add a notification

Azure Synapse support - you can store your data into Azure Synapse warehouse.

SSO - Simplified SSO setup:

Java upgrade - Upgraded JRE to the latest version 11.0.11+9.

Apache Tomcat upgrade - Upgraded the embedded web-server Apache Tomcat to the latest version 9.0.50.

Fixed: SnowMirror failed to start after upgrade.

Fixed: Auto Schema Update failed to change a data type of a column if an index was defined on that column.

Fixed: SnowMirror regularly cleans up Audit Log. With H2 as a configuration database, this operation has failed with "auditCleanupJob threw an unhandled Exception: java.lang.NullPointerException: null". This was a minor error - audit log is not enabled on H2 anyway.

Logs: After a successful start, snowMirror-server.log contains "SnowMirror is up and running." message.

LDAP Connection Tester: more descriptive error messages.

Notifications:

Documentation: New chapters: how to move SnowMirror from one server to another, differences between Basic and OAuth2, H2 Migration Failure, How to clean up SnowMirror Monitor. Updated chapters: User Roles.

Fixed: When an encoded query was defined, SnowMirror could have deleted records that should not have been deleted only to re-download them during the next synchronization.

Fixed: Settings → Database page did not show server and port if the name contained underscore character.

Fixed: SnowMirror could have missed records with SysIds shorter than 32 characters.

Fixed: SnowMirror failed to show a list of synchronizations, if a synchronization took more than 24 days to finish.

Windows installer: Show a warning when a user tries to install to a non-empty directory.

Docs: How to synchronize currency and price fields, How to change context in URL

Fixed: SSO - when loading roles from IdP, SnowMirror expected uppercase role names.

Fixed: SSO - failed to load roles from Azure AD.

Fixed: Data Archiving - did not load display values of restored records.

Fixed: XML Backup settings page

Fixed: Database views - do not download columns from inactive joins.

Fixed: SnowMirror Monitor - improved status information on Settings -> Stats page.

Fixed: SnowMirror Monitor - it does not re-try uploading an activity log file if it is too big to be uploaded.

SQL Server deadlock prevention: SnowMirror contains an advanced property "mirror.database.useSerializableAccess.delete". When enabled, only one delete operation on a table can be executed at once to prevent deadlocks. From now on, when a delete operation is being executed, no update can be executed at the same time.

OAuth2 supports Client Credentials flow - up until now, we supported only Resource Owner Password Credentials flow. Therefore, you had to provide us credentials of your user. From now on, we also support Client Credentials flow. That means you don’t have to provide user’s credentials - SnowMirror communicate with token provider and ServiceNow using Client Id, Client Secrets and tokens.

Fixed: SnowMirror failed to download metadata from sys_db_object, when a user did not have a read-access to the last record in the batch. From now on, we ignore such a record.

Fixed: Audit Delete strategy missed records deleted from child tables.

Fixed: A synchronization failed because it was not able to create an index in mirror database. It failed, when the following conditions were met:

Fixed: Windows installer did not contain "Require SSL" checkbox for PostgreSQL.

Fixed: Windows installer did not log potential errors into winInstaller-dbTester.log file.

New configuration option: Audit Delete strategy downloads data from sys_audit_delete and uses "sys_updated_on" column to query data. We have added an option to query data using "sys_created_on" column. It is configurable using "snowMirror.auditDelete.column" advanced property.

Fixed: Under some circumstances, differential synchronization could have deleted records that should not have been deleted.

Fixed: failed to delete a view synchronization.

Fixed: By default, Snowflake allows its users to put at most 20 transactions in the waiting queue. In SnowMirror, there was no such limit and therefore we were using more transactions per table than Snowflake was able to handle.

Fixed: Incremental synchronization of attachments failed when a record contained two or more attachments of the same name.

Fixed: Import of Directory name field option was ignored when importing synchronizations.

Fixed: REST API operation for getting a list of all synchronizations failed.

Fixed: Password fields disappeared when creating a new user and there was a validation error on that page.

Java upgrade - Upgraded JRE to the latest version 11.0.10+9.

Upgraded Apache Tomcat to 9.0.44 - we have upgraded the embedded web-server Apache Tomcat to the latest version 9.0.44.

Fixed: Invalid links in the documentation.

Use temporary tables - The option "Yes (copy data)" processes data in smaller batches. That is not desirable on Snowflake. From now on, Snowflake copies data in a single command.

Supporting a new ServiceNow release - Quebec. Of course, we are still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Use temporary tables - Added a new configuration option, which allows you to download data during "Clean And Synchronize" into a temporary table and at the end, copy the data to the target table. That is useful when you need to preserve you own custom indexes, permissions or view defined on the target table. You can configure it globally on General Settings page or per a single synchronization.

Notifications: added a new options not to send notifications about synchronizations in a "warning" state.

Fixed: Resume button did not show up in some synchronizations. Affected versions: 4.8.15 - 4.8.16

Since 4.5.2, SnowMirror started to download "dv_" columns for fields of "document_id" type. If a user has Auto Schema Update enabled and upgrades SM < 4.5.2 to the latest version, a clean synchronization is triggered. A user may not want to clean the table on production. Therefore, we added an option to ignore "dv_" columns of "document_id" fields.

Encrypted connections to PostgreSQL - added a new - simple to use - option to enable encrypted connections to PostgreSQL mirror database.

Missing columns - some customers did not see some of their columns because their sys_dictionary.function_filed was empty (instead of True or False). From now on, we treat an empty value as "false". Thanks to that, we can synchronize these columns. If you have such columns in ServiceNow and Auto Schema Update is enabled, SnowMirror adds and synchronize them automatically.

Validate button - added validation of "Table Rotation" setting.

Documentation: added a new chapter on differences between tables, views and attachments synchronizations/backups. Updated a list of supported databases, Explanation how Clean Synchronization works, information about locking, how to refresh a list of columns.

Fixed: Advanced Settings of view synchronizations did not show up.

Directory Column Name - Fixed: column name value was not used as a directory name.

Directory Column Name - Attachments are stored to a directory with a following structure: "SyncName/RecordDisplayValue". We have added a new option to customize the "RecordDisplayValue" part.

Fixed: Activity Log Retention - We have split database transactions into multiple smaller transactions. Every synchronization is processed in its own transaction. A single transaction can process up to 100 activity logs (configurable using "snowMirror.logs.retention.cleanupBatchSize" advanced property).

Fixed: Activity Log Retention was deleting old activity logs in a single database transaction. We have split the transaction into multiple smaller transactions. Because of that, other processes are not being blocked by Activity Log Retention.

We have split "execute synchronization" permission into multiple separate permissions: Execute clean synchronization, Execute differential synchronization, Execute incremental synchronization, Execute synchronization from a date/time

Fixed: synchronization of a view could have ended up in an infinite loop.

SSO Authentication - added a new configuration option "snowMirror.auth.sso.webSSOProfileOptions.maxAuthenticationAgeInMinutes". It defines a maximum time (in minutes) between user’s authentication and processing of the AuthNResponse message. The default value is (and always was) 120 (i.e. 2 hours).

Fixed: Synchronization of a view could load a list of its columns incorrectly. This could have happened when a view joins the same table multiple times, and the table has columns defined in ServiceNow. If you have such synchronization of a view, we recommend you to delete the synchronization and re-create it from scratch.

SMTP Server security: SnowMirror was using TLSv1.1 protocol to create encrypted connections to SMTP servers. We have upgraded Java mailing library to the latest version. Therefore, the default protocol is now TLSv1.2. TLSv1.1 is still supported.

Fixed: Failed to delete some synchronizations.

Fixed: Activity Log Retention was not properly cleaning "dataStorage" table. Therefore, the table could have grown way too much. This version automatically cleans up those logs that should have been cleaned up earlier.

HA Cluster: every node writes its current status into "temp/clusterNodeStatus" file. The file can be used for monitoring.

Fixed: Failed to delete a synchronization from PostgreSQL configuration database. It took too long to delete the synchronization. Therefore, SnowMirror failed on timeout error.

Fixed: Table Rotation field must be greyed out on existing synchronizations.

Improved logging: Some customers has encountered "There’s a row which does not have sys_id column or has empty value." error. This error is usually caused by insufficient permissions on that table in ServiceNow. However, it is hard to find the problematic record. We have improved logging so that we narrow down the set of problematic records.

Fixed: Failed to installed SnowMirror on MySQL when the database had "sql_mode" set to "ANSI".

Fixed: In some cases, SnowMirror checked for schema changes even if Auto Schema Update was disabled and notifications turned off. Because of that, SnowMirror re-downloaded meta-data more often than necessary.

Fast Difference Deleting Strategy - improved performance by reducing CPU usage

Silent installation on Windows - added new parameters: InstallationId, JvmMs, JvmMx

Data Truncation - In some cases, it did not cut off enough characters.

Disable Counting - Added an option to disable counting on a synchronization. SnowMirror counts records so that it can show a nice progress bar and verify if it downloaded all records. In case you have a huge table and ServiceNow times out, you can turn off counting records.

Fixed: Synchronization may have failed on "Null Pointer Exception". Because of that, it did not print the root cause of the error.

Supporting a new ServiceNow release - Paris. Of course, we’re still compatible with previous ServiceNow versions. See documentation for detailed information about upgrade process.

Meta-data: We have improved meta-data loading times and decreased the total number of requests needed to fetch the meta-data.

Version Checker - Added a version checker. If there is a new SnowMirror version, we show a notification in the footer.

Silent installation on Windows - added support for silent installation on Windows. You can install SnowMirror from command line.

Audit Log - add two new events: SyncWarning and SyncError. SyncWarning - logged when a synchronization ends up with a warning. SyncError - logged when a synchronization fails.

Java upgrade - Upgraded JRE to the latest version 11.0.8+10.

Upgraded Apache Tomcat to 9.0.37 - we have upgraded the embedded web-server Apache Tomcat to the latest version 9.0.37.

Fixed: Rest API did not return "columnsToExclude".

Documentation (new chapter): How To Create Thread Dump

Added a new configuration option (snowMirror.replication.tables.allowedNotToHaveSysId) which allows you to synchronize a table which does not have SysId column but its parent does.

Fixed: SnowMirror failed on NullPointerException when loading meta-data. It happened when ServiceNow returned a column without a name.

Missing Columns Warning - Sometimes, ServiceNow does not return all columns because of ACLs. If that happens, SnowMirror writes a warning into an activity log. Since 4.6.0 (March 13, 2020), the whole synchronization also ends up in a warning state. You can turn off the warning on selected synchronizations or or globally for all synchronizations.

In mirror tables, SnowMirror creates audit columns (mirror_created_on, mirror_updated_on, …​). We have added an option to configure their names. Go to Settings → Advanced Settings and see following properties: snowMirror.auditColumnNamePrefix, snowMirror.auditColumn.createdOn, snowMirror.auditColumn.updatedOn, snowMirror.auditColumn.deletedOn, snowMirror.auditColumn.deleted, snowMirror.auditColumn.shard
You should change the names only if no synchronization exists! Otherwise, you will lose the old columns. Alternatively, you can disable your synchronizations, change the properties and manually change the columns in your mirror database.

Fixed: By mistake, "exe" installer was not signed with our code signing certificate.

Fixed: Rest API - "scheduler.beginDate" and "scheduler.days" were mistakenly returned in UTC. The correct behavior is to return it in user’s timezone.

Fixed: incremental synchronization of views may have incorrectly failed because of alleged duplicate SysIds in master table. In reality, there were no duplicate SysIds.

Data Truncation - In some cases, data truncation did not truncate the data when updating records.

Data Truncation - In some cases, data coming from ServiceNow does not fit into a database column in mirror database. The most common and recommended solution is to change the data type in mirror database. Since 4.7.12, you can configure SnowMirror to truncate the data and store it truncated. By default, truncation is disabled. You can enable it
on selected synchronizations or or globally for all synchronizations.

Snowflake - we have decreased the number of requests to Snowflake. That results in reduced costs of Snowflake itself.

Fixed: In some cases, we started and committed a transaction in mirror database but did not run any query. By fixing this issue we decreased the number of requests being sent to the database.

Fixed: Progress bar of differential synchronization could have been rendered incorrectly. In some cases, it ended at 99% even if all records were correctly processed.

Fixed: Failed to insert default value of a new column using Auto Schema Update (no truncation). It failed when SnowMirror used Rest API to download data, and the target column was of a decimal type.

Fixed: Edit Mappings - in some cases, Edit Mappings did not change data type in mirror database after clicking "Save" button.

SnowMirror shows a warning message at the top of each page if a user does not utilize all ServiceNow nodes.

Added an option to download non-active columns - all columns in Dictionary have an "active" flag. By default, SnowMirror downloads only active columns. From now on, you can optionally download non-active columns.

Upgraded Apache Tomcat to 9.0.35 - we have upgraded the embedded web-server Apache Tomcat to the latest version 9.0.35.

Fixed: SQL Server used as a configuration database failed to compute sum of inserted, updated and deleted records when these contained really large numbers. Because of that, SnowMirror failed to display some pages (Dashboard, Synchronizations, Synchronization Detail).

Snowflake support for transient tables: When replicating data into Snowflake database SnowMirror utilizes short-lived tables to improve the overall performance. These short-lived tables are now created as "transient". Transient tables do not have a Fail-safe period and therefore lower the cost of storage.

Meta-Data Cache: added a new configuration option which allows synchronizations not to re-refresh meta-data cache on every synchronization run when Auto Schema Update is enabled. With such settings, SnowMirror decreases the number of meta-data requests. That may be beneficial (performance-wise) for installations with plenty of synchronizations or aggressive schedules.

Added a new Advanced Property: If you set the property snowMirror.stats.captureRequestCountsPerTable to true SnowMirror is going to log how many requests it will make to each table. You can access the stats on "settings/stats?minutesToShow=1440" page (open it manually in your browser). This property is useful for debugging.

Java upgrade - Upgraded JRE to the latest version 11.0.7+10.

Fixed: Filters - failed to filter synchronizations by "Auto Schema Update".

Fixed: Filters - some filters were not properly displayed.

Fixed: Installation of a Windows service under an account different than default failed to set a password properly.

Fixed: Invalid check for existence of a mirror table.

Fixed: Synchronizations Table rotation/extension enabled could have failed delete rotated table if the schedule had changed.

Fixed: Minor fixes in progress bars.

Fixed: Invalid "Back button" link on a page for creating attachment synchronizations.

Fixed: SnowMirror nodes may have failed to start. Only users using High Availability cluster were affected.

Fixed: Improved error messages

Fixed: H2 Migration Tool failed during migration of activity log state. Affected versions: 4.6.0 - 4.7.2

Fixed: Synchronizations of rotated tables deleted the rotated shard even if Deleting Strategy was set to None.

Fixed: Few information messages not properly displayed in web browser.

OAuth2 support - If you use REST API for communication with ServiceNow you can utilize OAuth2 protocol for authorization. OAuth2 allows SnowMirror to access your ServiceNow instance using access tokens rather than passing credentials with each request.

Fixed: Windows service was installed under a Local Service account instead of Local System account. Affected versions: 4.6.0 - 4.6.2

Fixed: Synchronization may have ended up in a hung state. It happened when all "getKeys" requests successfully finished and there was an error in Worker thread.

Fixed: Failed to close "Import Synchronizations" pop-up window when a user session expired.

Improved logging.

Fixed: Documentation did not contain a list of error codes.

Supporting a new ServiceNow release - Orlando. Of course, we’re still compatible with previous ServiceNow versions.

Reworked Activity Log page - Until now, you had to read an activity log to understand what was going on during the synchronization. Reading the log may be a tough task especially if you don’t know what exactly you are looking for.

Java upgrade - Upgraded JRE to the latest version 11.0.6+10.

Upgraded Apache Tomcat to 9.0.31 - we have upgraded the embedded web-server Apache Tomcat to the latest version 9.0.31.

Fixed: Progress bars - In some cases, progress bars did not show a proper percentage.

Fixed: Auto Schema Update (no truncation) failed to add data/time columns and fill leave them empty. Only users using REST API were affected. Affected versions: 4.2.0 - 4.5.13

Fixed: Diff. deleting strategy did not work without Aggregate plugin. Only SOAP API users were affected.

Fixed: Differential synchronization of attachments on SQL Server failed to deleted more than 2000 attachments at once.

Fixed: An attachment synchronization ended up in a hung state. Instead, the synchronization should have failed.

Fixed: An attachment synchronization may have failed when there were multiple attachments of the same name attached to the same record.

Fixed: An attachment synchronization ended up in a hung state. Instead, the synchronization should have failed.

Fixed: SnowMirror did not start when using MySQL 5.7.x as a configuration database.

Fixed: SnowMirror stopped to synchronize display values for columns of "domain_id" type. It deleted the "dv_" column of such columns from mirror database.

Fixed: When a column name was also an SQL keyword we automatically appended "_" at the end of the name. Since 4.5.0,
we incorrectly worked with the underscore and therefore incremental synchronization with such a column failed.

User-defined HTTP Headers - ability to specify custom HTTP headers of SOAP API and REST API calls

REST API - SnowMirror support for REST API was still flagged as beta version. From now on, it is a final version. We support SOAP API and REST API, and it is up to you which API you use.

User-defined Parameters - ability to specify custom parameters of SOAP API and REST API calls

Snowflake support:

Snowflake support:

Fixed: Synchronization failed on Out of memory exception when:

Fixed: Synchronizations sometimes failed due to "NullPointerException at …​ PerIdInMemoryHttpClientMetrics …​" error.

Fixed: Fast Difference Delete Strategy failed. Affected versions: 4.5.0 - 4.5.2

Fixed: Settings → Database page failed to save changes. Affected versions: 4.5.0 - 4.5.1

Fixed: Column names on Oracle must not exceed 30 characters. SnowMirror failed to properly shorten names of display value ("dv_") columns.

Fixed: Snowflake on Windows failed to properly create a directory for temporary files

Snowflake support - you can store your data into Snowflake warehouse.

Higher parallelization - From now on, database operations like insert/update/delete are performed in separate Data Processing threads. That is beneficial mostly for users whose database does not respond instantly.

Upgraded Java - we have upgraded the embedded Java to the latest version 11.0.4+11.

Fixed: In some cases, Windows installer failed to properly extract Java "jre" directory.

Fixed: SnowMirror refused to log in a newly created user.

Fixed: Import/Export - failed to properly process advanced settings of "Consistency Check".

Fixed: Failed to connect to MySQL configuration database due to incorrect parsing of snowMirror.properties file.

Fixed: Deactivated user was able to login.

Fixed: SnowMirror sometimes failed to load information about table rotation from a cache.

Fixed: Incorrect validation prevented users from properly setting LDAP in installation Wizard.

Supporting a new ServiceNow release - New York. Of course, we’re still compatible with previous ServiceNow versions.

Fixed: Failed to parse a date during Differential synchronization when:

Fixed: Failed to install SnowMirror on SQL Server when using schema other than "dbo".

Fixed: several minor issues on "Database → Settings" page

Fixed: Views with incorrectly defined order of tables in ServiceNow - synchronization of such views could have ended up in an infinite loop

Fixed: Synchronization of a view failed when "Settings → General Settings → Records delete" was set to Soft. Users with Hard (i.e. default value) option were not affected.

Fixed: Differential synchronization could have deleted more records than it was supposed to. Everyone who uses differential synchronization should upgrade.

Fixed: Improved replication times of differential synchronizations (especially in case ServiceNow nad SnowMirror contain a lot of data).

Use Unicode on SQL Server - added a new option "Use Unicode" which instructs SnowMirror to use unicode and NVarchar data type.

SSO - added a new advanced property "snowMirror.auth.sso.webSSOProfileOptions.includeScoping" which allows users to exclude "scoping" attribute from AuthnRequest.

Fixed: Windows installer failed to install SnowMirror as a service under a different account than Local System account.

Fixed: Test Connection button on Settings → ServiceNow page incorrectly evaluated permissions to "v_db_index" table.

H2 Migration Tool - Some SnowMirror installations use embedded H2 database. It is great for evaluation, but it is not suitable for production. We have created a tool which migrates data from an existing H2 database to some other database (SQL Server, Oracle, MySQL, PostgreSQL). Migration from H2 is optional but recommended.

Fixed: A synchronization failed to process "duration" fields when "Reference Field" was set to "Only Display Values" and REST API was used.

Fixed: Validate button on Settings → Performance page failed to validate the size of database batch.

SQL Server and Windows authentication - up until now, SnowMirror user interface did not support Windows authentication to SQL Server. You had to configure connection parameters in Advanced Properties. From now on, you can use it without any further configuration directly from "snowMirror.properties" file and Settings → Database page.

Windows Installer and Settings → Database page: added an option to make encrypted connections to SQL Server and MySQL.

Improved documentation on SSO

Fixed: Startup and shutdown scripts on some Linux environments did not execute SnowMirror’s embedded Apache Tomcat. They executed some other Apache Tomcat instead.
That happened only in an environment where some other Apache Tomcat had been already installed and environment variables CATALINA_* were set.

Fixed: Differential synchronization did not properly synchronize data in case:

SnowMirror contained duplicate and conflicting libraries. Because of that, SnowMirror failed to start on some systems.

We have upgraded the embedded web-server Apache Tomcat. Because of that, the name of the process has changed from "tomcat8" to "tomcat9". Some customers may have a rule in firewall (or some other security tool) which allows/prevents the process from running. If you have such a rule please change it so that the new name "tomcat9" is permitted.

Linux users must upgrade Java to the latest Adoptium release as described below in the Change Log. On Windows, Java is updated automatically when SnowMirror is updated.

Windows users who use SQL Server with SSO authentication (all other uses can ignore) must follow the instructions in How To chapter.

Supporting a new ServiceNow release - Madrid. Of course, we’re still compatible with previous ServiceNow versions.

Upgraded Java to 11.0.2+9 - SnowMirror is written in Java and needs Open JDK to run. Since Java 11, Oracle (the company that owns and develops Java) has changed licensing terms and support model. This topic is quite complex but the important message is that you don’t have to pay for Java.

Upgraded Apache Tomcat to 9.0.17 - we have upgraded the embedded web-server Apache Tomcat to the latest version 9.0.17.

Ability to configure SSO in cluster environment.

Added support for encrypted connections to SQL Server.

Improved performance of attachment synchronizations. The synchronization can download data from multiple ServiceNow connections at once.

Fixed: Test Configuration button on LDAP Security Realm page did not work if the form had not been saved before.

Fixed: Auto Schema Update (no truncation) failed to fill in default value for columns which have Mapping defined.

Fixed: A long column name was not properly shortened if the column was in excluded columns and only then it was chosen for synchronization.

Fixed: Clean and Synchronize button was influenced by Full Load Scheduler settings. In some cases that lead to differential synchronization instead of a clean synchronization.

Fixed: Auto Schema Update was not properly initialized when a synchronization was executed from "Synchronize…​" page and "Edit Advanced Settings" was enabled.

Fixed: Auto Schema Update (no truncation) did not properly handle display value columns a dynamic default values.

Soft Delete - besides "mirror_deleted_on", it is possible to add another column "mirror_deleted". It contains a flag which helps you to determine if a record was deleted or not. This is configurable through Advanced Properties.

Soft Delete - instead of deleting records, you can optionally mark them as deleted and keep them in mirror database.

Fixed: Differential Synchronization button did not adhere "Diff Check Updates" settings and never checked for updates.

Fixed: Page Settings → Performance and Settings → ServiceNow failed on "query did not return a unique result: 2" error. Affected releases: 4.2.0 - 4.2.3. Only users using REST API and users who changed "DB Batch Size" were affected.

Fixed: Synchronizations may have failed on "Missing lob entry, block:" error when embedded H2 database was used as a configuration database.

Fixed: If a synchronization was using table rotation, and the first shard was empty SnowMirror did not download all data. Only synchronizations using table rotation (introduced in 4.2.0) were impacted.

Fixed: A popup dialog "Synchronize From" on Detail Synchronization page incorrectly adjusted a timezone.

Fixed: Configurable synchronization execution did not respect user’s choice not to use "Diff Check Updates".

Fixed: When a new column was added using Auto Schema Update (no truncation) its default value was not properly converted to mirror database type. Therefore, the synchronization failed.

Fixed: When a new column was added using Auto Schema Update (no truncation) its "mirror_updated_on" column should not be modified.

Improvement: Added another option to Auto Schema Update (no truncation). After a new column is added, it can be left empty (i.e. we don’t fill in default value). This is configurable globally on General Settings page or per synchronization on Advanced Settings page.

Improvement: made some messages in activity logs clearer and understandable

Fixed: A folder "snow-mirror/role" containing update sets for SnowMirrorAgent role and SnowMirrorRestApiAgent role was not up-to-date or was missing.

Fixed: SnowMirror 4.2.0 introduced several new tables in configuration database. On MySQL, we have always created tables with options "ENGINE=InnoDB DEFAULT CHARSET=utf8". These were missing in 4.2.0. Because of that, SnowMirror was throwing an error. Only installations running on MySQL database were affected.

Release 4.2.0 changed the structure of its configuration database. Therefore, migration is performed during upgrade. On large installations, that may have taken a long time to finish and database timeout may have been reached. We have optimized the upgrade procedure.

Fixed: Import of synchronizations (exported in 4.2.0) may have failed.

This release introduces support for Table rotation/extension. It requires that your ServiceNow user has read permission to sys_table_rotation and sys_table_rotation_schedule. You need read permissions even if you don’t plan to use table rotation. We have also improved support for Data Archiving which requires sys_archive_log table.

Data Archiving support:

Table rotation/extension support - SnowMirror can read rotation/extension tables directly from ServiceNow shards, thus improving a performance of the replication. It can also be configured to store data into multiple mirror tables (i.e. mirror table per shard).

Configurable synchronization execution: A new Synchronize…​ page allows you to run synchronization and to choose all possible settings how to run it. Beside options which were already implemented in SnowMirror we have added:

Auto Schema Update (no truncation) - added a new option "Enabled (no truncation)" which does not truncate mirror table when a new column is created.

Exclude column from Auto Schema Update - up until now, when Auto Schema Update was enabled we synchronized all columns. From now on, you can exclude a column.

Authentication using SSO - SnowMirror can authenticate users using Single Sing-On over SAML 2.0 protocol and Web SSO profile.

Auditing - SnowMirror writes audit log when an important event occurs.

Server logs accessible from UI - you can access server logs directly from SnowMirror.

Database Batch Size - we have split Database Batch Size into two separate options so that you can fine-tune how many records to insert/update at once and how many records to read at once. By default, both options have the same value.

Consistency Check settings per synchronization - ability to enable/disable consistency check per synchronization regardless the global settings.

Incremental Offset - allows SnowMirror to pickup changes which started before a certain time but were committed to ServiceNow database later.

Synchronize From option of Full Load Scheduler - ability to schedule Synchronize from as a full load scheduler and define an offset in minutes.

Added support for MySQL 8. Older MySQL versions are still supported - no action is expected from users of older MySQL versions.

Introduced a new advanced property "snowMirror.security.showServiceNowUrls" which allows you to hide all links to ServiceNow from all pages.

Resume failed synchronizations - We have introduced Resume in 3.9.0 and dropped its support in 4.0.0. Now, Resume of failed synchronizations is back. Let’s say you are halfway through your synchronization and it fails for whatever reason. You can resume the failed synchronization and SnowMirror will continue from the point where the synchronization failed.

Upgraded embedded JRE to 1.8.0_202 and Apache Tomcat to 8.5.37

Fixed: SnowMirror identifies columns by their SysIds. If a user removes a column in ServiceNow and then creates it again (with the same name) we consider it a new column. That means a user creates and deletes a column with the same name at the same time. This can happen when ServiceNow changes its internal structure during upgrade from older ServiceNow instance to a newer one.

Fixed: There’s a Validate button on Edit Settings page which allows you to check if deletes from a table are audited or not. In some cases, the verification provided incorrect results because it was solely based on "Audit" flag in sys_dictionary. From now on, we also take into account no_audit_delete flag, glide.ui.audit_deleted_tables and glide.db.audit.ignore.delete system properties. Therefore, the verification is accurate.

Fixed: SnowMirror failed to load a list of table names if a name was empty in ServiceNow Dictionary.

Fixed: Audit deleting strategy used incorrect batch size when checking for archived records. Only installations with Archive Plugin support enabled were affected.

Added a new advanced property "snowMirror.logger.logFailingRestMessagesToActivityLog". When enabled, failing REST API responses are written to the activity log. Suitable only for debugging purposes. Do not forget to turn it off after you finish debugging!

Fixed: SnowMirror failed when max. length of a column in Dictionary was empty.

Fixed: SnowMirror failed to Clean and Synchronize incremental view.

Added a new advanced property "mirror.database.useSerializableAccess.delete". When enabled, all deletes on a particular table are synchronized. That means the transactions do not interfere with each other. This feature is disabled by default. Allowed values: true, false

Improved logging of SnowMirror’s communication with REST API.

Fixed: a memory leak - activity log loggers did not cleanup the memory properly.

Fixed: Since 4.1.0, synchronizations did not download display values when using SOAP API

Fixed: REST API did not initialize when Oracle was used as a configuration database

Supporting a new ServiceNow release - London. Of course, we’re still compatible with previous ServiceNow versions.

Supporting REST API - you can download data from ServiceNow using their REST API. To do that you need appropriate permissions. This feature is in a beta version.

Fixed: in some cases, SnowMirror did not close an activity log. This could lead to "Too many open files" error and SnowMirror went into a hung state.

Fixed: on Oracle, SnowMirror sometimes failed with ORA-24816 error.

Upgraded embedded Apache Tomcat to 8.5.33

Upgraded embedded JRE to 1.8.0_181

Upgraded embedded Apache Tomcat to 8.5.32

Fixed - on rare occasions, SnowMirror failed to measure duration of HTTP request/response properly and therefore a synchronization failed.

Fixed - Audit deleting strategy did not order records in ServiceNow correctly which could have lead to premature termination of the synchronization.

Fixed - Audit deleting strategy use an incorrect column name in its requests which could lead to infinite loop. Only incremental synchronizations of views were affected. Tables were not affected.

Fixed - SnowMirror failed to recover the connection to ServiceNow if ServiceNow kept responding with HTTP 202 status code.

Fixed - Attachment synchronizations finished when "get keys" phase returned less than 10,000 keys. From now on, it finishes when "get keys" returns zero keys. This is more convenient because in some cases it could have finished sooner than expected.

Reworked how synchronization priorities work. Up until now, synchronizations of higher priorities took over all available synchronization threads hence effectively blocking the lower priority synchronizations. From now on, all running synchronizations can be assigned a thread. Higher priority synchronizations gets the thread more often than lower priority synchronizations.

Fixed - Attachment synchronizations failed to unpack certain downloaded attachments.

Fixed - Possibly missing some records from sys_audit_delete table when Audit Deleting strategy. Only incremental views were affected.

Fixed - the "Data" tab did not adhere "snowMirror.auditColumnNamePrefix" property and resulted in an error. Only SQL Server and Oracle were affected.

Fixed - SnowMirror failed when trying to delete more than 1000 records at once. Only installations with Oracle mirror database were affected.

Fixed - Activity log cleaner did not work properly if the path of activity log dir started with "file:/", "file://" or "file:///" prefix. From now on, SnowMirror can handle paths with the prefix and without the prefix.

Fixed - On Linux, SnowMirror logged logs into two files (catalina.out, snowMirror-server.log) instead of just one file (snowMirror-server.log)

Fixed - differential synchronization of attachments was failing

Fixed - potential errors if there are misfired synchronization runs right after restart

Fixed - the new Audit Delete Strategy was causing deadlocks on SQL Server

Fixed - potential Null Pointer error while gathering stats about request/response duration. The error lead to synchronization failure and was solvable by restarting SnowMirror.

Dramatically improved performance - SnowMirror can utilize multiple ServiceNow nodes and download data in parallel. Therefore, replication times were improved. See Performance Tuning chapter for details.

Added a new Stats page which provides details of SnowMirror’s background processes.

Added a new deleting strategy Fast Difference - it counts records and downloads max. value of sys_updated_on column. If the values don’t match with the mirror database it splits the SysIds into smaller chunks and repeats until it gets until to 10,000 keys. Only then it downloads the keys and compares them with the mirror database. This strategy is useful when you have a large table and small amount of deletes because it reduces the number of requests to ServiceNow.

Simplified an installation process of High Availability Cluster. SnowMirror installations can be marked as Active or Passive. By default, one installation is Active (runs synchronizations) and the second one is Passive (takes over if the Active installation is down). See Installation Types chapter for details.

Updated Apache Tomcat to 8.5.30 and Java JRE to 1.8.0_172.

Fixed: When Auto Schema update forced a full load and synchronization was supposed to synchronize to temporary table, SnowMirror failed to initialize the temporary table and the synchronization failed.

Fixed: Audit Deleting strategy failed when synchronizing a view incrementally.

Fixed: Resume of a synchronization did not work when SnowMirror was shutdown.

New: Ability to hide "Data" tab using "snowMirror.viewData.enabled" advanced property.

Fixed: Since 3.9.1, SnowMirror started to create date/time columns on MySQL database as a "timestamp" datatype. That was a bug. Since 3.9.4 we’re switching back to "datetime" datatype.

Fixed: Upgrade scripts failed on MySQL 5.5.

Synchronizations with an encoded query have to download all updated sys_ids (regardless their query) during incremental synchronization run. Thanks to that, we can find out and delete those records that ceased to match the query. That is not a desired behavior for tables which are never updated. From now on, you can disable this by manually modifying configuration database - just set "checkRecordsMatchingQuery" field in "synchronization" table to "0" (or "false" depending on your DB server).
Example SQL (update synchronization with id = 1): Update synchronization Set checkRecordsMatchingQuery = 0 Where id = 1;

When Consistency check is enabled it is performed on each synchronization. The check may take a while on very big tables. You can optionally disallow the check on selected synchronizations. Set "consistencyCheckAllowed" field in "synchronization" table to "0" (or "false" depending on your DB server).
Example SQL (update synchronization with id = 1): Update synchronization Set consistencyCheckAllowed = 0 Where id = 1;

Mapping of ServiceNow data types to database data types can be changed globally.

Supporting a new ServiceNow release - Kingston. Of course we’re still compatible with previous versions.
Since Kingston, users can create function fields. Function fields are not available through SOAP API that SnowMirror uses. Therefore, you cannot synchronize them. If ServiceNow makes them available through SOAP we’ll support them.

Upgraded JRE to version 1.8.0_152 and Apache Tomcat to version 8.5.24.

Fixed: an incremental synchronization that run after a differential synchronization may have missed updated records. It incorrectly derived its start date from the diff. synchronization - and that was wrong it should have picked up the data from the synchronization preceding the diff. synchronization.

New feature: Resume failed synchronization. Let’s say you half way through your synchronization and it fails for whatever reason. From now on, you can resume the failed synchronization and SnowMirror will continue from the point where the last synchronization failed.

Not shortening column names to 30 characters on new SnowMirror installations. Up until now, a column name had to consist of at most 30 character. Longer names had to be shortened. From now on, SnowMirror shortens the column name only if the database server requires that.

Improved logging.

SnowMirror can optionally remove invalid XML characters from ServiceNow responses. The feature is disabled by default. You can enable it using "snowMirror.http.responseBodyCharReplacement.enabled" advanced property.

Fixed: If SOAP API did not return some records SnowMirror did not download all table names available in ServiceNow.

Fixed: From now on, LDAP Groups to SnowMirror roles mapping is case-insensitive

Fixed: Failed to export selected synchronizations.

Fixed: Access denied when creating a view / generating view’s SQL

Fixed: Possible XSS on "Show Log" page through synchronization name.

Fixed: Possible redirect to another domain using Change Timezone page.

Fixed: Diff. deleting strategy could not handle responses where some records were filtered out due to ACL settings. That resulted in deleting records which should not have been deleted.

Fixed: invalid link to ServiceNow on Synchronization detail page

Fixed: Bulk Edit Scheduler was inaccessible

Improvement: Added a new configuration options allowing to map LDAP group names to SnowMirror roles and define fallback role in case a user does not belong to any group.

Fixed: Page Not Found page was reloading endlessly.

Fixed: SnowMirror attachment/backup failed when a folder name contained tabulator.

Fixed: SnowMirror failed to change Reference field type from "Display value" to "Both".

New feature: Added prevention against CSRF attacks.

New feature: Added prevention against brute-force password attacks.

Improved: Prevention against XSS attacks.

Improved: Using "bcrypt" for password hashing instead of deprecated SHA-256. Existing user are migrated to the new function on the first login.

Improved: Revised permissions to AJAX testers and other SnowMirror pages.

New feature: Added a new property snowMirror.security.useChartsExternalLibrary which instructs SnowMirror (not) to access external Google API for drawing charts.

Updated: Updated embedded software to the latest versions: JRE - 8_u144 and Tomcat (8.5.20)

Auto Schema Update: improved logging.

Fixed: "Edit user" page did not load.

Fixed: LDAP connection made incorrect BINDs.

Fixed: Security Realm page: missing validation message

Fixed: Security Realm page: "Show Advanced Properties" option was checked even when it should not have been checked.

Fixed: "Change password" button was greyed out.

Fixed: Could not change security realm back to Internal database due to validation error.

Fixed: ServiceNow Test connection button sometimes ignored a username and was using a login of the saved connection.

SnowMirror version can be optionally hidden from GUI by setting "snowMirror.security.showVersion" property to "false".

Fixed: SnowMirror failed to generate SQL of some views when running on Java 8u20 and later.

Added support for LDAP authentication. See documentation for details.

Added support for Jakarta.

Added an option to hide SnowMirror version from a footer (see Security Properties chapter).

Fixed: bulk create/import took too long to commit the database transaction.

Fixed: EditMappings page failed to load if mirror table name did not match ServiceNow table name.

Fixed: EditMappings page failed to load if mirror table name did not match ServiceNow table name

Fixed: shutdown script on Linux ignored "snowMirror.controlport" settings. If it was set to non-default value if failed to stop SnowMirror.

Fixed: a list of databases did not show up if there was a validation error

Fixed: notification about failed backup mentioned mirror table but backups don’t have mirror tables

Fixed: pagination on the list of synchronizations failed when a filter was applied

Fixed: failed to filter "New" synchronizations

Fixed: The issue influenced only those using SnowMirror Monitor. The background job that sends data to SnowMirror Monitor in your ServiceNow instance was piling up triggers in the configuration database. This prevented SnowMirror to start in a reasonable time frame.

Fixed: Scheduled backup failed after synchronization started.

Improved backup tool: added support for Differential backups.

Improved backup tool: added support for attachments backup.

Support for NTLM proxy authentication. Can be enabled by setting "snowMirror.proxy.authScheme" advanced property to "ntlm".

Fixed: Import did not rename mirror table if its name was changed.

Fixed: If "Clean & Sync to temporary table" found a new indexes it did not create them in mirror database.

Fixed: Run button on filter form did not respond in some cases.

Waiting for a one second after getting max. sys_updated_on to so that getKeys can always pick up records upserted in the last second.

To tackle a possible issue with updating Oracle CLOBs we added a new advanced property "mirror.oracle.jdbc.useSqlLobValueObject". By default it is disabled and SnowMirror will function without any changes. If you enable it, it will treat CLOB values differently:
Before the change, we stored CLOBs into database prepared statements using setString method.
With this change enabled, we 1. Create temporary CLOBs, 2. store their locators to the mirror database and persist CLOBs 3. free temporary CLOBs so that they don’t stay in memory forever.

Fixed: Linux installations - in case we cannot write Tomcat PID file to /run or /var/run directory we write it to the bin inside SnowMirror installation folder. This prevents "permission denied" errors.

Fixed: Windows installer - service is installed no matter what database connection is provided.

Fixed: Import failed it "syncTime" was not present in the file.

Supporting licenses for our Backup tool.

Added a new option to skip failing records and allow the synchronization to finish despite errors.

Added a new option to perform incremental synchronization from a certain date.

Added a new option to choose a data type to which a column will be mapped in mirror database. See Edit Mappings for details.

Updated the embedded Apache Tomcat application server to the latest version.

Updated Java to the latest version. If you use Windows installer, it is upgraded automatically. Otherwise, you have to upgrade yourself (unless you’re already running Java 1.8.x).

Improved security: using X-Frame-Options, X-XSS-Protection and X-Content-Type HTTP headers.

Improved security: not disclosing versions of software libraries we use

Improved security: added new advanced properties: "snowMirror.security.showExceptionOnErrorPages" and "snowMirror.security.showSystemInfoOnErrorPages". They control what information will be shown on error pages. By default, we display both error stack trace and system information (SM version, database server, etc.). If you don’t want to show that on error pages go to Setting → Advanced Settings and set those properties to "false" (without quotes).

Added a new advanced property "snowMirror.mirrorTable.continueIfOneByOneSucceeds". If a batch of inserts or updates fails we start to insert/update one by one and the we fail the synchronization. This property allows you to continue if one-by-one succeeds. Turned of by default.

Windows Installer failed to install SnowMirror on newest MySQL servers.

Progress bar on the Detail Synchronization page did not show data properly.

Minor fix on login form (contained our contact information used on conventions)

Support for Istanbul release

New feature: advanced filtering of synchronizations and activity logs

New feature: synchronization queue which shows synchronizations which are running or waiting to be executed

New feature: synchronization priority. Each synchronization can have a priority. The synchronizations with higher priority will run first.

Fixed: Auto Schema Update on Oracle mirror database failed to replicate indexes on columns whose name was a reserved word.

Fixed: synchronization of columns with a name longer than 30 characters failed on NullPointerException when "Only display value" option was selected.

Fixed: Failed to setup a new SnowMirror while using Oracle as configuration database

Index replication: indexes are loaded from "v_db_index" table instead of unreliable "v_index_creator" table.

Optimized memory consumption: Differential Synchronization and Differential Deleting Strategy so that they don’t need to load all SysIds from database and from ServiceNow to the memory.

Fixed: Do not send notification about a new index in case the index is defined on a column not recognized by SnowMirror.

Updated memory settings - after start, SnowMirror allocates 700MB of memory. Then, it is allowed to allocate up to 2GB of memory.

Updated license agreement.

Fixed: Downgraded Oracle driver from the 12.1.0.2 to 12.1.0.1 because a bug in the newer version causes "ORA-01000: maximum open cursors exceeded" error.

Fixed: Database resources were not always properly closed when using Oracle. That resulted in "ORA-01000: maximum open cursors exceeded" error.

Fixed: SnowMirror may have missed some records during full load view synchronizations.

Fixed: REST API failed to convert "Warning" status to the JSON response (Internal Server Error).

Index replication - SnowMirror can find out what indexes are defined in ServiceNow and replicate them to you mirror database. See documentation for more details. This feature is disabled by default.

Notification on schema change - SnowMirror is able to find out changes in ServiceNow database schema (new columns, delete column, changes) and send you a notification. If Auto Schema Update is disable it won’t store the changes to your mirror database - it just sends the notification.

Primary index on sys_id columns - by default, SnowMirror creates a unique index on sys_id columns in mirror database. You can choose to create primary keys instead.

Improved database connection pool settings - turned on connection validation to prevent corrupted connections.

Minor bug fixes in Windows installer.

Several minor bug fixes and improvements.

Fixed: "UnknownFormatConversionException: Conversion = '.'" error. Happened only when "snowMirror.replication.denySyncWithoutIndex" was enabled to prevent synchronization without index on sys_updated_on column.

Fixed: Attachment replication failed to synchronize some attachments.

Fixed: If a field was of a list type and a synchronization was set to mirror only display value, SnowMirror created a field in mirror database with incorrect length.

Fixed: Improved performance of inserts/updates on PostgreSQL mirror database.

Fixed: When synchronization with a periodical scheduler was deactivated and then activated, the scheduling subsystem started to ignore minutes at which the synchronization should be executed.

Introduced a new advanced property 'mirror.database.batchSize' which allows users to increase or decrease database insert/update/delete batch size.

202 HTTP response issues fixed: if SnowMirror receives 202 HTTP response then it tries to send the request again several times. If it still does not receive the response it re-sends the request one more time with a new session.

Supporting ServiceNow Helsinki release

Added support for backups. It’s a beta version and is currently available only with trial license. You can store your data to XML or CSV file. Later, you can restore you data using these files. See detailed description in User manual.

Added support for replication of archived records - that means SnowMirror can delete records which were archived. SnowMirror can also synchronize archive tables (ar_*) - it uses "Archived" column to find out new records. The feature is disabled by default and can be enabled by setting advanced property "snowMirror.replication.dataArchiving.enabled" to "true".

Fixed: When "Timezone Data Types" option was set to "Yes" and timer data type was supposed to be stored as a number then SnowMirror created timer column in mirror database with wrong data type.

Fixed: SnowMirror imported inherited columns (which were listed in import file) even though "include inherited columns" was disabled.

Fixed: SnowMirror sometimes failed on deadlocks (usually on SQL Server). We have revised locking in SnowMirror and therefore got rid of those failures. We have also revised transaction management of the scheduler system to prevent some potential inconsistencies in scheduler settings.

Fixed: when using SQL Server as a configuration database SnowMirror sometimes failed on deadlock exception.

Fixed progress bars because they shown updated records incorrectly.

SnowMirror Monitor: changed integration pool name to "x_awns_smm_snowmirror_integration_pool" and fixed an issue in Monitor which did not process activity logs.

Fixed: SnowMirror failed to authenticate against SMTP server therefore failed to send notifications. From now on authentication have to be enabled using "snowmirror.mail.server.auth" advanced property (enabled by default). Some servers may require the use of STARTTLS command - to enable this a user have to set "snowmirror.mail.server.starttls" advanced property to true.

Fixed: SnowMirror 3.1.x loaded different column lengths (especially for reference columns) than SnowMirror 3.0.3 and lower. This was caused by a cache which was incorrectly returning an empty list of ServiceNow data types.

Fixed: Even if Include Inherited Columns was disabled SnowMirror 3.1.x considered columns inherited from parent tables as a new columns during Auto Schema Update. That resulted in unnecessary clean of mirror table and download of all data.

Fixed: Two bugs which manifested only when Clean and Synchronize to temporary table was enabled.
1) When Auto Schema Update detected a change in meta-data and required to clean mirror table then SnowMirror incorrectly showed a number of records to delete. Records are not deleted during the synchronization. Data are stored to temporary table which replaces the mirror table at the end of successful synchronization.
2) When meta-data changed and a user edited synchronization settings then SnowMirror cleaned the table. The expected behavior is to execute Clean and Synchronize to temporary table during the following synchronization. This way data in mirror table will be available even during full load synchronization.

Fixed: Improved Clean and Sync to temporary table. Up until now SnowMirror has been creating two activity logs - one for clean phase and second one for the data migration. From now on if you have "snowMirror.replication.cleanAndSyncToTmpTable" enabled then there will be only one activity log. Also the log itself was improved so that it is clearer how SnowMirror synchronizes to temporary table.

Fixed: Count Records button did not work for attachments

Added: SnowMirror logs general statistics about its usage to activity logs. It logs counts of synchronizations and schedulers, table sizes and longest running synchronizations. Stats can be disabled using "snowMirror.stats.logToActivityLog" property.

Supporting new ServiceNow release - Geneva.

Improved performance of Bulk Create.

SnowMirror is able to figure out lengths of Display Value columns.

SnowMirror requires new access right: access to sys_dictionary_override table. This is required so that SnowMirror can figure out lengths of display value columns. See AdministratorGuide.html for details on ACLs.

New SnowMirror Monitor application. It provides overview of synchronizations directly from ServiceNow.

SnowMirror failed synchronize to MySQL 5.7.9 because the way we accessed variables in MySQL was deprecated.

Database Setting form - failed to store SID or Service for Oracle

SnowMirror failed to synchronize display values for columns whose name was longer than 30 character.

Windows installer failed to validate database connection which prevented users from installing SnowMirror successfully.

New replication algorithm. Due to performance reasons we have changed replication algorithm so that it does not have to use sys_created_on column. New algorithm uses only sys_updated_on column. Sys_created_on is only used when synchronizing a table that does not have sys_updated_on column. Also new algorithm does not limit the date until it synchronizes data. This needed to be done because the data sets may change during synchronization.

Added filters - a user can filter synchronizations and their activity logs. To start using filters just click the funnel icon above the list of synchronizations.

Added "Validate" button to each synchronization - it allows a user to check that the table has sys_updated_on column, has appropriate indexes and is accessible using webservices.

Improved dashboard (home page) of SnowMirror web interface.

Improved logging - after synchronization finishes data replication SnowMirror logs number of records in ServiceNow, number of records upserted (inserted or updated) and deleted in ServiceNow since the last synchronization. It also logs the number of records in mirrored table. The logging can be turned off using "snowMirror.tasks.countRecordsAfterSynchronization.active" advanced property.

ServiceNow Dashboard (i.e. the update set allowing you to see synchronizations from within ServiceNow) was deprecated and removed.

Created a new ServiceNow application replacing the deprecated Dashboard update set.

After creating new synchronizations a user can choose to run them immediately

Columns too long are shortened to 30 characters. If the resulting names conflict then numbers are added to the end of the name.

Mirror Database Settings form now allows to choose if you want to use SID or Service on Oracle.

Fixed: After editing scheduler time, the time was shifted by one hour when Active From was in daylight savings time and current time was not.

Fixed: Windows Installer always made the service to start automatically even when the service was set to start only manually.

Fixed: Import scheduled a synchronization even though it was disabled.

Several minor improvements and bug fixes.

Fixed: SnowMirror allows to store dates and time to mirror database in any timezone a user chooses. Values of glide_date type were also converted to the chosen timezone. That did not adhere ServiceNow conventions hence glide_date values won’t be converted to the chosen timezone anymore.

Fixed: In some cases SnowMirror incorrectly loaded list of existing columns on Oracle. That made synchronization to fail.

Improved logging.

Improved replication reliability on very large tables.

Fixed: If SQL Server was used as configuration database SnowMirror failed to upgrade from older versions properly. That resulted in "Synchronization doesn’t exist" errors even though the synchronization existed.

Fixed: SnowMirror failed to synchronize a column whose name ended with underscore sign. This happened only if "Quote identifiers" was disabled (unchecked) and mirror database was either Oracle or PostgreSQL.

Fixed Diff deleting strategy - Diff deleting strategy was removing more records than it should have been removing. Detailed explanation: diff deleting strategy downloads sys_ids of all records from the table being synchronized. This is done in batches of 10,000 sys_ids. SnowMirror stopped downloading the sys_ids if it received less then 10000 sys_ids. We fixed this behavior because receiving less than 10000 sys_ids does not mean there are no more sys_ids - usually because of ACL settings.

SnowMirror started a synchronization from incorrect time - this happened when a record being synchronized was updated in ServiceNow during its synchronization.

On Oracle, SnowMirror uses Varchar2 data type for storing strings. The max. length is defined in character. In some cases (e.g. adding new columns using Auto Schema Update) SnowMirror defined max. length in bytes which could have lead to data truncation exceptions.

Synchronization with Weekly scheduler was sometimes scheduled to a different day than expected because it was incorrectly converted to UTC date.

Changed mapping of data types on Oracle. SnowMirror stored strings to to Varchar2 with length up to 4,000 characters. That type can only store up to 4,000 bytes hence is not suitable for storing multi-bytes characters. To tackle the issue SnowMirror now stores string to Varchar2 if the max. length does not exceeds 1000 characters. Anything bigger than 1000 characters goes to CLOB.
Also to avoid storing display values to CLOB we lowered the max. length of data type for display values from 4,000 characters to 1,000 characters. The change affects all databases not only Oracle.

Fixed: SnowMirror failed to synchronize to Oracle 12

Improvement: Import failed when table did not exist in ServiceNow. From now on the non-existing table is skipped and the rest of the import may proceed.

Improvement: Improved performance of configuration database on SQL Server.

Improvement: SnowMirror can download display values even for "domain_id" and "glide_list" types.

Fixed: SnowMirror failed to insert dates less than January 1, 1753.

Fixed: SnowMirror failed to recover after failed import

Fixed: SnowMirror failed to start on some Linux installations because it was loading dependencies in incorrect order.

Fixed: Some synchronizations incorrectly assumed there was no sys_updated_on column hence did not synchronized updates. This only applied to table synchronizations which were created using bulk create and the table has a parent table. To ensure such table is fully synchronized you can perform Clean and Synchronize.

Fixed: changing Reference Fields synchronization type failed when none of selected columns was reference field or choice list.

Improvement: If SnowMirror fails to insert or update data it will optionally log sys_id or all columns of the failing record. This is useful for debugging purposes. It can be enable through Advanced Settings by setting properties "snowMirror.mirrorTable.logFailingRecord" or "snowMirror.mirrorTable.logFailingRecordSysId" to "true".

Improved performance of embedded configuration database

Fixed: SnowMirror failed to import large export files

Fixed: Validation that table exists on Oracle was searching in all tablespaces.

Fixed data truncation exceptions - SnowMirror failed to synchronize columns which referenced to several values which resulted in error "Data truncation: Data too long." From now on this error is fixed.

Added support for display values synchronizations. For reference fields a user can either synchronize sys_ids of referenced records or display values of referenced records or both of them.

Added support for setting a different timezone in mirror database - SnowMirror stores dates and times in UTC. From now on a user can choose a different timezone to which dates and times are converted and stored to mirror database.

Added progress bars for long running task - Bulk create and Import

Fixed: SnowMirror failed to store big Decimal values to mirror database. For that cases it now uses a bigger precision.

Improved activity logs for easier debugging

Several minor bug fixes and improvements

Support for attachments synchronization - attachments are stored to filesystem (instead of mirror database).

Fixed an issue where SnowMirror failed to create a new synchronization and failed with error message "Cannot process ServiceNow response because it does not contain sys_id column.". The issue was affecting only tables containing more than 250 columns.

Added support for new ServiceNow release - Fuji.

Added support for Full Load scheduler. Beside the Incremental Scheduler, which periodically downloads increments, SnowMirror now allows to also define Full Load Scheduler. This way you can download all data once in a while.

Added password encryption - credentials to config database, mirror database and ServiceNow are encrypted using AES 256. IMPORTANT: if you are upgrading your Windows installation manually (not using Windows installer) you have to copy all files from "/snow-mirror/jce" to your JRE installation (more specifically into "jre/lib/security" folder). These files are suited for Java 7. If you run different version you can download the files from official Oracle website. On Linux installations you have to copy those files not only while upgrading but also when you are installing a new SnowMirror installation.

Improved views synchronization.

Improved UI - all necessary information is visible right away (e.g. status bars).

Automatic detection of user’s timezone when creating a new user.

Minor bug fixes for PostgreSQL database.

Added a new configuration property: snowMirror.mirrorTable.storeGlideDurationAsNumber. ServiceNow sends "glide_duration" to SnowMirror as a timestamp. For example duration of 10 minutes is sent as "1970-01-01 00:10:00". Since the first release SnowMirror stored this timestamp to mirror database. This property allows you to store duration fields as a number of milliseconds instead of the timestamp.

Added a new configuration property: snowMirror.mirrorTable.storeDateTimeWithLocalTimezone. By default SnowMirror stores all date/time values to the mirror database in UTC timezone and uses database data types which does not preserve the timezone information. In some cases it may be convenient to use different data types - those preserving the timezone information. If you enable this option SnowMirror will use different data types instead: "timestamp with local time zone" (Oracle), "timestamp with time zone" (PostgreSQL), "timestamp" (MySQL), "datetime" (H2), "datetimeoffset" (SQL Server).

Minor bug fixes: in some cases, "Generate SQL" button failed.

Import ignored a user’s timezone.

Improvement: Configuration changes made from Settings page don’t require to restart SnowMirror anymore.

Improvement: Optionally using DELETE FROM instead of TRUNCATE. When deleting all data from mirror database (e.g. while calling Clean and Synchronize) SnowMirror uses "TRUNCATE" SQL command. But in some environments SnowMirror is not allowed to call it. You can force SnowMirror to use "DELETE FROM" instead by enabling mirror.database.cleanDataWithDeleteCommand property on Settings / Advanced Properties page.

Improvement: Button bar on "Synchronizations" page is shown both above and below the table.

Improvement: Notifications can be optionally sent after every synchronization run.

Improvement: Renamed a role "Synchronization Reader" to "Synchronization Supervisor" to better reflect what the role means. Also added a new role "Synchronization Reader" which allows a user read-only access to SnowMirror.

Improvement: Added "Remember Me" function. Before logging in to SnowMirror a user can choose to stay logged in indefinitely.

Fixed: ServiceNow connection tester did not check permissions for tables sys_db_view, sys_db_view_table and sys_db_view_table_field.

Fixed: Rest API was always mistakenly returning "true" in "active" attribute. Now the Active attribute reflects whether the synchronization is enabled (true) or not (false).

Fixed: Improper handling of "serviceNow.problem.job.retryWaiting" property. If there are no more values available SnowMirror now uses the last value defined in "serviceNow.problem.job.retryWaiting" property. Before this change SnowMirror used 0 seconds instead.

Several minor improvements and bug fixes.

Improvement: Beginning 2.3.0 SnowMirror allows to choose events about which users will be notified. First option is to receive notifications about first failure and first successful synchronization after failure. The second option is to receive notifications about all failures and the first successful synchronization after failure.

Improvement: Notifications about failed synchronization now contain more detailed information about the failure including a log file.

Improvement: searchbox on the list of tables (on the new synchronization page) now provides more accurate results

Improvement: increased performance for updates and deletes to mirror database running on SQL Server.

Fixed: Every user (regardless his role) can change his own password

Several minor improvements and bug fixes.

New feature: SnowMirror needs to access ServiceNow instances which may be hidden behind a proxy. Now we provide a web form to easily specify a proxy settings.

Improvement: improved connection tester buttons - now the user sees a full checklist of what was being tested. In case SnowMirror is not setup correctly a user has a better feedback from the system.

Several minor improvements and bug fixes.

Improvement: it is now possible to specify "instance" name for SQL Server.

Fixed: Clean and Synchronization did not start synchronization when mirror table was empty.

New feature: update columns meta-model on synchronization run, i.e. adding/updating/removing columns in ServiceNow is reflected in SnowMirror

New feature: bulk create of new synchronizations

Improved performance

Several minor improvements and bug fixes

Improved performance of build-in database

Improved performance of the component for selecting columns

Improved Test Connection button in ServiceNow Settings page - now it warns users to set up proper privileges in ServiceNow

Fixed - in case of an error Clean and Synchronize button did not log the error to the activity log properly

Several minor improvements and bug fixes

Exposed a new REST API. It allows to perform CRUD operations on synchronizations and to run the synchronizations.

Added an option to generate SQL script for creating a view in client’s database. It can be used to tackle that it is not possible to perform incremental updates of views.

Configuration now supports "schema" for SQL Server database

Fixed: Periodical scheduler ignored a time when the first synchronization should start.

Several minor improvements and bug fixes

Now supporting Eureka ServiceNow instances

Added support for synchronization of database views

Added support for changing a schema when using PostgreSQL

Fixed: SnowMirror failed to import inherited columns

Several minor improvements and bug fixes

Added support for changing SnowMirror colors, logo and a text right next to the logo from Settings page

Improved Configuration Wizard so that it can restart itself after setup.

Several minor improvements and bug fixes

Configuration properties moved to database - added Advanced Properties to Settings page to easily tweak SnowMirror behavior

Improved Windows Installer. It allows to setup configuration database

Several minor improvements and bug fixes

SnowMirror works even without Aggregate Web Service Plugin.

Added import and export of synchronizations.

Several minor improvements and bug fixes.

Added filters and sorting on Synchronization History page.

Added a button on Synchronization Detail page for finding out a number of records in a table.

Several minor improvements and bug fixes.