Purpose

The SnowMirror Drive application moves attachments from ServiceNow into alternative cloud storage to save database size and costs on SN side.

SnowMirror Drive - Application Menu

The contents of the application menu:

• Workspace - User friendly UI where the users of the application can view statistics, live data and an interface for admins to control the application.
• Storage Providers
• Attachment Rules
• PA Data Collector Jobs - Scheduled Data Collection jobs that setup the scores of different metrics and graphs displayed on the Workspace.
• Required Actions
• Log - Error/info log menu for administrators
• Support - Link to official website, where user can create a support case
• Groups
  • Admin Group - Group used for distributing administrator privileges for the whole application.
  • User Group - Group user for distributing user privileges for the whole application.

SnowMirror Drive Workspace

The SnowMirror Drive Workspace is available from the Workspaces menu and from the application navigator under: SnowMirror Drive → Workspace

Home Page

The Home page provides a comprehensive overview of upload and download activity, storage utilization, key Performance Analytics (PA) metrics and any required configuration actions. The page is divided into clearly defined sections for quick access and visibility.

Information about your instance

This section displays key Performance Analytics (PA) scores, helping you evaluate your application’s efficiency and your instance's total capacity. Metrics included are:

• Actual Savings (GB / $): Actual storage and cost savings.
• Potential Savings (GB / $): Future maximum potential storage and cost savings.
• Instance DB Size (GB): Total size of your instance database. These insights allow you to monitor profitability and assess current resource utilization.

Transfer status

A toggle control that enables or disables the mass transfer process (bulk upload of attachments). Run Mass Download Once button that is used to start bulk download of attachments.

Transfer progress

View the current state of scheduled jobs for bulk uploading and downloading attachments. Monitor ongoing file uploads with this card:

• Progress Bar: Displays transferred versus total attachments to upload or download.
• Estimated Time: Shows remaining time for completion.
• Auto-Refresh: Progress bar updates automatically every 15 seconds via script for up-to-date information.
• When transfers are paused, the next configured upload window is displayed.

Storages

This section lists all configured storage records, with an at-a-glance indicator of their status:

• Green Dot: Storage is online.
• Red Dot: Storage is offline.
• No Storage Configured: If no storage is set up, a relevant message is displayed. Each storage name acts as a clickable link that navigates directly to the storage record form for easy management and troubleshooting.

Required Actions

This area highlights necessary configuration steps to ensure your application is set up and running optimally, particularly for first-time use. Required actions are managed through records in the x_gvs_offloader_required_action table, which contain scripts to dynamically determine their visibility.

• Severity Indication: Each action card uses a colored border to reflect its severity.

• Action Button: Each card includes a button that will either link to the appropriate page or execute a script to activate a feature or resolve an issue. Available Actions include:

• Basic setup required: Alerts if basic properties on the Settings page are incomplete. The button navigates to the last step of the Basic Setup wizard.

• Historical data collection didn’t run: Indicates if the [PA] Data Offloader (Historical) scheduled job has not run; the button navigates to the scheduled job.
• License key expired! Shown if the application license key has expired; the button takes you to Settings > Basic Setup to update the license.
• Mass upload not activated: Warns if mass upload is not activated; the “Activate” action runs a script to enable scheduled job.
• No rules defined: Alerts if no active attachment rules are set; the button navigates to the Attachment Rules page.
• Storage connection failing: Displays if at least one storage provider is offline.
• No storage configured: Appears if there are no storage records; the button navigates to Settings > Storage Setup.
• Automated table compaction will not run: Indicates the automatic table compaction size limit has been reached; the fix action links to ServiceNow Support for creating a Hi Ticket to reclaim database shards for the sys_attachment_doc table.

Upcoming actions will include warnings for: - License key expiring soon - Growing outside of licensed storage soon - Exceeded licensed storage

Attachment Rules Page

The Attachment Rules page provides simple UI for creating and configuring Attachment Rules records. Each rule can have specified table, condition, and external storage.

Create New Rule Button

This button allows user to create new rule. User has to provide values for these fields: - Name - Storage - Table - Condition (optional - if no condition is set, all attachments from selected table will be offloaded)

Attachment Rule List

Displays the list of all attachment rules.

The list is refreshed automatically when there are any changes on records. Additionally there is a refresh button on the page for manual refresh.

Transfer progress is refreshed by scheduled job every 10 minutes.

In each Attachment Rule card there are:

• Name of the rule
• Upload progress to external storage
• Enabled/Disabled switch - defines if the rule is currently active. If rule is disabled, all attachments that were uploaded to external storage with that rule will be downloaded back to the ServiceNow instance when user clicks on "Run Mass Download Once" button on the Home page.
• Edit, Statistics buttons - open modal enabling user to edit rule, and shows statistics for the rule.
• Delete button - deleting attachment rule will not download attachments uploaded by it automatically. User needs to click 'Run Mass download once" button on the Home page.

Attachment Rule Modal

Modal can be accessed through Edit and Statistics buttons. The modal is organized with two tabs: Edit and Statistics.

Edit Tab

Displays transfer progress and all configurable fields for selected Attachment Rule record: - Name - Storage - Table - Condition - Enabled/Disabled switch

Changes to fields need to be saved with Save button. Enabled/Disabled switch works automatically without need of saving.

Statistics Tab

Displays statistics for selected Attachment Rule record: - Interactive attachment download triggers: Count of manually triggered downloads from the past 4 days - Average duration of download: Average of the last 4 days manually triggered download durations - Median duration of download: Median of the last 4 days manually triggered download durations - Number of interactive placeholder downloads (currently not in use): Count of the user triggered attachment placeholder downloads

Insights Page

The Insights page provides analytical views used to evaluate instance storage growth and savings. Attachment Sizes (GB) A column visualization displaying the number of attachments per day, based on the corresponding PA indicator. Available grouping options include: - Table - Attachment Size - Attachment Age

Clicking an individual column opens the standard KPI Details page, allowing users to review historical trends and analyze changes over time.

Settings Page

The Settings page allows users to configure essential application settings, manage properties, and handle storage records efficiently. Whenever a user updates a property, the changes are automatically saved to the backend.

Wizard Step Persistence

As users progress through the setup wizard, their current step is stored in the backend (specifically, in a record within the Wizard Steps table). If the page is reloaded, the user's progress is restored, ensuring a seamless experience.

The Settings page is organized into three tabs:

• Basic Setup
• Storage Setup
• Advanced Setup

Basic Setup Tab

The Basic Setup tab features a step-by-step wizard for configuring general application properties. The setup steps include:

• Delete Attachment Data
• Days to Keep Attachments
• Excluded Tables
• License Key

Storage Setup Tab

The Storage Setup tab displays a list of existing storage records. To create a new record, click the New button, which launches a guided wizard for setting up a storage record, including all required dependencies.

Process Overview:

• Begin by choosing a provider from the dropdown (Azure / AWS). Google Cloud provider is not yet supported.
• For Azure, select Authentication Type (Shared Key or OAuth Profile)
• Shared Key option unlocks Shared Key Options and Storage Settings wizard steps.
• Steps available for OAuth Profile authentication type: Application Registry, OAuth Entity Scope, OAuth Credentials, Connection Alias, Connection, Storage Settings.
• For AWS Provider the following steps becomes available: Connection Alias, AWS Credentials, Storage Settings.

Use the Next and Back buttons to navigate through each step of the wizard. Use Cancel button to discard all changes and return to the list view.

Finalization:

• Upon completing the Storage Settings step, the Create Storage Record button will appear. Click this button to create the storage record along with all necessary relationships.
• For OAuth Profile authentication, after the storage record is created, the final step is to generate an OAuth token. Click Open Record to access the OAuth 2.0 credentials record, then use the Get OAuth Token related link to initiate token creation.

Validate Connection:

Connection validation is performed automatically when the record is created or updated. Check the Status field to view the result: Online or Offline.

To manually verify the connection, select the row and click "Validate Connection."

Amazon S3 - Storage Provider (S3 stands for Simple Storage Service)

Advanced Setup Tab

Contains additional list of properties for configuring the application.

Attachment Rule Table

Name: x_gvs_offloader_rule

Attachment rules replace the previously used whitelist condition that was set up through the Admin page of the old Dashboard. Attachment rules can be accessed and set up through the new Workspace by going the SnowMirror Drive > Workspace > Attachment Rules.

Attachment Rules provide an easy way to set up different conditions to be able to upload attachments to different Storage Locations, which can be specified for each rule.

The components

• Active rule means that every attachment that is contained by this rule should be uploaded
• Deactivated rule means that the upload for this rule is inactive and no attachment that belong to this rule will be uploaded, unless they belong to a different active rule
• In case attachments that belong to a rule have already been uploaded, if the rule gets deactivated these attachments will be downloaded next time when user triggers mass download on the Home page, unless they belong to a different active rule Rules directly on the sys_attachment table

If the table specified for a rule is set to the sys_attachment table or left empty the following cases will be true

• The user can specify a condition using the fields of the sys_attachment table
• Example: [table_name IS incident]
• If no condition has been provided a "default" condition is added as an additional safety check
• This default condition is built from the PROTECTED TABLES + NUMBER OF DAYS TO KEEP THE ATTACHMENTS ON THE INSTANCE
• The above properties can be found and adjusted in the SnowMirror Drive > Workspace > Settings > Advanced Setup menu Rules on other tables

These rules mean that we can specify conditions on any table and based on the conditions the attachment that are stored on the found records of this table will be uploaded

If the table specified for a rule is not empty and not set the sys_attachment the following cases will be true

• The user can specify a condition using the fields of the given table
• Example: If the table is incident we can use a condition like [category IS Software]
• For the above example, the rule will find all the incident records that satisfy the given condition and upload all attachment that are on these records
• If no condition has been provided, this will mean that all attachments that are on that table should be uploaded Other Attachment Rule Notes

An attachment even if it is part of multiple different rules can only be uploaded to 1 Storage Provider

If 2 rules collide the stronger one always wins and the attachments will be uploaded to the Storage specified in that rule

Example for the above:

• We have rule A with Storage S1, table is sys_attachment and condition is table_name=incident
• We have rule B with Storage S2, table is incident and condition is category=Software
• If both rules are active all the attachments including the ones that belong to rule B will be uploaded to Storage S1
• Same as the above if only rule A is active
• If only rule B is active, only the attachments that satisfy rule B will be uploaded to Storage S2
• If none of the rules are active, nothing will be uploaded and if anything has already been uploaded it should be redownloaded

Storage Table

Name: x_gvs_offloader_storage

Related Property Name: x_gvs_offloader.storage_name

Allows the user to setup and manage several different storages on one instance. The above mentioned related property contains the name of a Storage record that will be used for all uploads, downloads, etc.

The components

PA Data Collector Jobs

There are several different Scheduled Data Collection jobs in the application each with it's own use case. Most jobs run automatically, while there are a few that require manual starting. All the jobs use the same "Run as" user, a manually created user called "Offloader admin" that acts as a placeholder.

[PA] Data offloader (Historical)

• Historical data collection designed to run once when the application is freshly installed on a new instance.
• Used to gather the scores for the indicator [Attachment size in MB]
• The field "Relative start" defines how long it should go back, this can be adjusted
• Example: 30 means it will go back 30 days into the past and collects data attachment sizes with all the breakdowns for each given day

[PA] Data offloader (Test)

• Data collection job used for testing indicators
• Indicators can be freely removed or added to it

[PA] Data offloader (Daily)

• Daily data collection job that runs at a specific time defined in the "Time" field. Unlike the historical data collection, this only collects scores for the last 1 day but includes all indicators of the application for that.
• Assigned indicators should not be changed as it can severely affect the Workspace
• Runs automatically every day at a specific time

[PA] Data offloader (Regular)

• Updates some of the indicators used on the workspace that calculate actual savings
• Assigned indicators should not be changed as it can severely affect the Workspace
• Runs automatically every 10 minutes

Design

The application is a scoped application, with the possibility of distribution through both update sets and the application studio.

The main workflow

• The administrator configures the application in basic setup tab in workspace (how many days to keep attachments, set excluded tables, enter license key)
• The administrators sets up the external storage via storage setup tab in workspace
• The administrator creates Attachment Rule records defining what attachments will be uploaded to which external storage
• The administrator enables Mass Upload with a switch on workspace homepage
• Script uploads all attachments matching the conditions from Attachment Rules + excluded tables + days to keep attachments
• This will be a scheduled job running between start and end times defined in advanced setup tab in workspace
• To download uploaded attachments in bulk, user must disable or delete Attachment Rule that required attachments were uploaded by and click Run Mass Download Once button in workspace Homepage
• If user accesses the sys_attachment record from anywhere via the sys_attachment.do?sys_id= processor, there is a business rule running on the sys_attachment table that:
• Checks if its sys_attachment_doc records are there (more than 0).
  • Yes => Does nothing and lets the processor follow up with its usual attachment construction and download.
  • No =>
  • It synchronously puts a gif file in the sys_attachment_doc table with the sys_id of the sys_attachment that the user is trying to download.
  • It changes the Content type on the sys_attachment table so the user sees he is downloading a gif instead of the original file.
  • It triggers an asynchronous job that starts downloading the missing sys_attachment_doc parts from external storage. The query will ask for downloading the whole folder on the external storage side.
  • Once all parts are downloaded, it changes the sys_id in the created sys_attachment_docrecords and removes the placeholder gif parts from there.
  • It changes the Content type back to the original one.

The double queue structure

There are 2 custom queues in the application that handle the automatic and manual attachment upload/download separately. Both queues use the same events.

High priority queue:

• Handles manual attachment download events
• Manual uploads don't have an event created
• This queue checks for new events every second to ensure faster processing
• The number of workers can be customized in the Workspace / Settings / Advanced Setup

Low priority queue:

• Handles events created by the Mass Upload and Mass Download functions
• Checks for events in a bigger interval (OOB is 30sec) as these events contain a large array of attachment sys_ids
• The number of workers can be customized in Workspace / Settings / Advanced Setup
• Max size of this queue can be customized in the Advanced Setup, this means that if the queue fills up no new events will be added until there is enough space

Mass upload/download architecture

There are 2 different Scheduled job for handling the Mass Download and Mass Upload processes separately, but structurally they are almost the same. The Scheduled Job that handles the Mass Upload is called "GVS_OFFLOAD Upload Attachments" and the one that handles the Mass Download is called "GVS_OFFLOAD Download Attachments". While Mass Upload process runs automatically based on the schedule, Mass Download job runs on demand when user activates it by clicking on the "Run Mass Download Once" button from the Home page of the workspace.

Main structure

• Both jobs utilize the same message queue (a custom one with a configurable number of parallel workers) to handle the Mass Upload/Download of the attachments.
• Events are created for both upload and download in the "sysevent" table under the "GVS_OFFLOAD Low Priority Queue" and the first parameter (parm1) of these events contain at least one sys_attachment sys_id or a maximum number specified in the "x_gvs_offloader.batch_size" configuration property
• The second parameter (parm2) contains the sys_id of the Storage record in which the event is uploading to or downloading from
• The "name" of each event shows if it's either an upload or download of the given attachments
• When an event is created is starts in state "ready", during processing it gets queued up and once it finishes the state will be either "processed" (normally means successful) or "error"
• The Mass Upload runs periodically after it is activated and until it is disabled through the Workspace. Every minute it runs and checks for attachments to upload based on the Attachment Rules
• Mass Upload job works between a timeframe every day, which is specified in Advanced Setup
• Once out of this timeframe the Scheduled job will stay active, it will finish all the events that are being processed at the time, but it won't start creating new events. All events that are still in ready state will be automatically deleted from the queue.
• The Mass Download runs only on demand and deactivates automatically once everything has been downloaded back based on the Attachment Rules
• Both Scheduled jobs requires an active and valid license on the instance to function

Mass upload process

• Starts after the process has been enabled on the Workspace
• The logic will iterate through all active Attachment Rules in descending order based on their size field
• For each rule based on the table_name and condition fields it will consctuct an encoded query that will be used to find and upload all attachments to the storage record paired with the rule
• The cases can happen:
• For rules that are on the sys_attachment table, the condition query is extended with a blacklist condition if it's empty or not
• This blacklist condition contains a list of table names that are system tables or were added as manually excluded tables during the basic setup
• It also contains the setting of how long attachments should stay on the instance
• The overall condition will be the following: table_name=sys_attachment, condition + table_nameNOT IN {list of excluded tables} + days to keep attachments on the instance
• For rules that are not on the sys_attachment table, they will not get the above condition and onyl their condition field will be used
• Attachments that are already uploaded are automatically excluded in the process
• After using the query, each attachment is checked if they are already in the queue and being processed by another method (like download), if yes they are excluded
• The script iterates through all 'eligible' attachments and adds their sys_id into an array
• When this array reaches the length specified in the Advanced Settings (OOB is 100), it will queue up the whole batch of ids together for the upload
• Batches of attachment ids are added into the queue until there are no more or the queue is full
• The workers in the queue will automatically pick up the events in an asynchronous way and process the batches of attacment ids in a synchronous way

Mass download process

• Starts after the related button on the Workspace has been clicked
• The logic will iterate through all online Storage records
• For each Storage record it will find all the active Attachment Rules and does the following:
• It will separate each active rules based on the table_name and condition fields and merge the rules wherever it can
• The 3 categories it will use to merge is:
  • All active rules where table_nameISEMPTY or table_name=sys_attachment
  • All active rules where table_nameISNOTEMPTY and table_name!=sys_attachment
  • All active rules where table_nameISNOTEMPTY and table_name!=sys_attachment and conditionISNOTEMPTY
• The result at the end will be a combination of a table_name and a merged condition for that table
• This is necessary as Attachment Rules can have table specific conditions with fields that don't exist on the sys_attachment table
• If a rule has a table that is not sys_attachment but no condition, this means that the whole table can be upload -> the rule can be converted to be on the sys_attachment table
  • Example: table: incident and condition: - -> this is the same as saying table_name=incident on the sys_attachment table, all incident records can be uploaded
• Table specific rules with specific conditions will be handled separately thorugh join queries while the first 2 categories will be merged to create a query on the sys_attachment table
• All queries constructed here are logically negated in order to use for the download -> goal is to download attachments that don't match any of the active rules
• An example scenario would be the following:
  • Rule 1: table: sys_attachment, condition: table_name=incident
  • Rule 2: table: kb_knowledge, condition: -
  • Rule 3: table: problem, condition: priority=1
  • Result: It will create a object that contains a table_name as the key and condition as the value
  • { sys_attachment: table_name!=incident^table_name!=kb_knowledge, problem: priority!=1}
  • Notice how the conditions were negated logically and merged for each rule, for upload there is an OR condition between the rules as we iterate between them, after the logical negation this will become an AND
• The script iterates through an object like above and runs the mass download for each table
• Attachments that are not uploaded are automatically excluded in the process
• After using the query, each attachment is checked if they are already in the queue and being processed by another method (like upload), if yes they are excluded
• The script iterates through all 'eligible' attachments and adds their sys_id into an array
• When this array reaches the length specified in the Advanced Settings (OOB is 100), it will queue up the whole batch of ids together for the download
• Batches of attachment ids are added into the queue until there are no more or the queue is full
• The workers in the queue will automatically pick up the events in an asynchronous way and process the batches of attacment ids in a synchronous way
• The following edge cases are covered:
• If there are no online Storages, nothing will be downloaded
• If there are no active rules for a Storage, everything will be downloaded back from the Storage
• If there is 1 rule but it's not on the sys_attachment table an extra query will be created there automatically to make sure all attachments are covered in the download process and not missed
• A list of table names is generated from all the rules where the table name is not sys_attachment, this list is used when downloading data from the sys_attachment table and ensures that records on those tables will be skipped by that run. In the above example this would mean that the table problem would be on the list and would create an extra query table_nameNOT INproblem
• The script does the above logic to avoid recursive download/upload of attachments as table specific rules are prioritised over more generic rules that are on the sys_attachment table

Interactive download (aka manually triggered download)

Interactive download is triggered when a user opens a record that has attachments already stored in the external storage provider. During this process, the [External Storage ☁️ file name prexix is replaced with [Syncing 🔄 prefix while the files are being downloaded back into the ServiceNow instance.

It works through the Business Rule GVS_OFFLOAD Download attachment, where the encoded query is parsed: If a sys_id is present in the query, only that specific attachment is downloaded. If a table_sys_id is present, all attachments associated with that record are downloaded.

That’s why, when you open the sys_attachment list and click "Show Matching" on the 'Table sys ID' column, it triggers the download. This happens because the encoded query is formed like this: table_sys_id=0cb614453b49cb50f20e9b34c3e45a27

Attachment upload file consolidation

• There will be a header consisting of 2 parts, first is length of the header, second contains attachment and doc part related data

• For a 30MB file coming from 20 attachment parts

  • First header part (fixed length, just one numeric parameter) will contain:
  • Length of the second header
  • Example: 0000000192
  • Second header part:
  • The original byte size of the attachment
  • The original content type of the attachment
  • The original file extension of the attachment
  • The position of the doc parts together with their length
  • Example: { attachmentSize: 123456, contentType: image/png, fileExtension: .png 0: 123 1: 456 ... }

• The second part of the header is base64 encoded while the first part is not encoded. Both header parts are attached together and will be attached to the beginning of the first blob that is uploaded

• Once the header is constructed, the algorithm will iterate through the doc parts in Servicenow and attach them together instead of uploading each doc part 1 by 1. This greatly improves performance. These doc parts are base64 encoded by default within Servicenow and they are taken in the order of their order in the sys_attachment_doc table.
• Doc parts are concatenated together until
• There are no more doc parts left
• The maximum string length has been reached
• The maximum string length in Servicenow is 16777216 and there is a 4:3 base64 character to byte conversion. This means that every 4 base64 character equals to 3 bytes in size, so the maximum size of the uploaded blob parts is around 15MB
• If the string limit is reached during the concatenation, a new blob part will be created, but only the first part with the blob name "0" will contain the header.
• After every "put blob" upload request, there will be a status code and MD5 check to ensure that the file has not been corrupted. The expected status code is 201. In case of failure, a retry mechanism will try to upload the blob for another X amount of time, this can be configured (default is 3)
• If any of the blobs have failed to upload correctly, the whole upload process will be cancelled.

Multiple Storage Providers support.

Storage Provider Table (x_gvs_offloader_storage) Purpose The Storage Provider table defines and manages the configuration of external storage endpoints used by the SnowMirror Drive app. Each record represents a storage provider (for example, AWS S3, Azure Blob Storage, or Google Cloud Storage) that can be used to store, retrieve, and manage offloaded attachments. This table acts as the central configuration source that determines: which storage providers are available, how ServiceNow connects to them, and which provider is actively used during offload, download, or delete operations.

How the table is used by the system At runtime, the application: Reads the Storage Provider record. Identifies the provider type and connection details. Dynamically initializes the corresponding storage connector.

Script Includes GVS_OFFLOADER_RestConnectorBase An abstract base class that implements only the common methods shared across storage providers. All provider‑specific methods must remain unimplemented and should throw an exception if called directly.

GVS_OFFLOADER_AzureRestConnector Inherits from GVS_OFFLOADER_RestConnectorBase and provides concrete implementations for communicating with the Azure storage provider.

GVS_OFFLOADER_S3RestConnector Inherits from GVS_OFFLOADER_RestConnectorBase and provides concrete implementations for communicating with the Amazon S3 storage provider.

Additional classes to support AWS S3 signing process awsSignatureCalculator, CryptoJS

The key aspect is how the system determines which storage provider to use. This logic is handled in GVS_OFFLOAD_GlobalUtils.initialize Note: GVS_OFFLOAD_GlobalUtils methods that require STORAGE_PROVIDER to be initialized call the _requireStorageProvider guard function to ensure the value is set.

STORAGE_PROVIDER is initialized dynamically at runtime based on the type of the configured Storage Provider record.

During initialization, the system determines the storage provider type (for example, AWS, Azure) and uses a factory (GVS_OFFLOAD_StorageProviderRestConnectorFactory script include) to instantiate the corresponding REST connector implementation for that provider.

Extending the System to Support a New Storage Provider

Follow these steps to enable a new storage provider in the application.

Let’s assume we want to add Google Cloud as a new storage provider. - Create a Script Include, for example GVS_OFFLOADER_GoogleCloudRestConnector, extending GVS_OFFLOADER_RestConnectorBase. - Add a new option to the Provider field on the Storage Provider table (x_gvs_offloader_storage), using "google" as the technical value. - Update the create() method of the GVS_OFFLOAD_StorageProviderRestConnectorFactory class to map the new Provider value to the new Script Include by adding a case statement such as: ... case "google": return new x_gvs_offloader.GVS_OFFLOAD_GoogleCloudRestConnector(storageSysId); ... - Create a new Storage Provider record (x_gvs_offloader_storage), and configure the connection details and credentials. The connection will be validated automatically when the record is saved (see the Status field).

Amazon S3 - request signing

All Amazon S3 requests must be authenticated using an AWS SigV4generated signature. In the SnowMirror Drive application, this is implemented in awsSignatureCalculator().getSignature(). You can see this method invoked in every function that sends requests to S3.

For signature generation and for Authorization request header Secret Access Key and Access Key are required. They are obtained from AWS S3 by admin. In ServiceNow, the keys are stored as AWS Credentials.

Attachment Encryption with Field Encryption Enterprise

SnowMirror Drive works with standard ServiceNow attachment APIs. To protect sensitive data at rest and ensure regulatory compliance, attachment content can be encrypted using Field Encryption Enterprise. This section describes how to encrypt attachments (for example, on the Incident table) and the key considerations when SnowMirror Drive scheduled jobs process encrypted data.

1. Overview of Field Encryption Enterprise

Field Encryption Enterprise is an advanced ServiceNow security feature that uses the Key Management Framework (KMF) to encrypt fields and attachments at the application layer. Encryption and decryption are handled transparently by the platform based on access policies and cryptographic modules.

Key benefits when used with SnowMirror Drive:

• Protects sensitive attachment data (PII, PHI, financial documents) stored in ServiceNow before offloading.
• Ensures that only authorized users and system processes can decrypt attachments.
• Supports customer‑supplied keys (CSK) and enterprise key lifecycle management (rotation, revocation).
• Attachments remain encrypted in ServiceNow while SnowMirror Drive offloads and rehydrates them using standard APIs.

Official ServiceNow documentation:

• Field Encryption Enterprise overview https://www.servicenow.com/docs/r/platform-security/now-platform-encryption.html
• Attachment encryption walkthrough https://www.servicenow.com/docs/r/platform-security/attachment-encryption-walkthrough.html

2. Prerequisites and Required Roles

Before configuring attachment encryption, ensure the following prerequisites are met:

Plugins and licensing

Field Encryption Enterprise must be licensed and activated. Key Management Framework (KMF) must be available on the instance.

Required roles for setup

The user performing the configuration must have:

• admin
• sn_kmf.cryptographic_manager (to create and manage cryptographic modules)
• security_admin (role escalation may be required for encryption configuration)

These roles are required to define encryption modules, access policies, and encrypted field configurations.

3. Cryptographic Modules and Module Access Policies

A Cryptographic Module defines how data is encrypted (algorithm, key source, usage). For attachments, this module is referenced by the Encrypted Field Configuration. Important considerations for SnowMirror Drive:

SnowMirror Drive scheduled jobs run as a system‑level user. That user must be explicitly granted access to the cryptographic module via a Module Access Policy, otherwise attachment encryption/decryption will fail during upload or download jobs.

Best practices

Create a dedicated module (for example: SnowMirror_Attachment_Module). Define a Module Access Policy that grants:

Read/Decrypt access to:

• SnowMirror Drive run‑as user (System)
• Required admin roles

Validate access before enabling mass encryption.

Module access policies are mandatory for system processes acting on encrypted attachments.

4. Encrypted Field Configuration (EFC) for Attachments

The Encrypted Field Configuration defines what gets encrypted and under which conditions. Example: Encrypt attachments on the Incident table

Navigation: System Security → Field Encryption → Encrypted Field Configurations

Create a new Encrypted Field Configuration with the following values:

• Type: Attachment
• Table: Incident
• Active: ✔ (enable only when ready to encrypt)
• Crypto module: Select the cryptographic module created earlier
• Encrypt by default: Optional, based on requirements

Once active:

New attachments are encrypted automatically. Access enforcement is handled through Module Access Policies.

Encrypted Row Configurations (optional)

Encrypted Row Configurations can be used to encrypt attachments only when specific record conditions are met (for example: incident.category = Security) and allow fine‑grained control instead of encrypting all attachments on a table. This is useful when SnowMirror Drive is configured to offload only certain attachment sets.

5. Mass Encryption and Mass Decryption Jobs

When enabling attachment encryption on existing data, historical attachments are not encrypted automatically. ServiceNow provides background jobs for data migration.

Mass Encryption

Encrypts existing unencrypted attachments based on the Encrypted Field Configuration. Required after activating attachment encryption for an existing table (such as Incident).

Mass Decryption

Decrypts attachments if encryption is being disabled or reconfigured. Useful during testing, rollback, or module changes.

Navigation:

System Security → Security Jobs → Create New → Mass Encryption Job

Operational guidance

Run Mass Encryption during low‑usage windows. Monitor job progress and logs before enabling SnowMirror Drive mass upload jobs. Ensure SnowMirror Drive scheduled jobs remain disabled until encryption completes to avoid partial access issues.

Mass encryption and decryption are required to keep attachment state consistent across existing and new records.

Notes for SnowMirror Drive Operations

• SnowMirror Drive uses standard attachment APIs; encryption and decryption are handled transparently by ServiceNow.
• Ensure the SnowMirror Drive system user always retains cryptographic module access.
• Changes to encryption configuration should be coordinated with SnowMirror Drive mass upload schedules.
• Test attachment upload and re‑download scenarios after enabling encryption.

Automated Testing with Robot Framework

Installation and first run

1. Install Robot, Install Python https://robotframework.org/?tab=1#getting-started
2. Download the project files into a suitable folder (log in to git inside your IDE with your token and clone the repository)
3. Put attachments into the upload folder – it's in the project location
4. Define ${USERNAME} and ${PWD} for the instance in CommonKeywords.robot
5. Define ${INSTANCE_NAME} variable in CommonKeywords.robot
6. Ensure querySelectorDeep.js file exists in the project folder. It was created by GV team to handle shadow DOM.

Source files

Link to git https://gitlab.guidevision.cz

Note, this is not the final version. There is a lot of room for improvement and refactoring.

Project folder

It contains the following information files:

• run_robot_parallel_v4.py - this orchestration script drives the core loop scenario by executing the associated Robot Framework files. For detailed behavior and execution logic, refer to the comments within the robot files.
• CommonKeywords.robot
• test_setup.robot
• create_incident.robot
• assert_file_integrity.robot
• test_cleanup.robot
• querySelectorDeep.js - a helper script for selecting HTML elements located within a web page's shadow DOM.
• incident_numbers.json - storage file for mapping attachment filenames to incident numbers. Updated during the scenario execution. How it works

What you need to run is the orchestration file run_robot_parallel_v4.py. Below is an overview of the main steps it does:

(Make sure your terminal is in the project root folder, then paste "python run_robot_parallel_v4.py" in it and hit enter)

• Runs test_setup.robot and sets up prerequisites for testing:
• Clears the script generated Downloads folder (Chrome uses a custom downloads folder for testing)
• Updates system properties (days to keep and offload condition)
• Enables Mass Upload.
• Runs create_incident.robot for each file in parallel.
• Extracts and stores incident numbers in incident_numbers.json.
• Runs assert_file_integrity.robot for each file, passing the correct incident number.
• Runs test_cleanup.robot and restores the changed system properties.
• Generate unique output files for each Robot Framework execution.
• Collect all output files.
• Merge them into a single report using rebot.
• Places all Reports and Output files into the script generated Resources folder.

Manual Installation through Update Sets

Each release has 2 different version PATCH and FRESH INSTALL

Fresh Install:

• Should be used when the application is not yet installed on the instance
• Contains all the changes for the specific version, contains all system properties, but some have empty values that needs to be filled as part of the installation
• Ensures that the application can be installed on a brand new instance, contains everything

Installation steps

Go to "Retrieved Update Sets" > Click on "Import Update Set from XML" > Select the XML file that contains the version you want to install

1. Import the XML file
   • If the XML has been imported successfully, open the record with it's name that is on state "Loaded"
   • Click "Preview Update Set Batch"
   • If there are any conflicts, just click on "Accept remote update" for each
   • Once there are no conflicts click "Commit Update Set Batch"
2. The fix script called GVS_OFFLOAD Setup Application needs to be executed as it initializes some of the parts of the application
   • Use the option "Proceed in Background" while running it
   • This script can take up to a few minutes to finish.
3. Swap to "Data offloader" scoped application, needed to edit settings and run the fix scripts without any issue
4. License key is needed for application. Please reach out to SnowMirror Drive support to acquire a license key.
5. Go to "SnowMirror Drive > Workspace > Settings" and setup the application
6. In the "Basic Setup" menu go through each setting
7. In the "Storage Setup" menu
   • Click on "New" and setup at least 1 Azure or AWS remote storage for the application by going through the steps
8. (Optional) In the "Advanced Setup" menu go through the settings and change them if needed
9. Go to "SnowMirror Drive > Workspace > Attachment rules" to setup and activate the rules
10. Go to "SnowMirror Drive > Workspace > Home" under the "Transfer status" tab enable the automatic upload of attachments

Patch version:

• Should be used when the application is already installed on the instance
• Contains all the changes for the specific version, but doesn't overwrite anything specific for the instance
• Only includes the 2 XML for the scoped and global applications

Installation steps

Go to "Retrieved Update Sets" > Click on "Import Update Set from XML" > Select the XML file that contains the version you want to upload and upload it

1. Import the Global XML file
   • If the XML has been imported successfully, open the record with it's name that is on state "Loaded"
   • Preview the record
   • If there are any conflicts, just click on "Accept remote update" for each
   • Once there are no conflicts commit the update set
2. Import the Scoped XML file
   • If the XML has been imported successfully, open the record with it's name that is on state "Loaded"
   • Preview the record
   • If there are any conflicts, just click on "Accept remote update" for each
   • Once there are no conflicts commit the update set
3. (Optional) The fix script called "GVS_OFFLOAD Setup Application" can be executed as it runs some of the jobs which are responsible for refreshing the data on the Workspace

Amazon S3 setup

Create S3 Bucket Log in to the AWS Management Console. Navigate to S3 (Simple Storage Service). Click Create bucket.

Configure Bucket Settings Bucket name Enter a globally unique name (e.g. servicenow-offloader-bucket) Region Select the region where your bucket will be hosted (e.g. eu-central-1)

Note, the selected region must match the region configured in ServiceNow. A mismatch will result in authentication (signature) errors.

Object Ownership Set Object Ownership to: ACLs disabled (recommended)

Block Public Access Keep all Block Public Access settings enabled: Block all public access

Bucket Versioning (Optional) Enable Versioning if required.

Default Encryption (Recommended) Enable server-side encryption: Encryption type: Amazon S3 managed keys (SSE-S3)

Create Bucket Review the configuration Click Create bucket

Create IAM User Log in to the AWS Management Console. Navigate to IAM (Identity and Access Management). In the left navigation pane, select Users. Create a new user (e.g. servicenow-s3-user) Choose Attach policies directly (skip choosing policy for now), click Next. On Review and create page click Create user.

Create S3 Access Policy In IAM, navigate to Policies. Click Create policy. Open the JSON tab. Paste the following policy and replace your-bucket-name with your actual bucket name: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:"], "Resource": [ "arn:aws:s3:::your-bucket-name", "arn:aws:s3:::your-bucket-name/" ] } ] } Provide a name (e.g. S3FullAccessToBucket). Click Create policy.

Attach Policy to User Return to IAM -> Users. Select the previously created user. Open the Permissions tab. Click Add permissions. Select Attach policies directly. Search for the policy created in the previous step. Select it and click Add permissions.

Generate Access Keys Open the user in IAM. Go to the Security credentials tab. Scroll to Access keys. Click Create access key. Select Application running outside AWS. Click Create access key. Save the following values securely: Access Key ID Secret Access Key The secret key is only displayed once and cannot be retrieved later.

Integration Notes The bucket name must match exactly the value configured in ServiceNow. The IAM user created earlier must have access to: the bucket (arn:aws:s3:::bucket-name) all objects (arn:aws:s3:::bucket-name/*) No additional bucket policy is required if IAM permissions are configured correctly.