# Purpose The SnowMirror Drive application moves attachments from ServiceNow into alternative cloud storage to save database size and costs on SN side. # SnowMirror Drive - Application Menu ### The contents of the application menu: - **Workspace** - User friendly UI where the users of the application can view statistics, live data and an interface for admins to control the application. - **Storage Providers** - **Attachment Rules** - **PA Data Collector Jobs** - Scheduled Data Collection jobs that setup the scores of different metrics and graphs displayed on the Workspace. - **Required Actions** - **Log** - Error/info log menu for administrators - **Groups** - Admin Group - Group used for distributing administrator privileges for the whole application. - User Group - Group user for distributing user privileges for the whole application. ## SnowMirror Drive Workspace The SnowMirror Drive Workspace is available from the Workspaces menu and from the application navigator under: SnowMirror Drive → Workspace ### Home Page The Home page provides a comprehensive overview of upload and download activity, storage utilization, key Performance Analytics (PA) metrics and any required configuration actions. The page is divided into clearly defined sections for quick access and visibility. #### Information about your instance This section displays key Performance Analytics (PA) scores, helping you evaluate your application’s efficiency and your instance's total capacity. Metrics included are: - Actual Savings (GB / $): Actual storage and cost savings. - Potential Savings (GB / $): Future maximum potential storage and cost savings. - Instance DB Size (GB): Total size of your instance database. These insights allow you to monitor profitability and assess current resource utilization. #### Transfer status A toggle control that enables or disables the mass transfer process (bulk upload and download of attachments). #### Transfer progress View the current state of scheduled jobs for bulk uploading and downloading attachments. Monitor ongoing file uploads with this card: - Progress Bar: Displays transferred versus total attachments to upload or download. - Estimated Time: Shows remaining time for completion. - Auto-Refresh: Progress bar updates automatically every 15 seconds via script for up-to-date information. - When transfers are paused, the next configured upload or download window is displayed. #### Storages This section lists all configured storage records, with an at-a-glance indicator of their status: - Green Dot: Storage is online. - Red Dot: Storage is offline. - No Storage Configured: If no storage is set up, a relevant message is displayed. Each storage name acts as a clickable link that navigates directly to the storage record form for easy management and troubleshooting. #### Required Actions This area highlights necessary configuration steps to ensure your application is set up and running optimally, particularly for first-time use. Required actions are managed through records in the x_gvs_offloader_required_action table, which contain scripts to dynamically determine their visibility. - Severity Indication: Each action card uses a colored border to reflect its severity. - Action Button: Each card includes a button that will either link to the appropriate page or execute a script to activate a feature or resolve an issue. Available Actions include: - Basic setup required: Alerts if basic properties on the Settings page are incomplete. The button navigates to the last step of the Basic Setup wizard. - Historical data collection didn’t run: Indicates if the [PA] Data Offloader (Historical) scheduled job has not run; the button navigates to the scheduled job. - License key expired! Shown if the application license key has expired; the button takes you to Settings > Basic Setup to update the license. - Mass upload not activated: Warns if mass upload is not activated; the “Activate” action runs a script to enable scheduled jobs. - No rules defined: Alerts if no active attachment rules are set; the button navigates to the Attachment Rules page. - Storage connection failing: Displays if at least one storage provider is offline. - No storage configured: Appears if there are no storage records; the button navigates to Settings > Storage Setup. - Automated table compaction will not run: Indicates the automatic table compaction size limit has been reached; the fix action links to ServiceNow Support for creating a Hi Ticket to reclaim database shards for the sys_attachment_doc table. Upcoming actions will include warnings for: - License key expiring soon - Growing outside of licensed storage soon - Exceeded licensed storage ### Attachment Rules Page The Attachment Rules page provides simple UI for creating and configuring Attachment Rules records. Each rule can have specified table, condition, and external storage. #### Create New Rule Button This button allows user to create new rule. User has to provide values for these fields: - Name - Storage - Table - Condition (optional - if no condition is set, all attachments from selected table will be offloaded) #### Attachment Rule List Displays the list of all attachment rules. The list is refreshed automatically when there are any changes on records. Additionally there is a refresh button on the page for manual refresh. Transfer progress is refreshed by scheduled job every 10 minutes. In each Attachment Rule card there are: - Name of the rule - Upload progress to external storage - Enabled/Disabled switch - defines if the rule is currently active. If rule is disabled, all attachments that were uploaded to external storage with that rule will be downloaded back to the ServiceNow instance - `Edit`, `Statistics`, `Delete` buttons #### Attachment Rule Modal Modal can be accessed through `Edit` and `Statistics` buttons. The modal is organized with two tabs: Edit and Statistics. ##### Edit Tab Displays transfer progress and all configurable fields for selected Attachment Rule record: - Name - Storage - Table - Condition - Enabled/Disabled switch Changes to fields need to be saved with `Save` button. Enabled/Disabled switch works automatically without need of saving. ##### Statistics Tab Displays statistics for selected Attachment Rule record: - Interactive attachment download triggers: Count of manually triggered downloads from the past 4 days - Average duration of download: Average of the last 4 days manually triggered download durations - Median duration of download: Median of the last 4 days manually triggered download durations - Number of interactive placeholder downloads (currently not in use): Count of the user triggered attachment placeholder downloads ### Insights Page The Insights page provides analytical views used to evaluate instance storage growth and savings. Attachment Sizes (GB) A column visualization displaying the number of attachments per day, based on the corresponding PA indicator. Available grouping options include: - Table - Attachment Size - Attachment Age Clicking an individual column opens the standard KPI Details page, allowing users to review historical trends and analyze changes over time. ### Settings Page The Settings page allows users to configure essential application settings, manage properties, and handle storage records efficiently. Whenever a user updates a property, the changes are automatically saved to the backend. #### Wizard Step Persistence As users progress through the setup wizard, their current step is stored in the backend (specifically, in a record within the Wizard Steps table). If the page is reloaded, the user's progress is restored, ensuring a seamless experience. The Settings page is organized into three tabs: - Basic Setup - Storage Setup - Advanced Setup #### Basic Setup Tab **The Basic Setup** tab features a step-by-step wizard for configuring general application properties. The setup steps include: - Delete Attachment Data - Days to Keep Attachments - Excluded Tables - License Key | Name | Type | Purpose | | --- | --- | --- | | x_gvs_offloader.can_delete_attachment | Boolean | Decides if sys_attachment_doc records can be deleted | | x_gvs_offloader.data_offloader.attachments.days_to_keep | Integer | Number of days attachments should be kept on the instance and should not be uploaded | | x_gvs_offloader.excluded_tables | String | Blacklist query, tables where attachments should not be uploaded from | | x_gvs_offloader.license_data | String | SnowMirror Drive application license key | #### Storage Setup Tab **The Storage Setup** tab displays a list of existing storage records. To create a new record, click the New button, which launches a guided wizard for setting up a storage record, including all required dependencies. ##### Process Overview: - Begin by choosing a provider from the dropdown (Azure / AWS). Google Cloud provider is not yet supported. - For Azure, select Authentication Type (Shared Key or OAuth Profile) - Shared Key option unlocks Shared Key Options and Storage Settings wizard steps. - Steps available for OAuth Profile authentication type: Application Registry, OAuth Entity Scope, OAuth Credentials, Connection Alias, Connection, Storage Settings. - For AWS Provider the following steps becomes available: Connection Alias, AWS Credentials, Storage Settings. Use the Next and Back buttons to navigate through each step of the wizard. Use Cancel button to discard all changes and return to the list view. ##### Finalization: - Upon completing the Storage Settings step, the Create Storage Record button will appear. Click this button to create the storage record along with all necessary relationships. - For OAuth Profile authentication, after the storage record is created, the final step is to generate an OAuth token. Click Open Record to access the OAuth 2.0 credentials record, then use the Get OAuth Token related link to initiate token creation. ##### Validate Connection: The Validate Connection button tests communication with the external storage provider. To use it: - Select a storage record from the list. - Click Validate Connection. Possible results: “Connection successful” – The storage is reachable and properly configured. “Connection failed” – The connection attempt was unsuccessful. Amazon S3 - Storage Provider (S3 stands for Simple Storage Service) #### Advanced Setup Tab Contains additional list of properties for configuring the application. | Name | Type | Purpose | | --- | --- | --- | | x_gvs_offloader.permitted_tables | String | List of table names that are OOB excluded but their records should be uploaded nonetheless | | x_gvs_offloader.number_of_retries | Integer | Number of retries for failed attachment upload or download | | x_gvs_offloader.high_priority_queue_worker_count | Integer | Number of workers used per node in the high priority queue | | x_gvs_offloader.low_priority_queue_worker_count | Integer | Number of workers used per node in the low priority queue | | x_gvs_offloader.upload_queue_limit | Integer | Maximum number of events in the low priority queue | | x_gvs_offloader.batch_size | Integer | Number of records per event (100 is recommended) in the low priority queue | | x_gvs_offloader.queue_update_timer| Integer | Number of minutes before the Scheduled Job refills the low‑priority queue | | x_gvs_offloader.upload_start_time | Date/Time | Starting time of the Mass Upload Scheduled Job | | x_gvs_offloader.upload_end_time | Date/Time | Ending time of the Mass Upload Scheduled Job | | x_gvs_offloader.download_start_time | Date/Time | Starting time of the Mass Download Scheduled Job | | x_gvs_offloader.download_end_time | Date/Time | Ending time of the Mass Download Scheduled Job | | x_gvs_offloader.mid_server_name | String | Name of the MID Server used for all processes | | x_gvs_offloader.log_level | String | Minimum log level for Data Offloader scripts | | x_gvs_offloader.placeholder_id | String | Sys_id of the placeholder attachment used during the Azure download process | | x_gvs_offloader.archived_prefix | String | Prefix used to mark fully offloaded attachments. Do not change after data has been uploaded! | | x_gvs_offloader.syncing_prefix | String | Prefix used to mark attachments that are currently being downloaded | | x_gvs_offloader.high_priority_queue_name | String | Name of the custom queue for manual attachment processing | | x_gvs_offloader.low_priority_queue_name | String | Name of the custom queue for mass upload and download processing | | x_gvs_offloader.upload_job_name | String | Name of the Scheduled Job that manages the mass upload | | x_gvs_offloader.download_job_name | String | Name of the Scheduled Job that manages the mass download | ## Attachment Rule Table Name: x_gvs_offloader_rule Attachment rules replace the previously used whitelist condition that was set up through the Admin page of the old Dashboard. Attachment rules can be accessed and set up through the new Workspace by going the SnowMirror Drive > Workspace > Attachment Rules. Attachment Rules provide an easy way to set up different conditions to be able to upload attachments to different Storage Locations, which can be specified for each rule. #### The components | Name | Purpose | | --- | --- | | Name | The name of the Attachment Rule (only to improve the user experience) | | Storage | The Storage Provider record selected for the rule (attachment returned by this rule will be uploaded and downloaded from this location) | | Table | The table record that should be used to upload the attachments from (default one is sys_attachment) | | Condition | Condition builder where fields relate to the selected table | | ETA | The duration it should take for the rule to upload every attachment that it contains | | Size | The combined size of all the attachments that belong to this rule, includes the uploaded ones as well | | Uploaded Size | Size of attachments that belong to this rule and are offloaded to external storage | | Not Uploaded Size | Size of attachments that belong to this rule and are not offloaded to external storage | | Average Download Time | Average time of an attachment download calculated based on the past 4 days | | Median Download Time | Median of the attachment download times calculated based on the past 4 days | | Interactive Attachment Downloads | Number of manually started attachment downloads from the past 4 days | | Active | Specifies whether the rule is active or not | - Active rule means that every attachment that is contained by this rule should be uploaded - Deactivated rule means that the upload for this rule is inactive and no attachment that belong to this rule will be uploaded, unless they belong to a different active rule - In case attachments that belong to a rule have already been uploaded, if the rule gets deactivated these attachments will be automatically downloaded, unless they belong to a different active rule Rules directly on the sys_attachment table If the table specified for a rule is set to the sys_attachment table or left empty the following cases will be true - The user can specify a condition using the fields of the sys_attachment table - Example: [table_name IS incident] - If no condition has been provided a "default" condition is added as an additional safety check - This default condition is built from the PROTECTED TABLES + NUMBER OF DAYS TO KEEP THE ATTACHMENTS ON THE INSTANCE - The above properties can be found and adjusted in the SnowMirror Drive > Workspace > Settings > Advanced Setup menu Rules on other tables These rules mean that we can specify conditions on any table and based on the conditions the attachment that are stored on the found records of this table will be uploaded If the table specified for a rule is not empty and not set the sys_attachment the following cases will be true - The user can specify a condition using the fields of the given table - Example: If the table is incident we can use a condition like [category IS Software] - For the above example, the rule will find all the incident records that satisfy the given condition and upload all attachment that are on these records - If no condition has been provided, this will mean that all attachments that are on that table should be uploaded Other Attachment Rule Notes An attachment even if it is part of multiple different rules can only be uploaded to 1 Storage Provider If 2 rules collide the stronger one always wins and the attachments will be uploaded to the Storage specified in that rule Example for the above: - We have rule A with Storage S1, table is sys_attachment and condition is table_name=incident - We have rule B with Storage S2, table is incident and condition is category=Software - If both rules are active all the attachments including the ones that belong to rule B will be uploaded to Storage S1 - Same as the above if only rule A is active - If only rule B is active, only the attachments that satisfy rule B will be uploaded to Storage S2 - If none of the rules are active, nothing will be uploaded and if anything has already been uploaded it should be redownloaded ## Storage Table Name: x_gvs_offloader_storage Related Property Name: x_gvs_offloader.storage_name Allows the user to setup and manage several different storages on one instance. The above mentioned related property contains the name of a Storage record that will be used for all uploads, downloads, etc. #### The components |Name|Purpose| |---|---| |Name| Name of the property that gets stored on the Configuration page| | Provider| Dropdown list, used to mark the specific storage provider (Azure, AWS S3, Google Cloud)| | Account Name, Container/Bucket, Region, Service| Used for authentication| | Auth Type| Selected authentication type (Shared Key or OAuth2.0)| | Connection & Credential Alias| currently utilized for Amazon S3| | Use MID Server| True if MID Server should be used for the request, False otherwise| | Default MID Server| If "Use MID Server" is checked, providing a MID Server record is mandatory| | Status| Online if the Storage is available, Offline otherwise| | Shared Key| API Key Credentials that stores the account key in an encrypted format| | OAuth Profile| The entity profile used for authentication| ### PA Data Collector Jobs There are several different Scheduled Data Collection jobs in the application each with it's own use case. Most jobs run automatically, while there are a few that require manual starting. All the jobs use the same "Run as" user, a manually created user called "Offloader admin" that acts as a placeholder. **[PA] Data offloader (Historical)** - Historical data collection designed to run once when the application is freshly installed on a new instance. - Used to gather the scores for the indicator [Attachment size in MB] - The field "Relative start" defines how long it should go back, this can be adjusted - Example: 30 means it will go back 30 days into the past and collects data attachment sizes with all the breakdowns for each given day **[PA] Data offloader (Test)** - Data collection job used for testing indicators - Indicators can be freely removed or added to it **[PA] Data offloader (Daily)** - Daily data collection job that runs at a specific time defined in the "Time" field. Unlike the historical data collection, this only collects scores for the last 1 day but includes all indicators of the application for that. - Assigned indicators should not be changed as it can severely affect the Workspace - Runs automatically every day at a specific time **[PA] Data offloader (Regular)** - Updates some of the indicators used on the workspace that calculate actual savings - Assigned indicators should not be changed as it can severely affect the Workspace - Runs automatically every 10 minutes # Design The application is a scoped application, with the possibility of distribution through both update sets and the application studio. ## The main workflow - The administrator configures how long he wants to keep attachments. Then he clicks on the "Offload to Azure" button. - A script uploads all the sys_attachment_docrecords with `sys_attachmentISONEOF to the Azure storage (and optionally deletes them). - The naming convention will be that there is a folder called of the record being uploaded. Inside there will be the sys_attachment_doc records as individual files(blobs) called . So the path to the individual sys_attachment_doc record would be /. - This will be a scheduled job running every night (or on demand execution for first run). - If user accesses the sys_attachment record from anywhere via the sys_attachment.do?sys_id= processor, there is a business rule running on the sys_attachment table that: - Checks if its sys_attachment_doc records are there (more than 0). - Yes => Does nothing and lets the processor follow up with its usual attachment construction and download. - No => - It synchronously puts a gif file in the sys_attachment_doc table with the sys_id of the sys_attachment that the user is trying to download. - It changes the Content type on the sys_attachment table so the user sees he is downloading a gif instead of the original file. - It triggers an asynchronous job that starts downloading the missing sys_attachment_doc parts from Azure. The query will ask for downloading the whole folder on the Azure side. - Once all parts are downloaded, it changes the sys_id in the created sys_attachment_docrecords and removes the placeholder gif parts from there. - It changes the Content type back to the original one. ## The double queue structure There are 2 custom queues in the application that handle the automatic and manual attachment upload/download separately. Both queues use the same events. High priority queue: - Handles manual attachment download events - Manual uploads don't have an event created - This queue checks for new events every second to ensure faster processing - The number of workers can be customized in the Workspace / Settings / Advanced Setup Low priority queue: - Handles events created by the Mass Upload and Mass Download functions - Checks for events in a bigger interval (OOB is 30sec) as these events contain a large array of attachment sys_ids - The number of workers can be customized in Workspace / Settings / Advanced Setup - Max size of this queue can be customized in the Advanced Setup, this means that if the queue fills up no new events will be added until there is enough space ## Mass upload/download architecture There are 2 different Scheduled job for handling the Mass Download and Mass Upload processes separately, but structurally they are almost the same. The Scheduled Job that handles the Mass Upload is called "GVS_OFFLOAD Upload Attachments" and the one that handles the Mass Download is called "GVS_OFFLOAD Download Attachments". #### Main structure - Both jobs utilize the same message queue (a custom one with a configurable number of parallel workers) to handle the Mass Upload/Download of the attachments. - Events are created for both upload and download in the "sysevent" table under the "GVS_OFFLOAD Low Priority Queue" and the first parameter (parm1) of these events contain at least one sys_attachment sys_id or a maximum number specified in the "x_gvs_offloader.batch_size" configuration property - The "name" of each event shows if it's either an upload or download of the given attachments - When an event is created is starts in state "ready", during processing it gets queued up and once it finishes the state will be either "processed" (normally means successful) or "error" - Both Scheduled jobs run periodically OOB every 1 minute and fills up the queue with new elements if there are any - Both Scheduled jobs work between a timeframe every day, which is specified in Advanced Setup - The timeframes are unique for both Scheduled jobs, but they can collide (Mass Upload running between 20:00-06:00 and Mass Download running between 00:00-05:00) - Once out of this timeframe the Scheduled job will stay active, it will finish all the events that are being processed at the time, but it won't start creating new events. All events that are still in ready state will be automatically deleted from the queue. - Both Scheduled jobs requires an active and valid license on the instance to function #### Mass upload process Idea: There is a main function that queries attachments from the "sys_attachment" table, these attachments are sorted out to avoid adding an attachment to a new event when it's already in the low priority queue or has been uploaded. Attachment sys_ids are gathered into an array (batches), once the size of this array reaches the specified batch size (OOB is 100) an event is created to upload those attachments, if there is still space in the low priority queue. If the query ends and there are still ids in the array even if the batch size is less than the needed these will be uploaded. - The query used here is created the following way: - Whitelist condition + Blacklist condition + Upload time limit - Upload time limit refers to "the number of days an attachment should be kept" parameter - Attachments that have already been uploaded are also excluded from this query, this is done by checking if their name contains the archived prefix - After querying these attachments each one is checked individually and will be added to the array (batch) if eligible - Attachment is currently not being processed in one of the queues, means there is no event in the queue of the Scheduled job where the first parameter contains the sys_id if this attachment - Once the length reaches the specified batch size a condition checks if the low priority queue is full - If there is enough space in the low priority queue an event will be created, and the attachments will be queued up for uploading - If there is not enough space in the low priority queue the batch will be thrown away and the attachments will be picked up again during the next run of the Scheduled job - If the query has ended and there are still attachments in the batch, regardless if there is space or not in the queue an event will be created for the attachments #### Mass download process Idea: There is a main function that queries attachments from the "sys_attachment" table, these attachments are sorted to avoid adding an attachment to a new event when it's already in the low priority queue or has been downloaded. Attachment sys_ids are gathered into an array (batches), once the size of this array reaches the specified batch size (OOB is 100) an event is created to download those attachments, if there is still space in the low priority queue. If the query ends and there are still ids in the array even if the batch size is less than the needed, these will be downloaded. The logic of the function is made up in the following way: - All attachments that have the archived prefix (means these are uploaded) are queried and their sys_ids and added to an array - Based on the condition that the Mass Upload uses all attachments are queried and their sys_ids are added to a different array - A loop goes through the first array that contains the sys_ids of all attachment that are uploaded, for each id a condition checks if the given id is also in the second array (attachments that are or will be uploaded) - If the id is in the second array ➡️ do nothing - If the id is not in the second array ➡️ a second condition checks if it's already being processed in of the queues - If the attachment is not being processed in one of the queues, add the sys_id into a new third array, which will contain all the attachment sys_ids that should be re-downloaded - The length of the third array is checked after each addition, once the length reaches the specified batch size a condition checks if the low priority queue is full - If there is enough space in the low priority queue an event will be created, and the attachments will be queued up for downloading - If there is not enough space in the queue the batch will be thrown away and the attachments will be picked up again during the next run of the Scheduled job - If the query has ended and there are still attachments in the batch, regardless of if there is space an event will be created for the attachments ## Attachment upload file consolidation ![image1.png](../assets/images/Technical Documentation/image1.png) - There will be a header consisting of 2 parts, first is length of the header, second contains attachment and doc part related data - For a 30MB file coming from 20 attachment parts - First header part (fixed length, just one numeric parameter) will contain: - Length of the second header - Example: 0000000192 - Second header part: - The original byte size of the attachment - The original content type of the attachment - The original file extension of the attachment - The position of the doc parts together with their length - Example: { attachmentSize: 123456, contentType: image/png, fileExtension: .png 0: 123 1: 456 ... } - The second part of the header is base64 encoded while the first part is not encoded. Both header parts are attached together and will be attached to the beginning of the first blob that is uploaded - Once the header is constructed, the algorithm will iterate through the doc parts in Servicenow and attach them together instead of uploading each doc part 1 by 1. This greatly improves performance. These doc parts are base64 encoded by default within Servicenow and they are taken in the order of their order in the sys_attachment_doc table. - Doc parts are concatenated together until - There are no more doc parts left - The maximum string length has been reached - The maximum string length in Servicenow is 16777216 and there is a 4:3 base64 character to byte conversion. This means that every 4 base64 character equals to 3 bytes in size, so the maximum size of the uploaded blob parts is around 15MB - If the string limit is reached during the concatenation, a new blob part will be created, but only the first part with the blob name "0" will contain the header. - After every "put blob" upload request, there will be a status code and MD5 check to ensure that the file has not been corrupted. The expected status code is 201. In case of failure, a retry mechanism will try to upload the blob for another X amount of time, this can be configured (default is 3) - If any of the blobs have failed to upload correctly, the whole upload process will be cancelled. ## Multiple Storage Providers support. Storage Provider Table (x_gvs_offloader_storage) Purpose The Storage Provider table defines and manages the configuration of external storage endpoints used by the SnowMirror Drive app. Each record represents a storage provider (for example, AWS S3, Azure Blob Storage, or Google Cloud Storage) that can be used to store, retrieve, and manage offloaded attachments. This table acts as the central configuration source that determines: which storage providers are available, how ServiceNow connects to them, and which provider is actively used during offload, download, or delete operations. How the table is used by the system At runtime, the application: Reads the Storage Provider record. Identifies the provider type and connection details. Dynamically initializes the corresponding storage connector. Script Includes GVS_OFFLOADER_RestConnectorBase An abstract base class that implements only the common methods shared across storage providers. All provider‑specific methods must remain unimplemented and should throw an exception if called directly. GVS_OFFLOADER_AzureRestConnector Inherits from GVS_OFFLOADER_RestConnectorBase and provides concrete implementations for communicating with the Azure storage provider. GVS_OFFLOADER_S3RestConnector Inherits from GVS_OFFLOADER_RestConnectorBase and provides concrete implementations for communicating with the Amazon S3 storage provider. GVS_OFFLOAD_StorageProviderRestConnectorFactory Additional classes to support AWS S3 signing process awsSignatureCalculator, CryptoJS The key aspect is how the system determines which storage provider to use. This logic is handled in GVS_OFFLOAD_GlobalUtils.initialize STORAGE_PROVIDER is initialized dynamically at runtime based on the type of the configured Storage Provider record. During initialization, the system determines the storage provider type (for example, AWS, Azure) and uses a factory to instantiate the corresponding REST connector implementation for that provider. ### Extending the System to Support a New Storage Provider Create a Script Include, for example GVS_OFFLOADER_GoogleCloudRestConnector, inherit from GVS_OFFLOADER_RestConnectorBase Update Provider choices on Storage Provider table Update create() method of GVS_OFFLOAD_StorageProviderRestConnectorFactory class and define the mapping of the new ScriptInclude and the new Provider choice value. See create() method for how it's done for other Storage Providers. ### Amazon S3 - request signing All Amazon S3 requests must be authenticated using an AWS SigV4generated signature. In the SnowMirror Drive application, this is implemented in awsSignatureCalculator().getSignature(). You can see this method invoked in every function that sends requests to S3. For signature generation and for Authorization request header Secret Access Key and Access Key are required. They are obtained from AWS S3 by admin. In ServiceNow, the keys are stored as AWS Credentials. ## Attachment Encryption with Field Encryption Enterprise SnowMirror Drive works with standard ServiceNow attachment APIs. To protect sensitive data at rest and ensure regulatory compliance, attachment content can be encrypted using Field Encryption Enterprise. This section describes how to encrypt attachments (for example, on the Incident table) and the key considerations when SnowMirror Drive scheduled jobs process encrypted data. ### 1. Overview of Field Encryption Enterprise Field Encryption Enterprise is an advanced ServiceNow security feature that uses the Key Management Framework (KMF) to encrypt fields and attachments at the application layer. Encryption and decryption are handled transparently by the platform based on access policies and cryptographic modules. #### Key benefits when used with SnowMirror Drive: - Protects sensitive attachment data (PII, PHI, financial documents) stored in ServiceNow before offloading. - Ensures that only authorized users and system processes can decrypt attachments. - Supports customer‑supplied keys (CSK) and enterprise key lifecycle management (rotation, revocation). - Attachments remain encrypted in ServiceNow while SnowMirror Drive offloads and rehydrates them using standard APIs. Official ServiceNow documentation: - Field Encryption Enterprise overview https://www.servicenow.com/docs/r/platform-security/now-platform-encryption.html - Attachment encryption walkthrough https://www.servicenow.com/docs/r/platform-security/attachment-encryption-walkthrough.html ### 2. Prerequisites and Required Roles Before configuring attachment encryption, ensure the following prerequisites are met: #### Plugins and licensing Field Encryption Enterprise must be licensed and activated. Key Management Framework (KMF) must be available on the instance. #### Required roles for setup The user performing the configuration must have: - admin - sn_kmf.cryptographic_manager (to create and manage cryptographic modules) - security_admin (role escalation may be required for encryption configuration) These roles are required to define encryption modules, access policies, and encrypted field configurations. ### 3. Cryptographic Modules and Module Access Policies A Cryptographic Module defines how data is encrypted (algorithm, key source, usage). For attachments, this module is referenced by the Encrypted Field Configuration. Important considerations for SnowMirror Drive: SnowMirror Drive scheduled jobs run as a system‑level user. That user must be explicitly granted access to the cryptographic module via a Module Access Policy, otherwise attachment encryption/decryption will fail during upload or download jobs. ![access_policy_1.png](../assets/images/Technical Documentation/access_policy_1.png) Best practices Create a dedicated module (for example: SnowMirror_Attachment_Module). Define a Module Access Policy that grants: Read/Decrypt access to: - SnowMirror Drive run‑as user (System) - Required admin roles ![access_policy_2.png](../assets/images/Technical Documentation/access_policy_2.png) ![access_policy_3.png](../assets/images/Technical Documentation/access_policy_3.png) Validate access before enabling mass encryption. Module access policies are mandatory for system processes acting on encrypted attachments. ### 4. Encrypted Field Configuration (EFC) for Attachments The Encrypted Field Configuration defines what gets encrypted and under which conditions. Example: Encrypt attachments on the Incident table Navigation: System Security → Field Encryption → Encrypted Field Configurations Create a new Encrypted Field Configuration with the following values: - Type: Attachment - Table: Incident - Active: ✔ (enable only when ready to encrypt) - Crypto module: Select the cryptographic module created earlier - Encrypt by default: Optional, based on requirements Once active: New attachments are encrypted automatically. Access enforcement is handled through Module Access Policies. #### Encrypted Row Configurations (optional) Encrypted Row Configurations can be used to encrypt attachments only when specific record conditions are met (for example: incident.category = Security) and allow fine‑grained control instead of encrypting all attachments on a table. This is useful when SnowMirror Drive is configured to offload only certain attachment sets. ### 5. Mass Encryption and Mass Decryption Jobs When enabling attachment encryption on existing data, historical attachments are not encrypted automatically. ServiceNow provides background jobs for data migration. #### Mass Encryption Encrypts existing unencrypted attachments based on the Encrypted Field Configuration. Required after activating attachment encryption for an existing table (such as Incident). #### Mass Decryption Decrypts attachments if encryption is being disabled or reconfigured. Useful during testing, rollback, or module changes. #### Navigation: System Security → Security Jobs → Create New → Mass Encryption Job #### Operational guidance Run Mass Encryption during low‑usage windows. Monitor job progress and logs before enabling SnowMirror Drive mass upload jobs. Ensure SnowMirror Drive scheduled jobs remain disabled until encryption completes to avoid partial access issues. Mass encryption and decryption are required to keep attachment state consistent across existing and new records. #### Notes for SnowMirror Drive Operations - SnowMirror Drive uses standard attachment APIs; encryption and decryption are handled transparently by ServiceNow. - Ensure the SnowMirror Drive system user always retains cryptographic module access. - Changes to encryption configuration should be coordinated with SnowMirror Drive mass upload/download schedules. - Test attachment upload and re‑download scenarios after enabling encryption. ## Automated Testing with Robot Framework #### Installation and first run 1. Install Robot, Install Python https://robotframework.org/?tab=1#getting-started 2. Download the project files into a suitable folder (log in to git inside your IDE with your token and clone the repository) 3. Put attachments into the upload folder – it's in the project location 4. Define ${USERNAME} and ${PWD} for the instance in CommonKeywords.robot 5. Define ${INSTANCE_NAME} variable in CommonKeywords.robot 6. Ensure querySelectorDeep.js file exists in the project folder. It was created by GV team to handle shadow DOM. #### Source files Link to git https://gitlab.guidevision.cz Note, this is not the final version. There is a lot of room for improvement and refactoring. ##### Project folder It contains the following information files: - run_robot_parallel_v4.py - this orchestration script drives the core loop scenario by executing the associated Robot Framework files. For detailed behavior and execution logic, refer to the comments within the robot files. - CommonKeywords.robot - test_setup.robot - create_incident.robot - assert_file_integrity.robot - test_cleanup.robot - querySelectorDeep.js - a helper script for selecting HTML elements located within a web page's shadow DOM. - incident_numbers.json - storage file for mapping attachment filenames to incident numbers. Updated during the scenario execution. How it works What you need to run is the orchestration file run_robot_parallel_v4.py. Below is an overview of the main steps it does: (Make sure your terminal is in the project root folder, then paste "python run_robot_parallel_v4.py" in it and hit enter) - Runs test_setup.robot and sets up prerequisites for testing: - Clears the script generated Downloads folder (Chrome uses a custom downloads folder for testing) - Updates system properties (days to keep and offload condition) - Enables Mass Upload. - Runs create_incident.robot for each file in parallel. - Extracts and stores incident numbers in incident_numbers.json. - Runs assert_file_integrity.robot for each file, passing the correct incident number. - Runs test_cleanup.robot and restores the changed system properties. - Generate unique output files for each Robot Framework execution. - Collect all output files. - Merge them into a single report using rebot. - Places all Reports and Output files into the script generated Resources folder. # Manual Installation through Update Sets Each release has 2 different version PATCH and FRESH INSTALL ## Fresh Install: - Should be used when the application is not yet installed on the instance - Contains all the changes for the specific version, contains all system properties, but some have empty values that needs to be filled as part of the installation - Ensures that the application can be installed on a brand new instance, contains everything **Installation steps** Go to "Retrieved Update Sets" > Click on "Import Update Set from XML" > Select the XML file that contains the version you want to install 1. Import the XML file - If the XML has been imported successfully, open the record with it's name that is on state "Loaded" - Click "Preview Update Set Batch" - If there are any conflicts, just click on "Accept remote update" for each - Once there are no conflicts click "Commit Update Set Batch" 2. The fix script called `GVS_OFFLOAD Setup Application` needs to be executed as it initializes some of the parts of the application - Use the option "Proceed in Background" while running it - This script can take up to a few minutes to finish. 3. Swap to "Data offloader" scoped application, needed to edit settings and run the fix scripts without any issue 4. License key is needed for application. Please reach out to SnowMirror Drive support to acquire a license key. 5. Go to "SnowMirror Drive > Workspace > Settings" and setup the application 6. In the "Basic Setup" menu go through each setting 7. In the "Storage Setup" menu - Click on "New" and setup at least 1 Azure or AWS remote storage for the application by going through the steps 8. (Optional) In the "Advanced Setup" menu go through the settings and change them if needed 9. Go to "SnowMirror Drive > Workspace > Attachment rules" to setup and activate the rules 10. Go to "SnowMirror Drive > Workspace > Home" under the "Transfer status" tab enable the automatic upload of attachments ## Patch version: - Should be used when the application is already installed on the instance - Contains all the changes for the specific version, but doesn't overwrite anything specific for the instance - Only includes the 2 XML for the scoped and global applications **Installation steps** Go to "Retrieved Update Sets" > Click on "Import Update Set from XML" > Select the XML file that contains the version you want to upload and upload it 1. Import the Global XML file - If the XML has been imported successfully, open the record with it's name that is on state "Loaded" - Preview the record - If there are any conflicts, just click on "Accept remote update" for each - Once there are no conflicts commit the update set 2. Import the Scoped XML file - If the XML has been imported successfully, open the record with it's name that is on state "Loaded" - Preview the record - If there are any conflicts, just click on "Accept remote update" for each - Once there are no conflicts commit the update set 3. (Optional) The fix script called "GVS_OFFLOAD Setup Application" can be executed as it runs some of the jobs which are responsible for refreshing the data on the Workspace # Amazon S3 setup Create S3 Bucket Log in to the AWS Management Console. Navigate to S3 (Simple Storage Service). Click Create bucket. Configure Bucket Settings Bucket name Enter a globally unique name (e.g. servicenow-offloader-bucket) Region Select the region where your bucket will be hosted (e.g. eu-central-1) Note, the selected region must match the region configured in ServiceNow. A mismatch will result in authentication (signature) errors. Object Ownership Set Object Ownership to: ACLs disabled (recommended) Block Public Access Keep all Block Public Access settings enabled: Block all public access Bucket Versioning (Optional) Enable Versioning if required. Default Encryption (Recommended) Enable server-side encryption: Encryption type: Amazon S3 managed keys (SSE-S3) Create Bucket Review the configuration Click Create bucket Create IAM User Log in to the AWS Management Console. Navigate to IAM (Identity and Access Management). In the left navigation pane, select Users. Create a new user (e.g. servicenow-s3-user) Choose Attach policies directly (skip choosing policy for now), click Next. On Review and create page click Create user. Create S3 Access Policy In IAM, navigate to Policies. Click Create policy. Open the JSON tab. Paste the following policy and replace your-bucket-name with your actual bucket name: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:*"], "Resource": [ "arn:aws:s3:::your-bucket-name", "arn:aws:s3:::your-bucket-name/*" ] } ] } Provide a name (e.g. S3FullAccessToBucket). Click Create policy. Attach Policy to User Return to IAM -> Users. Select the previously created user. Open the Permissions tab. Click Add permissions. Select Attach policies directly. Search for the policy created in the previous step. Select it and click Add permissions. Generate Access Keys Open the user in IAM. Go to the Security credentials tab. Scroll to Access keys. Click Create access key. Select Application running outside AWS. Click Create access key. Save the following values securely: Access Key ID Secret Access Key The secret key is only displayed once and cannot be retrieved later. Integration Notes The bucket name must match exactly the value configured in ServiceNow. The IAM user created earlier must have access to: the bucket (arn:aws:s3:::bucket-name) all objects (arn:aws:s3:::bucket-name/*) No additional bucket policy is required if IAM permissions are configured correctly.