Masking runs

• Overview
• Create database masking run
  • Database run options
• Create file masking run
  • File run options
• Preview and confirm run
• Run logs
  • Run statuses
  • Filtering runs
  • Run details
  • Database masking job summary
  • File masking job summary
  • File discovery summary
  • Run Report
  • Connection and ruleset snapshots
  • Downloading a run log
  • Downloading a sensitive data discovery report
  • Downloading a discovery config snapshot
  • Cancelling a run
• Run History
  • Database masking runs
  • File masking runs
  • Verification of the ruleset hash in the run history
• Simultaneous runs

Overview

A masking run is the process of a applying a masking Ruleset to a database or file Connection. Runs are created from the Database Masking or File Masking dashboards. For automating the masking process, they can also be triggered via the API (see Best Practices).

Create database masking run

A new database masking run can be configured with the following steps:

1. Navigate to the Database Masking dashboard
2. Select a connection from the list of available connections. This is the run's source.
3. Select a ruleset from the list of available rulesets.
4. Set additional run options (detailed below).
5. Click PREVIEW RUN or PREVIEW DRY RUN.
6. After configuring the run, you will be taken to the Preview and confirm run screen.

*Dry Run allows you to test your rulesets without modifying the database. When a dry run is executed, DataMasque performs every operation as usual except:

• the final UPDATE operation of mask_table tasks, which would otherwise write the masked value to the database
• the value generation and subsequent UPDATE operation for mask_unique_key tasks
• the truncate_table operation
• the run_sql operation
• certain mask_table masking tasks (from_blob, from_unique_imitate, secure_shuffle), as these require temporary alterations to the database schema.

Database run options

Run options are displayed on the Run Options section of the Database Masking Dashboard.

The following options are available:

Option	Default	Description
Batch Size1	Varies by engine	The maximum number of rows that will be fetched, masked, and updated in a single operation by DataMasque. Larger batch sizes will reduce database operation overhead, but using a batch size value that is too large may result in DataMasque memory exhaustion. This value does not affect the total number of rows masked. The default is engine-specific, based on benchmarking: 10,000 for MariaDB, MySQL and Oracle; 50,000 for PostgreSQL and Microsoft SQL Server; 250,000 for Snowflake. Other database types fall back to 50,000. A mask_table or mask_unique_key task can set its own batch_size, which overrides this value for that task. Note: The Batch Size parameter is not applicable for Amazon Redshift and Amazon DynamoDB masking runs. Only for database masking runs.
Max rows	unset	The maximum number of rows that will be masked by each mask_table task. May be used for speeding up test iterations when developing rulesets. See usage notes below.
Run secret	A secure random value	The run secret is used in the random generation of masked values. Please see the Run Secret Options below.
Continue on failure	false	If there is a task failure, and this option is false, DataMasque will skip all remaining non-started tasks. If this option is true, DataMasque will continue performing other tasks even if there is a task failure. It can be useful to set this option to true when testing/debugging your masking ruleset to identify as many failures as possible in each run.
Notify me when this run completes	false	Email the current user when the job completes. The DataMasque instance must have SMTP configured.
Disable instance secret	false	If this option is set to true, DataMasque will exclude its instance-specific secret and generate masked values based solely on the run secret. You may wish to disable the instance secret in order to achieve consistent masking across DataMasque instances. However, by disabling the instance secret, any DataMasque instance using the same run_secret could replicate your data masking.
Override connection's Server Side Encryption settings	false	Only valid for Amazon DynamoDB connections. If checked, table SSE key settings may be overridden on a per-table basis. Click the gear icon to view the settings. See the Specifying SSE Options documentation for DynamoDB for more information about these settings
Max file size2	See description	The max file size is only applicable to Amazon Redshift and Amazon DynamoDB databases. For Redshift it sets the MAXFILESIZE of the UNLOAD command. For DynamoDB it determines the maximum pre-compression size of batch files split from the original files exported by DynamoDB to S3. In both cases, this determines the maximum file size in MB of records that will be masked at once by a single DataMasque worker. The max file size can be set to any integer value between 5 MB and 1000 MB, and defaults to 5 MB for Redshift and 100MB for DynamoDB (Redshift files are stored in the compact Parquet format, which expands to a greater extent when loaded into DataMasque's memory). Only for database masking runs.
Copy LOB data (CLOB, NCLOB, BLOB)	true	Only shown when copying to a destination. When unticked, CLOB, NCLOB and BLOB columns are created at the destination, but their source values are not copied, which makes the copy faster. The destination uses each column's default, or NULL if no default exists. See Current scope.
Enable Diagnostic Logging	false	With diagnostic logging enabled, extra information (Columns, constraints, foreign keys and indexes) is captured to assist with diagnosing problems with masking runs. Because of the extra information collected, masking runs may take longer to execute with this option enabled. System memory information will be output to the run log in the following format Memory: T:31.19GB / F:0.26GB / A:10.60GB where T, F, and A stand for Total, Free and Available memory respectively. Memory of the masking agent process will also be captured in the following format Memory for PID(58): 355.17MB. All workers of a run are threads of that one process, so this line always reports the same process ID.
Pull ruleset from latest git branch	false	Only shown when Git integration is configured (a Git repository URL is set in the Git Repository Settings). When enabled, DataMasque pulls the latest version of the selected ruleset from a Git branch before the run starts, overwriting the stored ruleset with the pulled content. Selecting this option reveals a Branch dropdown, populated from the repository and defaulting to the configured Git branch. If the pull fails, the run does not start.

Notes:

¹Batch Size applies to database types other than Amazon Redshift and Amazon DynamoDB.

²Max file size only applies to Amazon Redshift and Amazon DynamoDB databases.

Max rows Usage Notes

Warning: In the case that a table contains more rows than the Max rows value, the remaining rows will be left unmasked.

Note: Masking a column that is part of a unique constraint, such as a primary key column, with from_unique_imitate or imitate_uuid when Max rows is set is not allowed. This is because partial masking of unique values can cause the masked values to conflict with unmasked values.

For Amazon DynamoDB, this restriction does not apply. However, masking the key column can result in loss of rows if the final key column values are not all unique.

Note: The Max rows parameter has no effect on Amazon Redshift masking runs.

Note: Max rows does not apply to mask_unique_key tasks.

Create file masking run

A new file masking run can be configured with the following steps:

1. Navigate to the File Masking dashboard
2. Select a source connection from the list of available connections (files will be read from here).
3. Select a ruleset from the list of available rulesets.
4. Select a destination connection from the list of available connections (files will be written to here).
5. Set additional run options (detailed below)
6. Click PREVIEW RUN or PREVIEW DRY RUN.
7. After configuring the run, you will be taken to the Preview and confirm run screen.

*Dry Run allows you to test your rulesets without modifying the files. When a dry run is executed, DataMasque downloads files from the source, applies include/skip rules and masks them, but does not upload the masked data to the destination. This allows you to see which files would get masked, and if there are any errors in your ruleset.

File run options

Run options are displayed on the Run Options section of the File Masking Dashboard.

The following options are available:

Option	Default	Description
Run secret	A secure random value	The run secret is used in the random generation of masked values. Please see the Run Secret Options below.
Continue on failure	false	If there is a task failure, and this option is false, DataMasque will skip all remaining non-started tasks. If this option is true, DataMasque will continue performing other tasks even if there is a task failure. It can be useful to set this option to true when testing/debugging your masking ruleset to identify as many failures as possible in each run.
Notify me when this run completes	false	Email the current user when the job completes. The DataMasque instance must have SMTP configured.
Disable instance secret	false	If this option is set to true, DataMasque will exclude its instance-specific secret and generate masked values based solely on the run secret. You may wish to disable the instance secret in order to achieve consistent masking across DataMasque instances. However, by disabling the instance secret, any DataMasque instance using the same run_secret could replicate your data masking.
Enable Diagnostic Logging	false	With diagnostic logging enabled, extra memory information is captured to assist with diagnosing problems with masking runs. Because of the extra information collected, masking runs may take longer to execute with this option enabled. System memory information will be output to the run log in the following format Memory: T:31.19GB / F:0.26GB / A:10.60GB where T, F, and A stand for Total, Free and Available memory respectively. Memory of the masking agent process will also be captured in the following format Memory for PID(58): 355.17MB. All workers of a run are threads of that one process, so this line always reports the same process ID.
Pull ruleset from latest git branch	false	Only shown when Git integration is configured (a Git repository URL is set in the Git Repository Settings). When enabled, DataMasque pulls the latest version of the selected ruleset from a Git branch before the run starts, overwriting the stored ruleset with the pulled content. Selecting this option reveals a Branch dropdown, populated from the repository and defaulting to the configured Git branch. If the pull fails, the run does not start.

Copy mode (source and destination)

Note: see Current scope for supported databases, masks, and column types.

By default a run masks a database in place: the connection you select is both read from and written to. Choose Copy & mask and select a destination connection to copy the data. The source is only read, and masked rows are inserted into the destination.

Copy mode is intended for producing a masked copy of a production database without touching production itself. DataMasque sends every write to the destination and refuses tasks that would write to the source.

A copy run counts toward license usage once, against the source database.

Choosing what gets copied

A ruleset can declare a copy_scope block naming the tables to copy. Each entry names one table, or a glob or regex pattern, qualified with its schema:

version: "1.0"
copy_scope:
  include:
    - table: HR.DEPARTMENTS
    - glob: "HR.LOOKUP_*"
  truncate:
    - table: HR.AUDIT_LOG
  skip:
    - regex: "^HR\\.TMP_.*"
tasks:
  - type: mask_table
    table: HR.EMPLOYEES
    key: ID
    rules:
      - column: NAME
        masks:
          - type: from_fixed
            value: REDACTED

The two halves of the configuration have to agree, and the run preview tells you when they do not:

Ruleset	Destination	Result
Has copy_scope	Selected	Copy mode. The block decides which tables reach the destination. See Current scope.
Has copy_scope	Not selected	Refused. The preview explains why and START RUN is disabled.
No copy_scope	Selected	Allowed, with an informational warning: only the tables named in masking tasks are created and copied at the destination. A masked table's foreign-key parents must be masked too, or the run stops, naming them. To copy them unmasked, list them under include in a copy_scope block.
No copy_scope	Not selected	An ordinary in-place run, unchanged.

Warning: a ruleset written for copy mode that ran in place would mask the source database itself. A run whose ruleset declares copy_scope without a destination is therefore refused outright, before any masking work begins, rather than being allowed to proceed with a warning.

Filtering the copy

By default a copy takes every row of each table it copies. A filter in copy_scope copies only the rows tied to one table, the root: the root rows that pass the filter's conditions, the rows that reference them through foreign keys, and the parent rows those rows need. The copied rows are masked as in any copy; the filter decides only which rows reach the destination.

version: "1.0"
copy_scope:
  filter:
    table: SALES.CUSTOMERS
    if:
      - column: CUST_ID
        in:
          table_reference: former_customers
          column: CUSTOMER_ID
tasks:
  - type: mask_table
    table: SALES.CUSTOMERS
    key: CUST_ID
    rules:
      - column: SURNAME
        masks:
          - type: from_fixed
            value: REDACTED
  # and a mask_table task for every other table the filter reaches

• table: the root table, as an exact schema-qualified name. A pattern is not accepted, and the root must not be under truncate or skip.
• if: the conditions a root row must pass to be copied, in the same form as a mask_table rule's if, including and, or, not, table references and value_transforms. Each column must be a column of the root, named without a table prefix, since the filter has no joins.
• additional_cascades (optional): relationships the database does not declare as foreign keys. See Relationships that are not foreign keys.

A ruleset has at most one filter.

Which rows are copied

• The root keeps each row that passes the conditions.
• A table that references the root, directly or through other reached tables, keeps each row whose enabled foreign key references a kept row. A row with foreign keys onto two reached tables is copied when either of its parent rows is kept.
• A parent row is pulled in when a kept row references it through a NOT NULL foreign key, so the destination stays referentially whole. A pulled-in row brings in its own NOT NULL parents, but not the rows that reference it. This includes the root: a root row the conditions rejected is still copied when a kept row needs it as a parent.
• A nullable foreign key whose parent row is not copied is set to NULL at the destination, once every row is loaded and before the foreign keys are added. The run log gives the count for each foreign key, for example:

  Set the foreign key `CLAIM_POLICY_FK` on `SALES.CLAIM` (`POLICY_ID`) referencing `SALES.POLICY` (`ID`) to NULL in 312 rows at the destination, since the parent row is not in the copy.

• A self-reference, or a foreign key back into a table already reached through a cycle, does not carry the copy down. A customer's copy does not include the customers they referred, unless a kept row needs them as parents.
• A table the filter does not reach is copied whole if the ruleset copies it, as in a copy without a filter. A table under truncate or skip is not followed.

Every table the filter reaches must be named by a mask_table task or an include entry, or, for a table reached only as a parent, come in through auto_include_reference. Otherwise the run stops before the destination is touched, with one message listing every such table. Give each a mask_table task: an include entry copies the table unmasked, so use it only for a table that holds no personal data. A table the destination needs no rows of can go under truncate or skip instead, which the filter does not follow.

The run also stops before the destination is touched on a few shapes a filtered copy cannot yet take, such as a table copied whole whose NOT NULL foreign key references a table the filter reaches only as a parent. The message names the tables involved.

The tasks for the tables the filter reaches run in the order the filter reads them, so a task's place under serial or parallel has no effect in a filtered copy. If one of them fails, the tasks for the tables the filter reads after it are skipped, even with Continue on failure on, since those tables are filtered on the rows it copies. The include tables the filter does not reach are copied first, as in a copy without a filter, and the mask_table tasks for the tables it does not reach run last, after a failure too when Continue on failure is on.

Under a row limit, each table the filter reaches holds at most the limit's number of rows, counting the rows the filter keeps, including the parent rows it pulls in, rather than the rows it reads. Such a table is read in key order by one worker, so a run keeps the same rows each time, and its read stops once it holds the limit. A row the limit leaves out can be the parent of a copied row: a nullable foreign key to it is set to NULL, and a row whose NOT NULL foreign key names it is removed, so the destination's foreign keys stay validated.

A filtered copy reads every column of each batch before the filter drops the rows it does not keep, so a table with LOB columns can use much more agent memory than the rows copied suggest, even when few are kept. If a run runs short of memory, turn off Copy LOB data when the LOB values are not needed, or lower Batch Size for the run, or the batch_size of the mask_table task for that table.

Relationships that are not foreign keys

To follow a relationship the database does not declare, list it under additional_cascades. Each entry takes the fields of the subset task's, plus source_table. The source is the parent and the target the table that references it, both as exact schema-qualified names, with their columns in matching order:

copy_scope:
  filter:
    table: SALES.CUSTOMERS
    if:
      - column: STATUS
        equals: CLOSED
    additional_cascades:
      - source_table: SALES.CUSTOMERS
        source_columns:
          - CUST_ID
        target_table: SALES.NOTES
        target_columns:
          - CUSTOMER_REF

A row of the target is copied when its target columns hold the source columns of a row the copy keeps, and the target's own foreign keys carry the copy on from there. A cascade only carries the copy down: a NOT NULL target column does not pull in rows of the source.

A cascade matches rows by their source values and does not carry masked values to the target. If a source column is masked, mask its target column too, in a rule outside any if, or the run stops before the destination is touched.

A relationship that is neither declared nor listed is not followed. Check the plan a dry run logs against your own data map.

Checking a filter with a dry run

A filtered dry run reads no rows. It logs the plan, giving the role of each table the filter reaches and the order it reads them, and each table it does not reach, for example `SALES.REF_CODES` -> COPY (not reached by the filter, copied whole). It stops, as a run would, on a reached table that is not named.

A run logs, for each reached table, how many rows came from the filter and how many were pulled in as parents:

Copied 1,204 rows of `SALES.POLICY` to the destination: 1,198 from the filter and its cascade, 6 pulled in as parents.

When the filter is a single in against a table reference, the run also logs how many distinct values in the root's copied rows are among the reference's IDs:

12,362 distinct values in the copied rows of `SALES.CUSTOMERS` are among the 12,400 IDs in table reference "former_customers".

A count well below the IDs can mean the IDs and the root's column differ in case or padding, which value_transforms can correct.

Under on_empty_table_reference set to warn or none, an empty table reference lets the run go ahead: an in filter then copies no root rows, and a not_in filter copies every root row.

Restrictions in copy mode

A copy writes its masked rows to the destination and never to the source, so a task that only has a meaning when a table is rewritten in place is refused when a destination is set. The run is refused before any table is copied, not when the offending task's turn comes around:

• secure_shuffle, which permutes values already present in the table.
• mask_unique_key, which rewrites a key in place across its referencing tables.
• run_sql, whatever its target.
• truncate_table, subset and build_temp_table. Each of these writes down the run's single connection, which in a copy is the source.

mask_table is the one task a copy can run. These tasks are refused in a dry run too.

Settings that would leave the destination with fewer rows than the source are refused as well, because a row the run does not read is never written:

• The max_rows run option. The run page hides Max rows in copy mode.
• where and joins on a masking task. To copy only some rows, use a filter.

In copy mode the run page marks connections of other types than Oracle and Db2 for i Cannot copy, and shows no destinations for such a source.

The source and destination must run the same database version: the same major version for Oracle, for example 19, and the same version and release for Db2 for i, for example 7.4. A run between different versions fails before any table is copied, naming both versions. As for any failed run, its row is still written to the destination's DATAMASQUE_RUN_HISTORY table.

Source and destination on one server

If the source and destination connections use the same host and port, they may be the same database. The preview page then warns and lists the settings that differ, for example Database: PRODPDB (source), TESTPDB (destination). It compares the database, service name, username, schema, wallet and the other connection settings, but not passwords. START RUN then asks you to confirm before the run starts.

The check compares host names as text, ignoring case, and treats a blank port as 1521 for Oracle. It does not recognise an alias, or an IP address and a host name for the same server, so no warning does not prove that the databases are different.

Encrypted Oracle connections in copy mode

The source and destination each use their own encryption settings:

• Each connection uses the Oracle Wallet attached to it. The two connections can use different wallets, or only one of them can use a wallet.
• Oracle Native Network Encryption (NNE) applies only to the connection it is selected on. One connection can use NNE while the other uses a wallet or neither.

Connections a copy opens

A copy loads the tables it copies unmasked, those under include and the foreign-key parents auto_include_reference adds, up to 10 at a time. Each of these workers opens one connection to the source and one to the destination, and the run holds one more source connection throughout. Loading 10 or more such tables therefore holds 11 source connections and 10 destination connections at once.

To load fewer unmasked tables at once, and so open fewer connections, set workers in the copy_scope block, from 1 to 10:

copy_scope:
  workers: 4
  include:
    - glob: "HR.LOOKUP_*"

Masked tables load after the unmasked ones, through their mask_table tasks. Each task opens one source and one destination connection per worker, as the task's own workers setting says, and copy_scope.workers does not limit them.

Run history in copy mode

A copy writes its run history row to the destination, never to the source. The DATAMASQUE_RUN_HISTORY table is created in the destination connection's schema, or the default schema of its user when the connection names none, which need not be the schema the copied tables are created in. The row's connection snapshot describes the destination. If the destination cannot be reached when the run ends, no row is written. A source's own DATAMASQUE_RUN_HISTORY table is never copied.

Current scope

Copy mode copies Oracle to Oracle, and IBM Db2 for i to IBM Db2 for i. The rest of this section describes Oracle. IBM Db2 for i lists where a Db2 for i copy differs. A run is refused before it writes anything if the ruleset asks for more:

• Databases. The source and destination must be the same database type, and must be two different connections that do not share the same host, port and service name or SID. DataMasque writes only to the destination and only reads the source. See also Source and destination on one server.

• Column types. Every column of a table the copy creates, whether copied, masked or under truncate, must be one of the supported Oracle scalar types: NUMBER, VARCHAR2, CHAR, NCHAR, NVARCHAR2, DATE, TIMESTAMP (including WITH TIME ZONE and WITH LOCAL TIME ZONE), FLOAT, BINARY_FLOAT, BINARY_DOUBLE, RAW, CLOB, NCLOB or BLOB. Any other type is refused when the run first reads the table's structure.

  LOB values are copied whole. A LOB column is created without the source's storage clause, so it takes the destination's defaults. An empty source LOB (EMPTY_CLOB() or EMPTY_BLOB()) arrives at the destination as NULL, unless its column is NOT NULL. Such a column is created without NOT NULL, and once every task has finished, its empty LOBs are written and NOT NULL is restored. This includes a column a rule masks, so a row the rule skips keeps its empty LOB, and a mask that writes NULL leaves an empty LOB rather than failing the run. A table copied without masking is read at most 1,000 rows at a time when it has a LOB column whose data is copied, whatever the batch size.

  When the Copy LOB data run option is off (copy_lobs: false through the API), LOB columns are created, but their source values are not copied. The destination uses each column's default, or NULL if no default exists. The run log names these columns. The run is refused before anything is created if a table it copies or masks has:

  • a NOT NULL LOB column without a non-NULL default;
  • a LOB column that a masking rule writes to, do_nothing included.

  A TIMESTAMP WITH TIME ZONE value keeps its offset, including in a masked column where a condition skips the row or the mask keeps the value. A value a mask produces without an offset takes the run session's time zone, as in-place masking does.

  A FLOAT is copied with a known loss, and a table that needs it preserved exactly should not be copied yet:

  • A FLOAT is carried through a double, so a value carrying more than about 17 significant digits is rounded. The declaration is not the thing to check: FLOAT(126) is 126 binary digits, around 38 decimal ones. Values must fit in a double to be copied exactly. NUMBER is not affected and keeps its value exactly, whatever its precision.

  BINARY_FLOAT and BINARY_DOUBLE are copied exactly, including infinities and NaN.

• Text read under a source connection's data encoding is copied as the source stores it. Unmasked VARCHAR2 and CHAR values keep their stored bytes rather than being written as the text DataMasque decoded.

• One masking task per table. Combine all rules for a table into one mask_table task. Multiple tasks for the same table are rejected, with or without copy_scope.

• Masks. do_nothing, from_choices, from_file, from_fixed, from_format_string, from_random_boolean, from_random_date, from_random_datetime, from_random_number, from_random_text, imitate_unique, from_unique_imitate and imitate_uuid. Any other mask type is refused before the task starts.

• A masked key reaches the tables that reference it. When imitate_unique, from_unique_imitate or imitate_uuid masks a column that an enabled foreign key references, every table referencing it, and every table referencing those through the masked columns, has its foreign key columns masked the same way. Its other columns are copied unchanged, so each such table must be named under include. The run log lists these tables in the plan. The run stops before the destination is touched if such a table:

  • is not named under include, or is matched by truncate or skip;
  • has a mask_table task of its own, which includes the masked table itself when a cycle of foreign keys leads back to it;
  • references masked keys in more than one table;
  • has neither a primary key nor a unique key or index, or has a NULL in the key it is read by: its primary key, else its shortest unique key or index;
  • takes the masked values into an identity column;
  • is in a schema that no copy_scope pattern names; or
  • references a column masked inside an if, or masked by more than one rule.

  A sequence that a masked key column, or a column masked through its foreign key, takes its default from is moved past the masked values once every task has finished.

• DataMasque creates the destination's tables. Every table it copies, masks or truncates is created under the source's schema name, and nothing it creates may already exist at the destination. See Destination schema. The source connection's session schema is applied to the destination connection as well.

• Tables under include are copied unmasked. The list is for reference data the destination needs unchanged, so a pattern matching a table that holds sensitive data copies it as it stands. A table a mask_table task names is always copied masked.

• auto_include_reference copies foreign-key parents, also unmasked. It is off by default, so a parent nothing else names stops the run, which names every such table to add, grandparents included, and the tables that need each one. With it on, that parent is copied to keep the destination referentially whole.

• A table under include needs a primary key. The copy reads and writes it in key order, and copy_scope names no key of its own. A table without one stops the run, as does one whose primary key holds a NULL, which a key that is not validated allows.

• A table under skip is not created at the destination. Use truncate for a table that must exist at the destination but hold no rows: it is created empty.

• The destination must not be marked read-only.

• A masked column must not sit under an enabled primary key, unique constraint or recreated unique index, unless it is masked only with do_nothing, imitate_unique, from_unique_imitate or imitate_uuid, in one rule outside an if, since the other masks above do not keep values unique, and rows a condition skips keep source values that can collide with masked ones: the run stops before the destination is touched.

• A key with no index is not indexed. The temporary index index_key_columns would create is skipped, and the run log says so. One left on the source by an interrupted run is kept, with a warning, and is not created at the destination. key: ROWID is unsupported: copied rows receive new ROWIDs. Specify a key column instead.

• A masked table's destination indexes are managed as they are for an in-place run, under auto_manage_indexes. The source's indexes and triggers are never touched.

Destination schema

A copy creates the destination's tables, keys, indexes and sequences from their definitions at the source, before it writes any rows, and adds the foreign keys between them once every row is loaded. What it cannot create is either left out, with a warning, or stops the run, as listed below. Either way this is settled before the destination is touched. A dry run creates nothing: it lists the tables, sequences and foreign keys it would create, and logs the same warnings and refusals as a real run.

What is created:

• Tables, under the source's schema and table names, in the owning schema's default tablespace.
• Columns, in the source's order, with their type, NOT NULL, and their default when it is a literal or a sequence's NEXTVAL. INVISIBLE columns stay invisible, and are copied.
• Primary keys and unique constraints, with the names a user gave them and their deferrability. A disabled constraint is created disabled, and an ENABLE NOVALIDATE one ends ENABLE NOVALIDATE.
• Indexes, plain and bitmap, with their names, uniqueness and visibility, when the table's schema owns them. A destination that cannot create bitmap indexes, such as Standard Edition, gets each bitmap index as an ordinary index, with a warning, or leaves it out, with a warning, when another index or key on the table already covers the same columns.
• Identity columns, with the source's sequence options.
• Every sequence in the created tables' schemas, and any sequence in another schema that a created column's default draws from.
• Foreign keys between created tables, once every row is loaded, with the names a user gave them, their delete rule and deferrability, in the source's state: enabled and validated, ENABLE NOVALIDATE, or disabled (a DISABLE VALIDATE key arrives DISABLE NOVALIDATE). Being added last, cycles and self-references need no special handling. An enabled foreign key from a copied or masked table to a table the copy does not create stops the run, as auto_include_reference describes. A disabled one, or one from a table created empty, is left out, with a warning. For a parent table in another schema, the copy first grants REFERENCES on it to the child table's owner. When the destination connection's user owns the child table, or owns neither table and lacks GRANT ANY OBJECT PRIVILEGE, it cannot make that grant, so the key is left out with a warning, to add by hand.

What is not created:

• Check constraints.
• Views, materialised views, triggers, stored procedures and packages, synonyms, grants and comments.
• Partitioning, index organisation, tablespace and storage settings. Such tables are created as ordinary tables.
• Function-based (including descending), reverse-key, bitmap join, domain and other index types, and defaults that are expressions, such as SYSDATE, other than EMPTY_CLOB() and EMPTY_BLOB(). Each one left out is logged as a warning.
• Indexes owned by a schema other than their table's, each logged as a warning. A primary key or unique constraint enforced by such an index is still created, and gets its own index in the table's schema.
• DEFAULT ON NULL on a column that is not an identity column. Its default and NOT NULL are kept, so inserting an explicit NULL fails at the destination.
• The SCALE, EXTEND and SESSION sequence options.

A table with a virtual column stops the run: leave it out of the copy. So does a masked table whose mask_table key columns hold a NULL, an identity column masked with imitate_unique, from_unique_imitate or imitate_uuid, and a masked column on either side of an enabled and validated foreign key that the copy creates, since masked values would fail the key once the rows are loaded. The exception is a referenced column masked only with imitate_unique, from_unique_imitate or imitate_uuid, in one rule outside an if, since the tables referencing it are copied with the same masked values. A destination older than Oracle 12c stops the run when the copy would create an identity or INVISIBLE column, or a sequence NEXTVAL default.

Nothing the copy creates may already exist at the destination. A run that finds any of its tables, sequences, indexes, or key or foreign key names there, or a view or other object sharing a table's name, stops before creating anything, naming each one. So does a destination missing a schema the copy creates in: the schemas themselves are not created. It can only see what the destination connection can see. Creating the destination's tables and copying the include tables come first, and a failure there stops the run, even with Continue on failure on.

If creating the destination schema fails part-way, the run drops what it created, and names anything it could not drop. A run that fails after that leaves what it created in place, and its tables may be partly loaded: drop those objects, or point the destination connection at an empty database, before running again.

Sequences and identity columns start where copy_scope's sequences setting says:

• preserve, the default, starts each at the source's high-water mark, so the destination never issues a value the source already has. Values the source issues while the copy runs are skipped too, at the end of the run.
• offset starts each at the given offset, which it requires. Choose one above every key the copy loads, or the destination's new values collide with them.

copy_scope:
  sequences:
    mode: preserve
  include:
    - table: HR.DEPARTMENTS
      sequences:
        mode: offset
        offset: 1000000

An include entry's own sequences overrides the block's for that table's identity columns and the sequences its defaults draw from. A masked table carries the block's setting. A sequence whose tables carry different sequences settings, or that would start outside its MINVALUE/MAXVALUE range, stops the run.

GENERATED ALWAYS identity columns are created GENERATED BY DEFAULT, so the source's values can be loaded, and set back when the copy is finished, as below.

Finishing a copy by hand

Once every other task has finished, DataMasque sets GENERATED ALWAYS identity columns back, resets DEFAULT ON NULL identity columns to their start, enables ENABLE NOVALIDATE keys, under preserve, moves sequences and identity columns past values the source issued during the copy, writes the empty LOBs of NOT NULL LOB columns and restores their NOT NULL, in a filtered copy sets nullable foreign keys to NULL where their parent row is not copied, and adds foreign keys. A foreign key whose parent key could not be enabled is not added. This is logged as copy_scope (finish). If any step fails, the run is marked failed, naming it.

These steps do not run at all when the run is cancelled, or when a task fails with Continue on failure off. With it on, they still run after a failed mask_table task.

Whatever was left undone can be finished at the destination by hand. For n, use the source's current LAST_NUMBER for that sequence under preserve, or the offset under offset. An identity column's sequence is the one DBA_TAB_IDENTITY_COLS names for it. In place of ..., restate the identity's other options as the source has them, as DataMasque does:

Left undone	Statement
GENERATED ALWAYS	ALTER TABLE s.t MODIFY (c GENERATED ALWAYS AS IDENTITY (START WITH n ...))
An identity column's start	ALTER TABLE s.t MODIFY (c GENERATED BY DEFAULT ON NULL AS IDENTITY (START WITH n ...)), with BY DEFAULT or ALWAYS in place of BY DEFAULT ON NULL as the column is
A sequence's start	Draw once with SELECT s.q.NEXTVAL FROM DUAL, giving v. ALTER SEQUENCE s.q INCREMENT BY d, where d is n - v - i and i its own increment, draw once more, then set INCREMENT BY i back. Skip the rest if v + i is already at or past n
An ENABLE NOVALIDATE key	ALTER TABLE s.t ENABLE NOVALIDATE PRIMARY KEY, or ENABLE NOVALIDATE UNIQUE (c1, c2), since a key Oracle named at the source has a new name at the destination. This fails while rows break the key: remove them, or leave the key disabled
A NOT NULL LOB column	UPDATE s.t SET c = EMPTY_CLOB() WHERE c IS NULL (EMPTY_BLOB() for a BLOB), then ALTER TABLE s.t MODIFY (c NOT NULL). Do this only once the column's rows are all loaded, since each NULL then stands for an empty LOB
A nullable foreign key in a filtered copy	UPDATE s.child c SET c.c1 = NULL WHERE c.c1 IS NOT NULL AND NOT EXISTS (SELECT 1 FROM s.parent p WHERE p.p1 = c.c1), before adding the foreign key. For a key over several columns, set each nullable column to NULL and compare them all
A foreign key	ALTER TABLE s.child ADD CONSTRAINT n FOREIGN KEY (c1, c2) REFERENCES s.parent (p1, p2) ENABLE, with the source's ON DELETE and DEFERRABLE clauses, and ENABLE NOVALIDATE or DISABLE in place of ENABLE as the source has it. Leave out CONSTRAINT n for a name Oracle generated. For a parent in another schema, first GRANT REFERENCES ON s.parent TO <child owner>. This fails while rows break the key: fix them, or add it ENABLE NOVALIDATE

Copy mode privileges

On the source, the user needs CREATE SESSION, SELECT on every table the copy reads, and SELECT_CATALOG_ROLE (or SELECT ANY DICTIONARY).

On the destination, what the user needs depends on whose schema the copy creates in. The owning schema also needs quota on its default tablespace.

Needed for	Creating in the user's own schema	Creating in another schema
Connecting	CREATE SESSION	CREATE SESSION
Creating tables, their keys and indexes	CREATE TABLE	CREATE ANY TABLE, CREATE ANY INDEX, ALTER ANY TABLE
Creating sequences and identity columns	CREATE SEQUENCE	CREATE ANY SEQUENCE
Drawing from sequences	Nothing extra	SELECT ANY SEQUENCE
Loading rows	Nothing extra	INSERT ANY TABLE
Finishing the copy	Nothing extra	ALTER ANY TABLE, ALTER ANY SEQUENCE, and UPDATE ANY TABLE when a copied table has a NOT NULL LOB column
A foreign key to another schema's table	Nothing extra	GRANT ANY OBJECT PRIVILEGE, unless the user owns the parent table. Without it, the key is left out with a warning
A masked table's index management	Nothing extra	ALTER ANY INDEX, SELECT_CATALOG_ROLE

Without the privileges to finish the copy, the tables load but the finishing steps fail, and must be done by hand.

IBM Db2 for i

A Db2 for i copy follows the Oracle copy above, except:

• Databases. An IBM i system holds one database, so the destination must be a different system: a connection with the source's host is refused, whatever its port, so the preview page does not warn about it. The source and destination connections must name the same library.

• Column types. SMALLINT, INTEGER, BIGINT, DECIMAL, NUMERIC, REAL, DOUBLE, FLOAT, DECFLOAT, CHAR, VARCHAR, GRAPHIC, VARGRAPHIC (each with its CCSID), BINARY, VARBINARY, DATE, TIME and TIMESTAMP. LOBs (CLOB, BLOB, DBCLOB) are refused whatever Copy LOB data says, as is any other type. A generated column (row change timestamp, row begin or end, transaction ID, or generated expression) stops the run.

• Hidden columns are created IMPLICITLY HIDDEN.

• Keys and foreign keys keep their names, including names the system generated, as the catalog does not mark which names were generated. A disabled foreign key at the source stops the run. A foreign key to another library's table is created without a grant.

• Indexes. Descending indexes and UNIQUE WHERE NOT NULL indexes are created. Expression, sparse (CREATE INDEX ... WHERE) and encoded vector indexes are left out, with a warning.

• Sequences. Every sequence in the created tables' libraries starts where the block's sequences setting says, since a Db2 for i column default cannot draw from a sequence. Under preserve, a sequence starts one past the last value it issued, or at its START if it issued none. IBM i's catalog does not show an ALTER SEQUENCE ... RESTART value, so a sequence restarted since its last value starts from that last value, not from the restart value.

• Identity columns have no ON NULL option. Under preserve, each is moved on with ALTER TABLE ... ALTER COLUMN ... RESTART WITH n, where n is the source's next identity value.

• Finishing a copy by hand. For n, use the source's next value under preserve: for a sequence, MAXASSIGNEDVAL plus INCREMENT in QSYS2.SYSSEQUENCES, or START if MAXASSIGNEDVAL is null, or 0 when 0 is outside the sequence's range; for an identity column, NEXT_IDENTITY_VALUE in QSYS2.SYSPARTITIONSTAT. Under offset, use the offset.

  Left undone	Statement
  GENERATED ALWAYS	ALTER TABLE s.t ALTER COLUMN c SET GENERATED ALWAYS
  An identity column's start	ALTER TABLE s.t ALTER COLUMN c RESTART WITH n
  A sequence's start	ALTER SEQUENCE s.q RESTART WITH n
  A foreign key	ALTER TABLE s.child ADD CONSTRAINT n FOREIGN KEY (c1, c2) REFERENCES s.parent (p1, p2), with the source's ON DELETE and ON UPDATE rules. This fails while rows break the key: fix them first

• Privileges. On the source, the user needs SELECT (*OBJOPR and *READ) on every table the copy reads, and *EXECUTE on its library. On the destination:

  Needed for	IBM i authority
  Creating tables, their keys and indexes, and sequences	*EXECUTE and *ADD on the library, and *USE on the CRTPF, CRTLF and CRTDTAARA commands
  Loading rows and finishing the copy	Nothing extra when the user owns what the copy creates. Otherwise INSERT (*OBJOPR and *ADD) on the tables and ALTER (*OBJALTER) on the tables and sequences
  A foreign key	REFERENCES (*OBJREF) on the parent table, unless the user owns it
  A masked table's index management	INDEX (*OBJALTER) on the table, and *USE on the CRTLF command

• Filtering the copy. The keys the filter follows compare as Db2 for i compares them: a CHAR, VARCHAR, GRAPHIC or VARGRAPHIC key matches without its trailing blanks. A FOR BIT DATA key compares byte for byte. A key on a GRAPHIC or VARGRAPHIC column in a double-byte (non-Unicode) CCSID stops the run before the destination is touched. Declare the column in a Unicode CCSID such as 1200 so the filter can follow it.

• Not applicable: the ENABLE NOVALIDATE, deferrable, bitmap, and 12c items above.

Run Secret Options

A run secret is a critical component that controls the consistency of masked data generation. When the same run secret is used across multiple runs, DataMasque will generate consistent masked values, ensuring reproducibility when needed.

Note: To generate consistent masked values across multiple DataMasque instances, the Instance Secret should be disabled. Hashing may also be used. Please refer to the Consistent Masking guide for more information.

Configuration Options

The following options are available for configuring the run secret.

1. Securely Generated Random Run Secret

• Automatically generates a cryptographically secure random value for each run.
• Best choice when you want unique masked data for each run.
• No configuration needed - simply select this option.
• Each run will produce different masked values.

2. Specify Your Own Run Secret

• Allows you to input a custom run secret.
• Requirements:
  • Minimum length: 20 characters.
  • Maximum length: 128 characters.
  • Can include letters, numbers, and special characters.
• Use this option when you need reproducible masked data across multiple runs.
• On the same DataMasque instance, the same run secret will generate the same masked values for repeated runs.

3. Load from AWS Secrets Manager

Note: This option is not yet supported when DataMasque is deployed as a Snowflake Native App. Use a randomly generated or manually specified run secret instead.

Retrieves the run secret from AWS Secrets Manager. This is ideal for:

• Team environments where secret management is centralised.
• Maintaining consistent masking across different environments.
• Meeting security compliance requirements, by limiting secret access to specific IAM roles.

The secret selection is a combo box, which allows you to select from a dropdown of available secrets, or entering a secret ARN.

The AWS IAM Permissions used:

• secretsmanager:GetSecretValue is required to read the selected secret at masking time.
• secretsmanager:ListSecrets is optional, and only required to populate the dropdown list.
  • Without the ListSecrets permission, any ARN that your role has access to can still be manually specified, provided it can be accessed with the secretsmanager:GetSecretValue permission.
  • The expected ARN format is arn:aws:secretsmanager:<region>:<account-id>:secret:secret-name

Click the refresh icon to the right of the ARN selection to reload the list of run secret ARNs from AWS.

Note: Random Seed Priority: If both a random_seed value is in the rule set, and a run secret has been provided, the random_seed takes precedence.

Preview and confirm run

This screen shows the preview of the configured run. Check the run parameters here before proceeding with execution via the START RUN button. If the ruleset uses a feature your license does not include, the screen shows a warning and the START RUN button is disabled. See License warnings in the web UI. After the run has been started, you will be redirected to the Run logs page where you can monitor the run output and progress.

You can view a curl command for starting an equivalently configured run using the DataMasque API by clicking the VIEW RUN COMMAND button. For more information, see the Best Practices guide and API Reference

Run logs

The Run Logs screen displays a log of all historic runs, their statuses, and their individual log outputs. To access the Run Logs screen, choose the Run logs item from the main menu.

Run statuses

A run's final status indicates how it completed:

• Finished: the run completed with no warnings.
• Finished With Warnings (shown in orange): the run completed, but at least one message was logged at the Warning level. The DataMasque Warnings page catalogs the warnings DataMasque may generate, and how to disable the ones that can be disabled.
• Failed: the run stopped before completing; refer to the run log for the error.
• Cancelled: the run was cancelled before it completed.

Filtering runs

The Run Logs panel can be narrowed by Connection/Ruleset Name, Run Status and Mask Type. Click the Search button to apply the filters. To return every filter to its default and search again, click the Reset button.

The Run Status filter accepts any number of statuses at once. Select the statuses you want to see using the checkboxes in its dropdown, for example Finished and Finished With Warnings together. Select All to clear the selection and show runs of every status.

Run details

When a run is selected in the Run Logs panel, its details and log history are displayed in the Masking Run panel. While a run is still being executed, its log output will be streamed for continuous feedback on the run progress.

The following information is logged at the start of a run:

• The target database type, e.g. Oracle
• The version of the target database (where available), e.g.
  Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
• The 64-character SHA256 hash of the ruleset content

The run options used by this run can be found on the first log line.

Timing metrics

DataMasque records timing metrics during masking runs to help identify performance bottlenecks. These metrics are logged at the Debug log level and can be viewed in the run log viewer by setting the Display Log Level dropdown to Debug.

Database masking

For each table masked, DataMasque logs a per-table timing breakdown showing:

• DB time — time spent interacting with the database (fetch + write + bounds calculation)
• Masking time — time spent applying masking rules in memory
• Batch count — how many batches the table was processed in

At the end of the run, a job-level summary aggregates these timings across all tables.

File masking

For each file masked, DataMasque logs a per-file timing breakdown showing:

• IO time — time spent reading the source file and writing the masked output (read + write)
• Masking time — time spent applying masking rules in memory

At the end of the run, a job-level summary shows the total time and number of files masked.

Database masking job summary

On completion of a database masking run, the run log will display the following information:

• Masking run status: The final status of the run on completion. This will indicate whether the run completed successfully or failed.
• Started at: The time at which the masking run was started.
• Finished at: The time at which the run completed, failed or was cancelled.
• Total time: The total time taken for the run.
• Total tables masked: The total number of tables that were successfully masked.
• Total columns masked: The total number of columns that were successfully masked.
• Total rows masked: The total number of rows that were successfully masked.

An example of a successful database masking run log can be seen below:

When a run fails, if continue_on_failure was not enabled, the total tables, columns and rows masked will be displayed as 0. While some rows or column may be masked, it cannot be determined which particular rows or columns are masked on a failed run, so all tables should be considered unmasked.

If continue_on_failure was enabled for the run, the tables, columns and rows will instead reflect the total numbers of tables, columns and rows masked in successful mask_table and mask_unique_key tasks.

File masking job summary

On completion of a file masking run, the run log will display the following information:

• Masking run status: The final status of the run on completion. This will indicate whether the run was completed successfully or failed.
• Started at: The time at which the masking run was started.
• Finished at: The time at which the run completed, failed or was cancelled.
• Total time: The total time taken for the run.
• Total files masked: The total number of files that were successfully masked. On failed runs, this will reflect the number of files that were masked by successful file masking tasks.

An example of a successful file masking run log can be seen below:

An example of a failed file masking run log can be seen below:

File discovery summary

At the end of a run containing a run_file_data_discovery task, the run log shows a summary of every file that was considered for data discovery: counts per file extension for the files that were scanned, and for the files that were not scanned, grouped by the reason each file was not scanned. The reasons are the same skip_reason values used in the run report, which remains the complete per-file record. Each section of the summary lists at most five extensions, with a final line aggregating any remainder; the run report always holds the full per-file breakdown.

The summary is omitted in two cases: when the discovery task never ran, because an earlier task failed without the Continue on failure option, and when the source contained no files. In both cases the run log states the reason.

An example summary:

252 files were considered for data discovery:
9 files were scanned, 243 files were not scanned.

Scanned:
    .csv: 8 files
    .ndjson: 1 file

File type unsupported by data discovery:
    .jpg: 62 files
    .png: 20 files
    .gif: 5 files
    .bmp: 4 files
    .tiff: 4 files
    3 other extensions: 3 files

File type could not be determined:
    No extension: 2 files

Did not match any include filter:
    .parquet: 43 files

Matched a skip filter:
    .json: 100 files

If a run_file_data_discovery task does not complete, the files it had selected but not yet scanned cannot be reported individually. The summary ends with a line counting them:

3 files were not reached because a data discovery task did not complete.

Those files are not included in the scanned and not scanned counts above, and have no row in the run report. The run log shows the error that ended the task.

DataMasque warns when files could not be scanned, and when no files were scanned at all. See the Files Could Not Be Scanned Warning and the No Files Scanned Warning.

Run Report

The run report is a CSV file that holds more detailed information about the run that was performed.

A run produces a report when it has something to report about individual files: a run with a run_file_data_discovery task, or a file masking run where a license limit left a file out entirely, or stopped a JSON or XML document partway. A tabular file cut short by the per-file row limit is counted rather than listed, so it does not produce a report row.

For a discovery task the report lists every file that was considered: the files that were scanned, and any files that were not scanned, with the reason each file was skipped or could not be read.

The run report will be generated within the datamasque_admin-server_1 container, under the directory /files/user/file_task_reports/<YYYY>/<MM>/, with the file name <run-id>.csv.
The directory structure /<YYYY>/<MM> corresponds to the year and month digits representing when the run started.
For example, the report for the run with run ID 123, that started on April 1st, 2024, would be located at /files/user/file_task_reports/2024/04/123.csv.

The report will contain the following information about each file:

• path: The path of the file that was discovered.
• file_size: The size of the file.
• file_type: The file type discovered if the extension is a supported file type: avro, avsc, csv, json, ndjson, parquet, xml.
• file_name: The name of the file.
• file_extension: The file extension.
• skip_reason: The reason the file was skipped, or was masked only in part, detailed below.

The following are the reasons and explanations for files to be skipped during a run_file_data_discovery task. They are listed in the order their sections appear in the file discovery summary:

Reason	Explanation
File archived with Glacier	On Amazon S3, the file has a Glacier storage class and cannot be read. See the Glacier Files Warning and file connections.
File type unsupported by data discovery	The file's extension is not a type supported by file discovery. Supported types are: CSV, JSON, NDJSON, Parquet. DataMasque determines a file's type by its extension, not its content.
File type could not be determined	The file has no extension, so DataMasque could not determine its type.
An error occurred while running data discovery	An unexpected error occurred during the discovery process.
File contains a column with unsupported data type	For Parquet files, not all data types are supported by file discovery yet. See Supported Parquet Column data types for more information.
File was empty	The file contained no data.
Did not match any include filter	The file did not match any include filter in the ruleset.
Matched a skip filter	The file was skipped due to matching a skip filter in the ruleset.

These reasons appear only in a file masking run, where a license limit left a file out or masked only part of it:

Reason	Explanation
Exceeds Free Trial file limit	The file was past the Free Trial's file-count limit, so it was not masked. See Licensing.
Exceeds Free Trial size limit	The file was larger than the Free Trial's per-file size limit, so it was not masked. See Licensing.
Exceeds Sandbox file limit	The file was past the Sandbox license's file-count limit, so it was not masked. See Licensing.
Masked to the Sandbox value limit	The file was masked up to the Sandbox license's per-file value limit. The values past it were written through unmasked.

There are several ways you can retrieve the run report:

• On the Run Logs page, select the run and then click the DOWNLOAD RUN REPORT button at the bottom of the page.

• Make a GET request to the endpoint /api/runs/{id}/run-report/.

• SSH into the machine running DataMasque and copy the report using the following command:

  docker cp datamasque_admin-server_1:/files/user/file_task_reports/<YYYY>/<MM>/<run-id>.csv <path-on-local-filesystem>
  

  Replacing the <YYYY>, <MM>, <run-id>, and <path-on-local-filesystem> as appropriate.

Connection and ruleset snapshots

Snapshots of the connection and ruleset are kept for every masking run, maintaining an historical record of the exact configuration that was used for each run. Connection and ruleset snapshots can be viewed for the selected masking run by clicking on the connection or ruleset name displayed on the run detail panel.

A modal window will open to display the snapshots, as captured at the time of masking run creation. The snapshot status indicates whether the current connection or ruleset configuration has been changed since this snapshot was taken. Clicking the edit link will allow you to edit the current connection or ruleset corresponding to the displayed snapshot.

There are three different statuses that may be displayed for a snapshot:

• current: The details have not been changed since the run was performed.
• modified: The details have been changed since the run was performed. The details shown no longer reflect the current state of the connection or ruleset.
• deleted: The connection or ruleset no longer exists. If this is the case, there will be no option to edit.

Downloading a run log

While hovering over a run in the Run Logs panel, a download button will be shown. Clicking on this button will start a download containing the logs for this masking run.

Downloading a sensitive data discovery report

When a run_data_discovery task is included in the masking ruleset, the resulting report for each run can be downloaded by clicking either the shield icon on the run row in the Run Logs list, or the Discovery Report chip on the Masking run detail panel. The report will be downloaded in CSV format and may be opened in a text editor or spreadsheet viewer such as Microsoft Excel. See Sensitive Data Discovery for more details.

Downloading a discovery config snapshot

For discovery runs, the discovery configuration snapshot — a record of the exact discovery rules used for that run — can be downloaded by clicking the document icon on the run row in the Run Logs list. The snapshot is downloaded as a .zip archive. This button is shown only for discovery runs (not masking runs), and may not be available for older discovery runs.

Cancelling a run

If you wish to cancel a run, you may do so with the following steps:

1. Select the run you wish to cancel from the list in the 'Run Logs' panel.
2. Click the CANCEL RUN button at the bottom of the screen.
3. After clicking YES on the confirmation dialog, the run status will be updated to cancelling and DataMasque will proceed to stop the run's in-progress masking tasks.
4. Once all the run's tasks have been stopped, the run status will be updated to cancelled.

Note: For database masking runs, cancelling a run will leave the database partially masked. DataMasque tries to restore any temporary changes it made to the database schema, indexes, and so on, but this may not always occur, depending on whereabouts in the run the cancellation occurred and whether cancellation could execute cleanly.

See also the Database Connections page for information about the side-effects of cancelling a run on particular databases.

Note: For file masking runs to an AWS S3 destination, cancelling a run can leave an incomplete multipart upload in the destination bucket. See Incomplete multipart uploads for the lifecycle rule that removes it.

Run History

When a masking run completes, regardless of success or failure, DataMasque will attempt to record the result of a masking run in the database being masked, or in a file in the base directory of the destination connection for file masking runs. A database copy records it in the destination: see Run history in copy mode.

History is only written for masking runs that actually mask data (assuming there is data to mask, no errors occur, and so on). For example, dry runs or schema/data discovery runs will not be recorded in the history.

Database masking runs: Run History table

The table DATAMASQUE_RUN_HISTORY is created if it does not exist on the default schema for the user performing the masking. This table acts as a record of masking runs, and to help identify that a masking run did actually occur. It does not guarantee that any amount of data has actually been masked.

The format of the table is as follows:

Column Name	Data Type	Description	Example Value
completion_time	DATETIME or TIMESTAMP	Time the run completed, in UTC	2024-01-15 15:23:11
dm_version	VARCHAR(80)	The DataMasque version	2.18.0
run_status	VARCHAR(15)	The result of the masking run	finished, finished_with_warnings, failed, or cancelled
run_id	NUMBER or INT	The ID of the run (as used in API or run logs)	156
ruleset_uuid	VARCHAR(36)	A unique identifier for each ruleset	516aef12-98ce-5192-a9c8-154a8c9b8d12
ruleset_content_sha256	VARCHAR(64)	A hash of the contents of the ruleset used for masking. This hash is logged at the start of a run log.	64-character SHA256 Hash
connection_snapshot	VARCHAR(150)	Summary of the database connection, including type, username and host. Does not include the password.	MySQL - mysql_user@mysql-host:3306
run_hash	VARCHAR(64)	A hash to compare with the hash of the masking run	64-character SHA256 Hash
metadata	See note below	Extra information specific to each masking run	Currently unused

Note: the metadata column is stored as a CLOB for Db2 for i, Db2 LUW, and Oracle, JSONB for Postgres, JSON for MySQL/MariaDB, and NVARCHAR(MAX) for MSSQL.

Note: Run history table is not supported for Amazon DynamoDB and Amazon Redshift databases.

A new row is added for every masking run, assuming the ruleset is valid and a run finishes.

Insertion and creation of the DATAMASQUE_RUN_HISTORY table is automatic, and will not cause problems if any part of it fails. For example, if the user for the database connection of the masking run does not have the access required to create a table, this operation will silently fail, and masking will not be affected.

The DATAMASQUE_RUN_HISTORY table is automatically excluded from any masking runs, schema discovery, and so on.

For MongoDB, Amazon DocumentDB and Azure Cosmos DB connections, run history is stored in a DATAMASQUE_RUN_HISTORY collection in the connection's database, rather than a table. Each masking run adds a document with the same fields as the run history table, plus a run_history_version field (currently "1.0"). metadata is stored as an embedded document. The collection is created automatically, and is excluded from masking and discovery in the same way as the run history table.

For Apache Cassandra connections, the table is created in the connection's keyspace, with a run_history_version column and completion_time stored as text. The table name is case-sensitive, so quote it in CQL: "DATAMASQUE_RUN_HISTORY".

File masking runs: Run History file

DataMasque writes the run history to a file .datamasque_run_history.ndjson in the base directory of the destination connection. The format of the file is NDJSON, with one line per history entry. Each history entry is a JSON object with the following fields:

Field Name	Data Type	Description	Example Value
run_history_version	string	Version of the run history format used for this entry. Currently always "1.0"	"1.0"
completion_time	string	Time the run completed, in UTC	"2024-01-15 15:23:11.508877+00:00"
dm_version	string	The DataMasque version	"2.20.0"
run_status	string	The result of the masking run	"finished", finished_with_warnings, "failed", or "cancelled"
run_id	int	The ID of the run (as used in API or run logs)	156
ruleset_id	string	A unique identifier for each ruleset	"516aef12-98ce-5192-a9c8-154a8c9b8d12"
ruleset_hash	string	A hash of the contents of the ruleset used for masking. This hash is logged at the start of a run log.	64-character SHA256 Hash
connection_snapshot	string	Summary of the file connection, including type and base directory. Does not include any credentials.	"AWS S3 Connection - path my/base/directory in bucket mybucket"
run_hash	string	A hash to compare with the hash of the masking run	64-character SHA256 Hash
metadata	object or null	Extra information specific to each masking run	Currently unused

As for the database history table, creation or modification of the file is done automatically and will not cause the masking run to fail should there be an error writing the file.

The .datamasque_run_history.ndjson file is automatically excluded from any masking runs or data discovery, even if matched by a glob statement such as glob: *.ndjson.

Verification of the ruleset hash in the run history

As the ruleset content is not captured in the history table or file, manual validation is required to compare it against the ruleset content hash (the ruleset_content_sha256 column in the table, or the ruleset_hash field in the file).

To ensure consistency, ensure that the ruleset content is encoded in UTF-8 format and includes any leading or trailing spaces. Then hash the ruleset content with the SHA256 algorithm. The result should match the hash stored in the DATAMASQUE_RUN_HISTORY table or .datamasque_run_history.ndjson file.

Simultaneous runs

To the same database

Simultaneous runs to the same database are not currently supported in DataMasque, as simultaneous masking can result in data being incorrectly masked. When there is a masking run in the status of queued, running or cancelling, subsequent masking runs to the same database connection cannot be scheduled.

If you wish to mask multiple tables in the same database simultaneously, it is recommended to use parallel tasks in a single ruleset.

To different databases

Simultaneous runs to different databases are supported in DataMasque.

For file connections

Simultaneous runs to the same source or destination file connection are not supported.