Ruleset YAML specification

New to rulesets? Work through the Masking Tutorial first. This page is a reference, not a lesson.

The ruleset YAML configuration provides instructions that DataMasque will follow when performing a masking run against a target database. Rulesets are comprised of one or more tasks, which can contain many different types of instructions. The most common use case is the application of tasks for masking sensitive data in database tables with masking rules.

• Reference pages
• Ruleset properties
• Example ruleset
• Schema versioning
• Writing comments

Reference pages

This page covers the top-level ruleset keys, a complete example, schema versioning and comments. Every other part of the specification has its own page.

Page	Contents
Database tasks	run_data_discovery, run_schema_discovery, subset, truncate_table
Table masks	The mask_table task
Mask a primary key or unique key	The mask_unique_key task (deprecated)
Build a temporary table	The build_temp_table task
Run SQL	The run_sql task
Parallel and serial tasks	The parallel and serial tasks
File tasks	File data discovery, and choosing which files to mask
Object file masks	The mask_file task
Tabular file masks	The mask_tabular_file task, and Parquet data types
Masking functions	Index of every mask function
Definitions	task_definitions, rule_definitions, mask_definitions
Advanced ruleset features	Skip values, frozen values, trigger and index management
Deterministic masking	Repeatable masking across runs and databases
Deterministic masking with files	Repeatable masking across file data sources
Regular expressions	Common patterns and worked examples
Date parsing	Date and time format directives
Choosing unique key masking	Which unique mask to use

Ruleset properties

The following properties are specified at the top-level of a ruleset YAML file:

• version (required): The schema version used by this ruleset. The default value present in the editor when creating a new ruleset is generally the value that you should be using. See Schema Versioning for more information.

• name (deprecated; will be removed in release 3.0.0): A unique name that will be used to refer to your ruleset. This field may only contain alphanumeric characters, underscores, and spaces, and its length is limited to 64 characters. Leading and trailing whitespace is trimmed.

• tasks (required): A list of tasks to be performed in order on the target database. See Database Task Types for the comprehensive list of the available task types and their associated parameters.

• task_definitions (optional): A list of task definitions to be referenced from the ruleset's list of tasks through YAML anchors and aliases.

• rule_definitions (optional): A list of rule definitions to be referenced from a list of rules through YAML anchors and aliases.

• mask_definitions (optional): A list of mask definitions to be referenced from a list of masks through YAML anchors and aliases.

• skip_defaults (optional): See Default values to skip.

• disable_warning_on_duplicate_rules (optional) If multiple tasks in the same ruleset attempt to mask the same table or column DataMasque will show a warning. Set this to true to suppress the warning. Defaults to false.

• auto_manage_triggers (optional): When set to true, DataMasque will automatically detect and disable database triggers on the target table, or ON UPDATE expressions on the table's columns, before masking. After masking, triggers and expressions are enabled. This can help prevent trigger-related issues during the masking process. This setting applies to all mask_table and mask_unique_key tasks in the ruleset but can be overridden at the task level. For detailed information on database support and usage, see Automatic trigger management. Defaults to true for supported databases.

• auto_truncate_value (optional): When set to true, a masked value that is wider than its target string column is truncated to fit after all masks in a rule have been applied, instead of failing the masking run. Only string-type columns are truncated; binary and other column types are left unchanged. A warning is shown for each affected column (see Value Truncation Warning). This applies to mask_table column rules and can be overridden per rule. Key and unique-constrained columns are never truncated, to preserve referential integrity and uniqueness: auto_truncate_value has no effect on a column in the task's key or covered by a unique constraint or unique index. Truncating such a column could collide two values, so an over-width value is handled as it would be without the option (a value-too-long failure on engines that enforce the limit), and a warning is emitted when the skip applies. Only takes effect on database engines that expose fixed column widths, which is every supported engine except the schemaless document stores (MongoDB, Amazon DynamoDB, Amazon DocumentDB and Azure Cosmos DB). Defaults to false.

• disable_warning_on_auto_truncate_value (optional): When set to true, the warning shown when a masked value is truncated by auto_truncate_value is suppressed. This applies to all rules in the ruleset but can be overridden per rule. Defaults to false.

• pre_calculate_row_counts (optional): When set to true, DataMasque reports the number of rows each mask_table and mask_unique_key task will mask in the run logs before masking begins, so the scale of each table is visible before any rows are changed. The count is exact, not an estimate, and reuses a count DataMasque already performs, so enabling this issues no additional queries. Every task reports its own tables, including tasks nested in parallel and serial blocks.

  For mask_table tasks, the count honours the task's joins and where clause, and the Max rows run option, so it reflects the rows that will actually be masked rather than the size of the whole table. Child tables masked alongside the target table are reported too. The count is reported after DataMasque has determined the batches to mask, which on a large table can itself take some time after the task starts. On MongoDB, Amazon DocumentDB, Azure Cosmos DB and Amazon DynamoDB the count covers the whole collection or table, as their row counts cannot be filtered by a where clause or joins.

  For mask_unique_key tasks, the count covers the target table only, and is the number of rows that will be given new key values. Rows with a NULL in any key column are set to NULL rather than counted, and updates cascaded to referencing tables are not counted. The Max rows run option does not apply, as the task masks the whole key regardless.

  A table with no rows to mask reports zero rather than nothing at all. The counts are also reported during a dry run, showing what a live run would mask.

  Defaults to false.

• random_seed (optional): Deprecated in favour of the Run secret option. See Freezing random values.

Example ruleset

The following ruleset provides an example for replacing the last_name column of every row in the users table with the fixed value "redacted last name":

# My ruleset.yml
version: '1.0'
tasks:
  - type: mask_table
    table: users
    key: user_id
    rules:
      - column: last_name
        masks:
          - type: from_fixed
            value: 'redacted last name'

Note: The tasks must be indented using two spaces. For example:

tasks:
  - type: mask_table
^^

- type: mask_table is indented two spaces from the parent tasks:

Important note on case-sensitivity: For all tasks except run_sql, database identifiers, such as table and column names, should be referenced as you would otherwise reference them in an SQL query. When masking a case-sensitive database, identifiers must be referenced in the ruleset using the correct case.

To refer to a case-sensitive table or column in a database, the identifier must be enclosed in double quotation marks. However, when writing YAML, quotations are used to denote a string value, so any enclosing quotations are not considered as part of the value. As such, it is necessary to enclose the entire name - including double quotation marks - in an outer set of single quotation marks. For example:

# Case-sensitive table name; enclosed in both single and double quotations.
table: '"CaseSensitiveTableName"'

To refer to a case-sensitive table in a schema, the schema name must also be enclosed in quotation marks if the schema name is case-sensitive. The entire combination of schema and table name must be enclosed in single quotation marks. For example:

# Case-sensitive schema and table name; enclosed in both single and double quotations.
table: '"CaseSensitiveSchemaName"."CaseSensitiveTableName"'

If referencing a combination of table and column, you will need have quotation marks around both the table and column names within the surrounding quotation marks. For example:

# Case-sensitive table and column name; enclosed in both single and double quotations.
column: '"CaseSensitiveTable"."CaseSensitiveColumn"'

Identifier names containing double quotation marks, backslashes, periods, and whitespace should always be enclosed in double quotation marks. Also, literal double quotation marks and backslashes must be preceded by a backslash:

# Case-sensitive table and column name containing special characters.
column: '"Case\"Sensitive\\Table"."Case.Sensitive Table"'

Notes:

• For Redshift databases, DataMasque follows PostgreSQL's rules for case-sensitivity of quoted/unquoted identifiers if enable_case_sensitive_identifier is true, and treats all identifiers as case-insensitive if enable_case_sensitive_identifier is false. See: https://docs.aws.amazon.com/redshift/latest/dg/renablecasesensitiveidentifier.html

Schema versioning

Schema changes to the DataMasque ruleset specification are tracked using the version field of the ruleset. The version number consists of two fields in the format major.minor. Minor version increments reflect backwards-compatible changes to the schema, whereas major version increments represent breaking changes which will require some form of migration from previous versions. Wherever possible, DataMasque will handle such migrations for you automatically when you upgrade.

Each release of DataMasque only supports the most recent major ruleset version at the time of release. As such, the major schema version of your rulesets must equal the major version supported by your DataMasque release. The minor schema version of your rulesets must be equal to or less than the minor version supported by your DataMasque release.

The ruleset schema version supported by this release of DataMasque is "1.0".

Writing comments

While creating a ruleset, it is possible to write comments in the ruleset. A commented block is skipped during execution, and it helps to add description for specified ruleset block.

If you begin a line with # (hash symbol), all text on that line will become a comment.

version: '1.0'
# This line will become a comment.
tasks:
  - type: mask_table

---

If you place # on a line, all text after that # on that line will become a comment. Any text before it will still be part of the ruleset.

version: '1.0'
tasks:
  - type: mask_table # The name of this task type will not be affected by this comment.

---

In the DataMasque ruleset editor, the shortcut key combination for commenting ruleset blocks is CTRL + / on Linux and Windows and ⌘ + / for Mac operating systems. If your cursor is on a line and this shortcut is used, the entire line will be commented out. Highlighting multiple lines at once will cause all highlighted lines to be commented out.