Advanced ruleset features

Deterministic masking has its own pages: Deterministic masking and Deterministic masking with files.

• Default values to skip
  • Disabling skip_defaults
• Freezing random values
• Automatic trigger management
  • Automatic trigger management: database support
  • How automatic trigger management works
  • Configuration levels
  • Trigger management default behaviour
  • Trigger management examples
  • Trigger management considerations
• Automatic index management
  • Automatic index management: database support
  • How automatic index management works
  • Index management default behaviour
  • Index management considerations
  • Index management examples

Default values to skip

You can specify a list of skip_defaults values at the ruleset level:

version: '1.0'
skip_defaults:
  - null
  - ''
  - matches: 'administrator_.*'
tasks:
  - type: mask_table
    table: users
    key: id
    rules:
      - column: role
        masks:
        - type: from_fixed
          value: 'customer'

These will be used as the skip configuration for any masking rule that does not have skip explicitly set.

Disabling skip_defaults

To disable the skip_defaults for a particular column, set the skip option for that column to an empty array: skip: [].

Example

• role column:
  • The role column does not have a specific skip rule defined, so it will use the skip_defaults specified at the top of the ruleset. This means that any null or empty string ('') values in the role column will not be masked; they will be skipped over.
• details column:
  • The details column, however, explicitly sets skip: [], which overrides and disables the skip_defaults for this column. As a result, even if the value is null or an empty string (''), it will still be masked according to the rule provided (in this case, replaced with the string REDACTED).

version: '1.0'
skip_defaults:
  - null
  - ''
tasks:
  - type: mask_table
    table: users
    key: id
    rules:
      - column: role
        masks:
        - type: from_fixed
          value: 'customer'
      - column: details
        skip: []
        masks:
        - type: from_fixed
          value: 'REDACTED'

Summary

• skip_defaults: Defines which values (like null or '') should not be masked by default.
• skip: []: Overrides the skip_defaults for a particular column, ensuring that every value in that column (even those normally skipped) will be masked.

---

Freezing random values

A run_secret string can be provided as an option when starting a run. If you provide a run_secret, then all randomly chosen and generated values will be exactly the same for multiple runs with the same secret. This can be useful for testing or retaining consistency across multiple masking runs. The run_secret can consist of any sequence of at least 20 characters, however it is recommended to use a randomly generated string.

Notes:

• Output values will be frozen only across identical repeated masking runs. Variations in database content (such as inserted or deleted rows) or Run Options (i.e. Batch Size, Max rows) can still result in different outputs even when using the same run_secret. Additionally, using a non-unique key may result in non-deterministic results. To retain consistent masking results across variable data sources, use the Deterministic masking features.
• As the parallelised unloading of Amazon Redshift data into S3 files cannot guarantee consistent ordering between runs, a fixed run_secret may still result in different outputs between masking runs on Amazon Redshift databases. To retain consistent masking results with Amazon Redshift, use the Deterministic masking features.

---

Automatic trigger management

The auto_manage_triggers parameter allows DataMasque to automatically manage database triggers during masking operations. When enabled, DataMasque will detect triggers on the target table, temporarily disable them before masking, and re-enable them after the masking operation completes.

For MySQL and MariaDB databases, the triggers are dropped and recreated, as these databases do not support trigger disabling.

Also for MySQL and MariaDB databases, this feature also manages column ON UPDATE expressions (computed columns). Before masking, ON UPDATE expressions for columns on the target table are removed, then re-added after masking.

This feature is particularly useful when database triggers interfere with the masking process, such as:

• Triggers that enforce business rules that conflict with masked data.
• Triggers that perform additional data modifications during updates.
• Triggers that log or audit changes that might create noisy logs due to many updates during masking.

Automatic trigger management: database support

The auto_manage_triggers feature is currently supported for the following database types:

• Db2 for i
• Db2 LUW
• MariaDB (includes column ON UPDATE expression management)
• Microsoft SQL Server
• MySQL (includes column ON UPDATE expression management)
• Oracle
• PostgreSQL

For unsupported database types, setting auto_manage_triggers: true will raise an error indicating that the feature is not supported for that database type.

How automatic trigger management works

When auto_manage_triggers is enabled:

1. DataMasque queries the database to identify all triggers defined on the target table. For MySQL and MariaDB, column ON UPDATE expressions are also detected.
2. All detected triggers are temporarily disabled (or dropped) before masking begins. For MySQL and MariaDB, ON UPDATE expressions are temporarily removed from column definitions.
3. The normal masking process proceeds with triggers and computed columns disabled.
4. Triggers and computed columns are automatically enabled (or recreated) after masking completes, regardless of whether the masking operation succeeded or failed.

Configuration levels

The auto_manage_triggers parameter can be configured at two levels:

• Ruleset level: Applies to all tasks in the ruleset
• Task level: Overrides the ruleset-level setting for specific tasks

When specified at the ruleset level, all mask_table and mask_unique_key tasks will inherit this setting unless explicitly overridden at the task level.

Trigger management default behaviour

By default, auto_manage_triggers is true for supported databases, meaning triggers are automatically managed. You can disable this feature by setting auto_manage_triggers: false either at the ruleset level (for all tasks) or at the task level (for specific tasks).

Trigger management examples

# Example 1: Enable for entire ruleset
version: "1.0"
auto_manage_triggers: false  # Applies to all tasks

tasks:
  - type: mask_table
    table: users  # Inherits auto_manage_triggers: false
    key: user_id
    rules:
      - column: email
        masks:
          - type: from_random_text

  - type: mask_table
    table: orders
    key: order_id
    auto_manage_triggers: true  # Override to enable for this task
    rules:
      - column: order_id
        masks:
          - type: from_random_number

# Example 2: Selective disablement
version: "1.0"
# auto_manage_triggers not specified, defaults to true

tasks:
  - type: mask_table
    table: products  # Uses default (true)
    key: product_id
    rules:
      - column: name
        masks:
          - type: from_random_text

  - type: mask_unique_key
    table: accounts
    auto_manage_triggers: false  # Disable only for this task
    target_key:
      - column: account_id

Trigger management considerations

• Enabling/disabling triggers adds a small overhead to the masking process.
• The database user must have appropriate permissions to disable and enable triggers.
• For MySQL and MariaDB, the database user must also have permissions to modify column definitions (ALTER TABLE) to manage ON UPDATE expressions.
• Be aware that disabling triggers may bypass important business logic or data integrity checks.
• For MySQL and MariaDB, temporarily removing ON UPDATE expressions may affect timestamp or other automated column updates.
• Trigger and computed column management operations are logged in the DataMasque run output for auditing purposes.

Automatic index management

The auto_manage_indexes parameter allows DataMasque to automatically manage non-unique indexes during masking operations. When enabled, DataMasque will detect non-unique indexes on the target table that contain columns being masked, disable or drop them before masking, and re-create or re-enable them after the masking operation completes.

This feature is particularly useful to avoid the overhead of index management during masking to improve performance.

Automatic index management: database support

The auto_manage_indexes feature is currently supported for the following database types:

• Databricks Lakebase
• Db2 for i
• Db2 LUW
• MariaDB
• Microsoft SQL Server
• MySQL
• Oracle
• PostgreSQL
• Snowflake (hybrid tables)

For unsupported database types, setting auto_manage_indexes: true will raise an error indicating that the feature is not supported for that database type.

How automatic index management works

When auto_manage_indexes is enabled:

1. DataMasque queries the database to identify all non-unique indexes defined on the target table.
2. All detected indexes are filtered and those that contain columns being masked are temporarily disabled (or dropped) before masking begins.
3. The normal masking process proceeds with indexes disabled.
4. Indexes are automatically enabled (or recreated) after masking completes, regardless of whether the masking operation succeeded or failed.

Index management default behaviour

By default, auto_manage_indexes is true for supported databases, meaning indexes are automatically managed. You can explicitly disable this feature by setting auto_manage_indexes: false in your task configuration.

Index management considerations

• Enabling/disabling indexes adds a small overhead to the masking process.
• Snowflake indexes take 4-5 seconds to create and only one can be created at a time.
• The database user must have appropriate permissions to disable and enable (or drop and create) indexes.
• Index management operations are logged in the DataMasque run output for auditing purposes.
• For Oracle databases indexes will be disabled and enabled using UNUSABLE in mask_table tasks for better performance, but as mask_unique_key tasks can create indexes, existing indexes will be dropped and re-created to avoid conflicts.

Where possible all types of indexes are handled including indexes with a function/expression, multiple columns and column ordering.

A non-exhaustive list of additional index types we support:

• Databricks Lakebase - BTREE, GIN and partial indexes (with WHERE clauses)
• Db2 LUW - CLUSTER, ALLOW REVERSE SCANS
• MariaDB/MySQL -FULLTEXT, INVISIBLE, IGNORED
• Microsoft SQL Server - COLUMNSTORE
• Oracle - BITMAP
• PostgreSQL - BTREE, GIN and partial indexes (with WHERE clauses)

Index management examples

A mask_table task with index management on:

- type: mask_table
  auto_manage_indexes: true
  table: driversLicence
  key: ID
  ...

A mask_unique_key task with index management on:

- type: mask_unique_key
  auto_manage_indexes: true
  table: car_registration
  target_key:
    - column: number_plate
  ...

A mask_table task without specifying automanageindexes. Default to true for supported databases, so indexes may be disabled when masking this table.

- type: mask_table
  table: driversLicence
  key: ID
  ...