Deterministic masking

If you wish to make the masked values deterministic based on one or more input values, you can provide hash columns in a ruleset. DataMasque will use the hashed value(s) from the specified sources as one of the random generator inputs (along with the instance seed and the run secret). This allows for consistent output values to be generated even across different databases or inside documents.

For database or tabular file masking, hash_columns can be specified at two levels:

• Task level: Provides default deterministic behavior for all columns in the table
• Column level: Completely replaces the task-level default for specific columns (does not append to it), or can be set to null/[] (empty array) to disable deterministic masking for that column

For file masking, the parameter is called hash_sources and the hash used for the entire file.

DataMasque will use the hashed value(s) as one of the random generator inputs (along with the instance seed and the run secret). This allows for consistent output values to be generated even across different databases, inside documents (XML/JSON) or when masking files.

Changes to hashing values in 2.8.0: Prior to DataMasque 2.8.0, values to hash were converted to strings then hashed. This may have resulted in duplicate output values when hashing on null. From version 2.8.0, hashing occurs on binary representations of data. This means that hashed outputs from DataMasque 2.8.0 onwards will differ to those generated by prior versions.

Changes to hashing complex values: Hash sources that resolve to objects or arrays (for example JSON objects or lists) are serialized to JSON with sorted keys before hashing. This can change deterministic outputs compared with earlier versions for hash_columns or hash_sources that hash on complex values.

• Deterministic masking with databases or tabular files
  • Example: Task-level hash_columns
  • Example: Task-level with column-level override
  • Example: Task-level with column-level disable
  • Example: Column-level hash_columns
  • Example with case_transform
  • Example with trim
  • Example with source: self
  • Example with xpath
• Seeding masking from a table reference (table_reference)
  • Worked example: one entity, two systems, keyed differently
  • Keys
  • on_missing behaviour
  • Reusing a reference across rules
  • Supported key and value types
• Materialising a large map into the target database (build_temp_table)

Deterministic masking with databases or tabular files

Hash columns can be specified at the task level to provide a default deterministic behavior for all columns being masked in that table. Individual columns can then override this default by specifying their own hash_columns, or disable deterministic masking entirely by setting hash_columns: null or hash_columns: [] (empty array).

Each hash_columns item can either be just the name of the column, or can specify a json_path or xpath to fetch a hash value from inside JSON or XML, respectively. For more information on json_path or xpath please refer to the JSON documentation/XML documentation. For JSON/XML hash_columns the following logic is applied:

• If the value extracted from the json_path/xpath is null the hash will be performed on the null value.
• If the value does not exist at the json_path/xpath the hash will be performed on null.
• If the JSON/XML data cannot be decoded (is invalid) the hash will be performed on the entire column, even if null.

hash_columns can also trim (strip) leading/trailing white space by setting the trim parameter to true.

Note Since null is the default fallback for invalid data, this may result in unexpected repeated masked values. Warnings will be added to the run logs when this fallback occurs. Both json_path and xpath optional parameters cannot be set for the same hash column. It is possible to hash a column on to itself (i.e. hash_columns contains the current column). If you hash a column on itself, and that is used as the hash for other columns, the mask on the hash column should come last in the rules. If not, that column will use the pre-masked value as the hash and other columns use the post-masked value, meaning different values will be generated in the same row for the same mask definition.

Example: Task-level hash_columns

This example demonstrates task-level hash_columns where all columns in the table use the same deterministic behavior based on user_id:

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    hash_columns:
      - user_id
    rules:
      - column: email
        masks:
          - type: imitate
      - column: phone
        masks:
          - type: imitate

In this example, both email and phone columns will be masked deterministically based on the user_id column. Rows with the same user_id will always get the same masked values.

Example: Task-level with column-level override

This example shows how to override the task-level hash_columns for specific columns:

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    hash_columns:
      - user_id  # Default for all columns
    rules:
      - column: email
        masks:
          - type: imitate  # Uses task-level hash_columns (user_id)
      - column: backup_email
        hash_columns:
          - backup_email  # Override: hash on itself instead
        masks:
          - type: imitate

In this example, email uses the task-level hash (based on user_id), while backup_email overrides to hash on itself.

Example: Task-level with column-level disable

This example shows how to disable deterministic masking for specific columns while keeping it enabled for others:

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    hash_columns:
      - user_id
    rules:
      - column: email
        masks:
          - type: imitate  # Deterministic (uses task-level hash_columns)
      - column: temp_token
        hash_columns: null  # Disable hashing - completely random
        masks:
          - type: imitate

In this example, email will be deterministically masked, but temp_token will be completely random for each row.

Example: Column-level hash_columns

This example will mask the date_of_birth column with a date value that has been deterministically generated based on the hash of date_of_birth and first_name column values combined with a one-off randomly generated secret for this run.

For example, in every row where date_of_birth = '2000-01-01' and first_name = 'Carl', the date_of_birth will be replaced with a deterministically generated value (e.g. 1999-03-03). This same replacement value will be generated for repeated rows that match the same date_of_birth and first_name values within this run. However, this value will be different for each run. To retain consistency of deterministically generated values between multiple runs, see Consistent masking.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: date_of_birth
        hash_columns:
          - date_of_birth
          - name
        masks:
          - type: from_random_date
            min: '1980-01-01'
            max: '2000-01-01'

An equivalent alternative version of YAML syntax is shown below:

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: date_of_birth
        hash_columns:
          - column_name: date_of_birth
          - column_name: name
        masks:
          - type: from_random_date
            min: '1980-01-01'
            max: '2000-01-01'

Example with case_transform

If case_transform is specified under hash_columns,then the value of the first_name column will be transformed before being used for deterministic masking, ensuring that case will not affect the consistency of the mask. In the example below, where case_transform is specified on the hash column first_name, values such as "carl" and "Carl" will both be converted into "CARL" before hashing, ensuring that the final masked value is consistent regardless of case.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key:
      - id
      - first_name
    rules:
      - column: date_of_birth
        hash_columns:
          - column_name: first_name
            case_transform: upper
        masks:
          - type: from_random_datetime
            min: '1980-01-01'
            max: '2000-01-01'

Note: This example shows column-level hash_columns with advanced options. The same case_transform options are available when using task-level hash_columns.

Example with trim

If trim is specified as true, then any whitespace in the first_name column will be removed before the value is used for deterministic masking. In this case, values such as " John", " John " and "John " will all be converted into "John" before hashing, ensuring that the final masked value is consistent regardless of surrounding whitespace.

version: '1.0'
tasks:
  - type: mask_table
    table: users
    key:
      - user_id
      - first_name
    rules:
      - column: date_of_birth
        hash_columns:
          - column_name: first_name
            trim: true
        masks:
          - type: from_random_datetime
            min: '1980-01-01'
            max: '2000-01-01'

Example with source: self

If source: self is specified under hash_columns, the value being masked is used as its own hash source. This ensures identical input values produce identical masked outputs.

version: '1.0'
tasks:
  - type: mask_table
    table: customers
    key: id
    rules:
      - column: city
        hash_columns:
          - source: self
        masks:
          - type: from_file
            seed_file: DataMasque_cities.csv
            seed_column: city

Example with xpath

This example will mask the date_of_birth column with a date value that has been deterministically generated based on the hash of the date_of_birth attribute from the following XML document column.

<Root>
    <Info>
        <Employee date_of_birth="2022-10-08">Steve</Employee>
        <Employee date_of_birth="2022-10-08">Fred</Employee>
    </Info>
</Root>

To get the date_of_birth attribute from the XML document we need to specify the path with an xpath. The following xpath will get the date attribute from the first Employee element of the XML document, Info/Employee[1]/@date_of_birth. If you wanted to perform the hash on a list of date_of_birth attribute's values you can specify Info/Employee/@date_of_birth.

version: "1.0"
tasks:
  - type: mask_table
    table: xml_test
    key: id
    rules:
      - column: date_of_birth
        hash_columns:
          -  column_name: xml_data
             xpath: "Info/Employee/@date_of_birth"
        masks:
          - type: from_random_text
            min: 5
            max: 10

As shown in the results below, when the date_of_birth attribute has the same values for both Employee elements the resulting masked date_of_birth column contain the same values.

<Root>
    <Info>
      <Employee date_of_birth="2000-10-08">Steve</Employee>
      <Employee date_of_birth="2022-10-08">Fred</Employee>
    </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2001-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2002-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2003-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2004-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2000-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2001-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2000-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2000-10-08">Steve</Employee>
    <Employee date_of_birth="2012-10-08">Fred</Employee>
  </Info>
</Root>

<Root>
  <Info>
    <Employee date_of_birth="2000-10-08">Steve</Employee>
    <Employee date_of_birth="2022-10-08">Fred</Employee>
  </Info>
</Root>

Seeding masking from a table reference (table_reference)

A table reference is a named identity map that lets the same entity mask identically across a database, a tabular file, and a JSON document, even when each system keys it differently — it translates each system's own key into a shared seed value.

Worked example: one entity, two systems, keyed differently

A customer identity map ties each customer's identifiers together: the CRM database's customer_id, the billing system's billing_ref, and the account_ref that identifies the customer's account to both. It is stored as a table reference named customer_identity with these columns:

In the database, the customers table keys each customer by customer_id. We seed the name mask from the map's account_ref, matching on customer_id:

version: "1.0"
tasks:
  - type: mask_table
    table: customers
    key: customer_id
    rules:
      - column: first_name
        hash_columns:
          - table_reference: customer_identity
            source_key: [customer_id]
            target_key: [customer_id]
            value: [account_ref]
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed

A monthly billing extract identifies the same customers by their billing reference, in its own billing_account column, instead of by customer_id. We seed from the same map, matching the file's billing_account against the map's billing_ref:

version: "1.0"
tasks:
  - type: mask_tabular_file
    rules:
      - column: given_name
        hash_columns:
          - table_reference: customer_identity
            source_key: [billing_account]
            target_key: [billing_ref]
            value: [account_ref]
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed

Both tasks resolve customer 1 to the same account_ref (ACC-1001), which DataMasque hashes to choose the replacement name. The account reference is never written to the masked column; it only decides which name the customer masks to, so both systems land on the same one.

The customers table in the database:

The billing extract:

The two systems key the customer differently and spell the name differently, so hashing on either the local key or the name itself would give them different seeds. Seeding from the map is what makes them agree.

Keys

A table_reference hash source can be used anywhere hash_columns is accepted — at the task level of mask_table and mask_tabular_file, and at the rule (column) level. Instead of naming a column, the entry is a mapping with the following keys:

• table_reference (required): The name of the table reference to read the map from.
• source_key (required): One or more columns on the row being masked whose values identify the entity.
• target_key (required): One or more columns on the map to match source_key against. source_key and target_key are paired by order — the same convention as joins — and must list the same number of columns.
• value (required): One or more columns on the map whose value(s) seed the masking. When several columns are listed, they combine into a single seed, as a multi-column hash_columns list does.
• on_missing (optional): What to do when the row's source_key value is not found in the map. One of error (default) or self. See below.

At run time, for each row DataMasque looks up the row's source_key value(s) in the map by matching them against the map's target_key column(s), takes the matching row's value column(s) as the seed, and masks deterministically from that seed. Because every system that shares the map resolves to the same seed for the same entity, they all produce the same masked value.

on_missing behaviour

on_missing controls what happens when a row's source_key value has no matching row in the map:

• error (default): the run fails with an error naming the table reference. Use this when a missing key means the data is wrong and you would rather stop than mask inconsistently.
• self: DataMasque logs a warning and falls back to seeding on the row's own source_key value(s), as though the rule had hashed on those columns directly. The record is still masked deterministically, but because its seed came from its own value rather than the shared map, it will not match the same entity in another system. self keeps the run going at the cost of cross-system consistency for the unmatched rows.

Reusing a reference across rules

Several rules — across tasks, or across whole rulesets in the same run — can seed from one table reference. DataMasque builds the map once per distinct combination of reference name, target_key columns, and value columns, and rules that share that combination share a single built map. Column order is part of that identity: a target_key (or value) of [a, b] and one of [b, a] are treated as different maps and each built separately, so an inconsistent order across rules doubles the memory and disk the maps occupy for no benefit. List the columns in the same order everywhere you seed from a given reference.

Supported key and value types

Keys and values are matched and hashed by their canonical form, so a value resolves the same way across systems regardless of how each source typed it:

• Integers and integer-valued decimals are exact at any magnitude, and a whole-number float matches them — 1001, Decimal("1001.00"), and 1001.0 all resolve to 1001.
• Strings are compared literally and not reinterpreted as numbers, so a leading-zero id like "01001" never collapses into integer 1001. A plain integer string still matches its number, since "1001" and integer 1001 canonicalise to the same digits.
• Dates and datetimes are canonicalised, so one calendar day matches whether a source stored it as a DATE or as a midnight timestamp.
• Nulls match other nulls.

A CSV map is the one exception to literal string comparison. A CSV file carries no types, so a plainly numeric cell — "1001", or a spreadsheet's "1001.0" — is recovered to the integer it losslessly represents as the map is read, and so matches a typed 1001 from a database or parquet map. A cell whose text is meaningful — a leading zero ("01001"), a thousands separator, an exponent, or a real fraction — stays a string, so a text id is never collapsed into a different entity. This recovery applies to a CSV map only: the row being masked, and parquet or database maps, keep the types their source gave them.

Use an integer or a string as an identifier. Avoid a genuinely fractional floating-point identifier: its canonical form can differ between systems — a float 1e-05 versus the string "0.00001", say — so the same entity may fail to match. A whole-number identifier delivered as a float too large to represent exactly (beyond 2^53) fails the run rather than risk masking two different ids to one value.

Materialising a large map into the target database (build_temp_table)

A very large map (millions of rows) may not fit comfortably in memory. For relational database targets, you can materialise it into a temporary table and join to it in SQL instead — the masked result is identical. Give a build_temp_table task a table_reference, then join and hash on the joined column as usual:

version: "1.0"
tasks:
  - type: build_temp_table
    table_name: customer_identity_map
    table_reference: customer_identity
  - type: mask_table
    table: customers
    key: customer_id
    joins:
      - target_table: customer_identity_map
        target_key: customer_id
        source_table: customers
        source_key: customer_id
    rules:
      - column: full_name
        hash_columns: ["customer_identity_map.masked_name_seed"]
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed

This masks full_name from the same masked_name_seed the in-memory table_reference lookup would use, so a database materialised this way stays consistent with a file or document masked through the in-memory lookup.

build_temp_table accepts table_reference as an alternative to sql_select_statement or sql_select_file (the three are mutually exclusive):

• table_reference (this OR sql_select_statement OR sql_select_file required): The name of the table reference whose map populates the temporary table.

Notes:

• Materialising a map is supported on most relational database targets. It is not available for Amazon Redshift, nor for the non-relational targets (Amazon DynamoDB, MongoDB, Amazon DocumentDB, and Azure Cosmos DB). To seed masking from a map against a file target, use the in-memory table_reference hash source above.

• DataMasque infers column types in the temporary table from the map:

  • A Parquet map keeps its native column types.
  • A CSV map is all text, except integer columns, which a lossless check detects and types as integers so the common integer-ID join needs no casts. An identifier with leading zeros (e.g. 00123) stays text, so it never becomes a number. Dates and decimals remain text; use a Parquet map if you need them fully typed.
  • Canonical identifiers should be integers or strings. Avoid a fractional/floating identifier — it can hash differently between the in-memory and materialised paths and break consistency.