Unique Masks

Unique masks are masks that will ensure every value is replaced with a unique value for that column. For other masks see all mask functions.

• Imitate Unique (imitate_unique)
  Masks strings or numbers while preserving format and uniqueness
• Imitate NZ IRD (imitate_nz_ird)
  Masks New Zealand IRD numbers while preserving validity and format
• Imitate UUID (imitate_uuid)
  Masks UUID/GUID values while preserving format and uniqueness
• From Unique Imitate (from_unique_imitate) (deprecated)
  Transforms strings or numbers to be random, retaining format and uniqueness
• From Unique (from_unique) (deprecated)
  Generates random but unique strings or numbers, from a format string (databases only)

Imitate Unique (imitate_unique)

Note: When used in a mask_table task, this mask type will be skipped in a dry run as it requires writing to the database.

The imitate_unique mask replaces characters with others from the same configured character group, retaining format and uniqueness.

• Characters in the configured charset groups are replaced with other characters from the same group.
• Characters not in any configured charset group are left unchanged.
• The masked values are guaranteed to be consistent for the same given input, and different for different inputs.

By default, the character set groups are uppercase letters (A-Z), lowercase letters (a-z), and digits (0-9). Custom charset groups can be configured to support additional characters such as non-Latin scripts.

This mask can only be used on columns of integer or string (char / varchar) type.

null values will always be left as null. Empty strings are preserved as empty strings.

Important! imitate_unique does not support IDENTITY columns in Oracle or Microsoft SQL Server.

imitate_unique does not support directly masking partitioning tables (child tables of partitions) in Postgres 10+. Please mask the parent table instead.

imitate_unique does not support directly masking hybrid tables in Snowflake.

Note on Sequence Reset Warnings: When masking columns that have default values containing function calls (e.g. gen_random_uuid(), now() in PostgreSQL, or custom functions), you may see a warning: "Unable to reset the next value of the sequence for column… If a sequence exists, it may generate a value that already exists in the column."

This warning occurs because DataMasque cannot determine whether functions in default values internally use sequences. The warning is informational and harmless for columns where the default function does not use a sequence. DataMasque takes this defensive approach to ensure that if a sequence does exist within a function call, you are notified that it could not be reset. For columns with non-sequence function defaults, this warning can be safely ignored and masking will work correctly.

You can apply an imitate_unique mask to a primary key column or a column that is used as a foreign key in another table. References will be updated automatically. Composite primary keys are supported.

Parameters

• charsets (optional): A list of charset groups to use for masking. Each group is masked independently; characters are only replaced with other characters from the same group. Each entry has a type of either named or custom:

  • type: named with charset: one of DIGITS (digits 0-9), ALPHA (uppercase and lowercase letters A-Z and a-z), ALPHA_UPPERCASE (uppercase letters A-Z), or ALPHA_LOWERCASE (lowercase letters a-z).
  • type: custom with characters: a string of the characters to include.

  Characters must not appear in more than one configured group. If charsets is omitted, the default groups are DIGITS, ALPHA_UPPERCASE, and ALPHA_LOWERCASE.

• checksum (optional): A checksum algorithm to apply to the masked output. The available options are the same as for from_unique_imitate: brazilian_cpf, credit_card, luhn, icp, australian_business_number, australian_company_number, or weighted_dual_checksum. For alphanumeric checksums (such as icp), the configured charsets must include both letters and digits.

• on_invalid (optional): How to handle values that fail checksum validation. Can only be used in combination with the checksum parameter. One of:

  • error (default): Raise an error and stop masking.
  • mask: Mask the value with an invalid checksum if the format is correct, or mask without consideration of a checksum otherwise.
  • skip: Leave the value unchanged. See Using on_invalid for detailed behaviour.

• retain_prefix_length (optional): The number of maskable characters (characters in the configured charset groups) to retain at the start of the value. Must leave at least one maskable character available. By default, no prefix is retained. See Retaining Prefixes for details and examples.

• retain_suffix_length (optional): The number of maskable characters (characters in the configured charset groups) to retain at the end of the value. By default, no suffix is retained.

• include_leading_zeros (optional): Whether to mask leading zeros in input values and allow leading zeros in masked values. One of:

  • true: Leading zeros will be masked.
  • false: Leading zeros will be preserved.
  • warn (default): Like false, but also issues a warning in the run log if a string value with all zero digit(s) is encountered. See the from_unique_imitate documentation for detailed behaviour and examples.

• min_digits (optional): Only applicable when include_leading_zeros is true. Pads integer-type values to the given number of digits before masking. Valid values are between 1 and 18.

• on_too_long (optional): Only applicable when include_leading_zeros is true and min_digits is specified. Determines what action to take when an integer value has more than min_digits digits. One of:

  • error (default): Raise an error and stop masking.
  • mask_suffix: The last min_digits digits will be masked; earlier digits will be left as-is.
  • mask_all: The entire value will be masked.

• seed (optional): An additional seed value to adjust the RNG. Use this to produce different masked outputs for the same input across different columns or tables.

Example

version: '1.0'
tasks:
  - type: mask_table
    table: customers
    key: id
    rules:
      - column: customer_code
        masks:
          - type: imitate_unique
            charsets:
              - type: named
                charset: ALPHA_UPPERCASE
              - type: named
                charset: DIGITS
              - type: custom
                characters: "等级的活动"

Limitations

The maskable segment length is limited to 65,535 characters (characters in the configured charset groups after any prefix/suffix preservation).

Imitate NZ IRD (imitate_nz_ird)

Note: When used in a mask_table task, this mask type will be skipped in a dry run as it requires writing to the database.

The imitate_nz_ird mask is designed for masking New Zealand Inland Revenue Department (IRD) numbers. Valid IRD numbers are masked to other valid IRD numbers with correct checksums.

• The mask is format-preserving: valid IRD numbers are masked to valid IRD numbers.
• The masked values are guaranteed to be consistent for the same given input, and different for different inputs.
• IRD numbers can be either 8 or 9 digits. An 8-digit IRD number is equivalent to a 9-digit number with a leading zero (e.g., 12345678 is equivalent to 012345678). The mask handles both formats.

This mask can only be used on columns of integer or string (char / varchar) type.

null values will always be left as null. Empty strings are preserved as empty strings.

Note: The IRD numbers shown in this documentation are randomly generated for illustrative purposes. Any resemblance to actual IRD numbers is coincidental.

Important! imitate_nz_ird does not support IDENTITY columns in Oracle or Microsoft SQL Server.

imitate_nz_ird does not support directly masking partitioning tables (child tables of partitions) in Postgres 10+. Please mask the parent table instead.

imitate_nz_ird does not support directly masking hybrid tables in Snowflake.

You can apply an imitate_nz_ird mask to a primary key column or a column that is used as a foreign key in another table. References will be updated automatically. Composite primary keys are supported.

When imitate_nz_ird is applied to a primary key or unique key column, DataMasque temporarily disables the constraint, masks the values, then re-enables the constraint after masking completes.

Parameters

• on_invalid (optional): A string to specify the action to take if the value is not a valid IRD number. One of:
  • error (default): Raise an error and stop masking.
  • mask: For 8-9 digit values with an invalid checksum, mask to an output that also has an invalid checksum. For values with fewer or more digits, mask using from_unique_imitate (length-preserving).
  • skip: Skip to the next value, the value remains unchanged.
• retain_length (optional): A boolean to preserve the digit count of the input. When true, 8-digit inputs produce 8-digit outputs and 9-digit inputs produce 9-digit outputs. Defaults to false (generally preferable), meaning the output may vary between 8 and 9 digits. See details and examples below.
• skip_letters (optional): A boolean to enable masking of alphabetic characters separately. When true, letters are masked via from_unique_imitate while digits follow IRD masking logic. Defaults to false (letters are preserved as-is).
• seed (optional): A string or integer combined with the instance secret and run secret to control randomness. Allows different columns using the same mask type to produce different outputs for identical inputs. Note that separate runs with the same seed will produce different output unless the instance secret and run secret are the same.

A value is considered invalid if:

• It has fewer than 8 or more than 9 digits
• It is outside the valid IRD range (10,000,000 to 149,999,999)
• It has 8-9 digits but the checksum is incorrect

retain_length Details

By default, imitate_nz_ird treats 8-digit and 9-digit IRD numbers as equivalent (since 12345678 and 012345678 represent the same IRD). This means the same IRD stored as 12345678 (integer) and "012345678" (string) will mask to the same numeric value. String outputs preserve their original length when possible—an 8-digit string stays 8 digits if the output value is under 100,000,000, and a 9-digit string always stays 9 digits.

Setting retain_length: true guarantees the exact digit count is always preserved, regardless of the output value. This is useful when your data contains both "12345678" and "012345678" as distinct string values in the same column that should remain distinguishable after masking.

Example Rulesets

Basic IRD Masking (Integer Column)

version: '1.0'
tasks:
  - type: mask_table
    table: taxpayers
    key: id
    rules:
      - column: ird_number
        masks:
          - type: imitate_nz_ird

Masking with Format Preservation (String Column)

version: '1.0'
tasks:
  - type: mask_table
    table: records
    key: id
    rules:
      - column: formatted_ird
        masks:
          - type: imitate_nz_ird

Using retain_length for Consistent Digit Count (String Column)

version: '1.0'
tasks:
  - type: mask_table
    table: mixed_format_irds
    key: id
    rules:
      - column: ird_number
        masks:
          - type: imitate_nz_ird
            retain_length: true

Handling Invalid Values

version: '1.0'
tasks:
  - type: mask_table
    table: legacy_data
    key: id
    rules:
      - column: ird_number
        masks:
          - type: imitate_nz_ird
            on_invalid: mask

Imitate UUID (imitate_uuid)

Note: When used in a mask_table task, this mask type will be skipped in a dry run as it requires writing to the database.

The imitate_uuid mask is designed specifically for masking UUID (Universally Unique Identifier) and GUID (Globally Unique Identifier) values while preserving their format and maintaining uniqueness.

This mask automatically detects and handles various UUID formats:

• Standard hyphenated format: 550e8400-e29b-41d4-a716-446655440000
• Compact format (no hyphens): 550e8400e29b41d4a716446655440000
• Prefixed formats: urn:uuid:550e8400-e29b-41d4-a716-446655440000, uuid:550e8400-e29b-41d4-a716-446655440000
• Binary/byte format: 16-byte binary representation (e.g. Oracle RAW(16) columns)

Key features:

• Deterministic masking: The same input UUID always produces the same masked output, ensuring consistency across related data.
• Format preservation: The original UUID formatting (hyphens, prefixes, case) is maintained in the output.
• Referential integrity: When applied to primary keys, foreign key references are automatically updated.
• Uniqueness guarantee: Each distinct input UUID maps to a unique output UUID.

null values will always be left as null. Empty strings are preserved as empty strings.

Binary/Byte Format Support

imitate_uuid can mask UUIDs stored as binary data (e.g. Oracle RAW(16) columns). When processing byte values:

• The input must be exactly 16 bytes to be recognised as a valid UUID
• The same UUID represented as bytes or string will produce equivalent masked results
• All retention parameters work identically for byte and string formats
• The output maintains the byte format when the input is bytes

Important: The on_invalid: mask option has limitations with byte values. Since from_unique_imitate could produce non-hexadecimal characters, it cannot safely be used for byte masking. When byte input is invalid:

• on_invalid: error raises an error (default behaviour)
• on_invalid: skip returns the original bytes unchanged
• on_invalid: mask reverts to error behaviour for safety

Important! imitate_uuid does not support IDENTITY columns in Oracle or Microsoft SQL Server. imitate_uuid does not support directly masking partitioning tables (child tables of partitions) in Postgres 10+. Please mask the parent table instead. imitate_uuid does not support directly masking hybrid tables in Snowflake.

Note on Sequence Reset Warnings: When masking UUID columns that have default values containing function calls (e.g. gen_random_uuid() in PostgreSQL), you may see a warning: "Unable to reset the next value of the sequence for column… If a sequence exists, it may generate a value that already exists in the column."

This warning occurs because DataMasque cannot determine whether functions in default values internally use sequences. The warning is informational and harmless for UUID columns, as UUID generation functions typically do not use sequences. DataMasque takes this defensive approach to ensure that if a sequence does exist within a function call, you are notified that it could not be reset. For UUID columns with function defaults, this warning can be safely ignored and masking will work correctly.

You can apply an imitate_uuid mask to a primary key column or a column that is used as a foreign key in another table. References will be updated automatically. Composite primary keys containing UUID columns are supported.

Parameters

• seed (optional): An additional seed value to create different masking patterns while maintaining determinism. Useful for generating different UUID mappings for different environments or columns.
• retain_uuid_version (optional): A boolean to preserve the UUID version indicator (13th hexadecimal character). When true (default), the version nibble is retained to maintain UUID version consistency.
• retain_prefix_length (optional): Number of hexadecimal characters to retain from the beginning of the UUID. Valid values are 0-31. Must not overlap with suffix retention.
• retain_suffix_length (optional): Number of hexadecimal characters to retain from the end of the UUID. Valid values are 0-31. Must not overlap with prefix retention.
• on_invalid (optional): Action to take when the input value is not a valid UUID. One of:
  • error (default): Raise an error and stop masking.
  • skip: Leave the value unchanged.
  • mask: Fall back to character-by-character masking using from_unique_imitate. Note: For byte values, this option reverts to error behaviour since from_unique_imitate cannot safely mask binary data.
• default_case (optional): The case to use for output when the input UUID contains only digits or has mixed case. Options: upper, lower (default).
• warn_if_masking_invalid (optional): A boolean that controls whether to log warnings when invalid values are masked (only applies when on_invalid is mask). Defaults to true.
• warn_on_mixed_case (optional): A boolean that controls whether to log warnings when UUIDs contain mixed uppercase and lowercase characters. Defaults to true.
• uuid_format (optional): Controls the format compliance of masked UUIDs. Options: default, rfc_4122. When set to rfc_4122, masked UUIDs are guaranteed to have a valid RFC 4122 variant (variant bits set to 10xx) and a valid version (1-5). This is useful when downstream systems validate UUID format strictly (e.g. yup). Defaults to default.

Examples

Basic Operation

This example shows standard UUID masking with default settings:

version: '1.0'
tasks:
  - type: mask_table
    table: users
    key: id
    rules:
      - column: user_id
        masks:
          - type: imitate_uuid

The format (with or without hyphens) is preserved.

Using Seed for Different Mappings

The seed parameter creates different masking patterns. Compare the same UUID masked with and without a seed:

version: '1.0'
tasks:
  - type: mask_table
    table: audit_log
    key: id
    rules:
      - column: uuid_with_seed
        masks:
          - type: imitate_uuid
            seed: "custom_seed"
      - column: uuid_without_seed
        masks:
          - type: imitate_uuid

The seed ensures different columns get different masked values for the same input UUID. Note that DataMasque's run secret and instance secret are combined with the seed to control the random output, i.e. separate runs with the same seed will produce different output unless the instance secret and run secret are the same.

Note: The seed parameter is used for controlling masking patterns:

• Use different seeds for different columns to ensure the same UUID doesn't map to the same output across columns
• Example: Without different seeds, if user UUID 123e4567... appears in both created_by and modified_by columns, it would mask to the same value in both places, potentially revealing relationships

Prefix and Suffix Retention

Retain specific hex characters from the start or end (hyphens not counted):

version: '1.0'
tasks:
  - type: mask_table
    table: devices
    key: id
    rules:
      - column: device_uuid
        masks:
          - type: imitate_uuid
            retain_prefix_length: 10    # First 10 hex chars (dashes not counted)
            retain_suffix_length: 4     # Last 4 hex chars

The first 10 hex characters (550e8400e2) and last 4 (0000) are preserved (note that dashes are not counted).

Note: When using retain_prefix_length and retain_suffix_length:

• Only hexadecimal characters are counted (0-9, a-f, A-F)
• Hyphens and other formatting characters are ignored in the count
• Total retention (prefix + suffix) must be less than 32 characters
• Example: In 550e8400-e29b-41d4-a716-446655440000, the first 12 hex characters are 550e8400e29b, not 550e8400-e29

Handling Invalid Values

The on_invalid parameter controls behaviour for non-UUID values:

Skip invalid values (leave unchanged):

- type: imitate_uuid
  on_invalid: skip

Mask invalid values (use from_unique_imitate):

- type: imitate_uuid
  on_invalid: mask

Error on invalid values (stop masking):

- type: imitate_uuid
  on_invalid: error  # Default

Note: When on_invalid: mask is set:

• Non-UUID values are processed using from_unique_imitate
• This maintains uniqueness but doesn't produce valid UUIDs
• Original format and character types are preserved where possible
• Example: DEVICE-001 might become ZXCVBN-123

Case Handling

The default_case parameter applies when input has mixed case or only digits. When the input has consistent casing (all uppercase or all lowercase), that case is retained in the output:

- type: imitate_uuid
  default_case: upper  # or 'lower' (default)

Warning Messages

Warning messages are logged to the run log in these situations:

• warn_on_mixed_case: Logs once per column per masking run when mixed-case UUIDs are detected (default: true)
• warn_if_masking_invalid: Logs once per column per masking run when invalid values are masked with on_invalid: mask (default: true)

These warnings appear only once per affected column during each masking run, regardless of how many mixed-case or invalid values are encountered in that column. If multiple columns use imitate_uuid, each column can generate its own warning. The warnings help identify data quality issues without interrupting the masking process or cluttering the log with repeated messages.

Warning: imitate_uuid is designed for masking genuine, randomly-generated UUIDs. When applied to sequential or predictable UUID-formatted data (such as incremental counters in UUID format) combined with high retention settings that leave only a few characters to be masked, collisions may occur due to limited output space. For such data, consider using from_unique_imitate instead.

From unique (from_unique)

A mask that generates string values that are guaranteed to be unique within the target column.

Note: To automatically cascade primary and unique key values to foreign keys or mask composite keys, consider using mask_unique_key instead.

Parameters

• format (required): The format that will be used to generate values. See format string syntax for details on how to construct a format string.

version: '1.0'
tasks:
  - type: mask_table
    table: drivers
    key: id
    rules:
      - column: licence_plate
        masks:
          - type: from_unique
            format: "{[A-Z],3}{[0-9],3}"

Note

• All values produced by from_unique will be strings. To convert values to other data types, you can chain your from_unique mask with a typecast mask.
• Unlike from_format_string, from_unique ensures generated values are unique. This is achieved using the same underlying unique value generation procedure as mask_unique_key.
• The specified format must allow for a sufficient number of unique values to cover the full number of rows in the target table or file (rounded up to the nearest multiple of the batch_size for mask_table tasks), otherwise an error will be reported when executing the run.
• Using the from_unique mask type differs from the mask_unique_key task type in the following ways:
  • from_unique cannot perform any cascading of values to related columns, such as foreign keys.
  • from_unique cannot be used to guarantee joint uniqueness across the columns in a composite key, unless guaranteeing uniqueness within a single column within the composite key is sufficient for the target use case.
  • from_unique can be used to update unique columns at the same time as other columns in a mask_table task, without the need for a separate database update performed by a mask_unique_key task.
  • from_unique can be used to generate unique values in mask_tabular_file tasks or in mask_table tasks for databases that do not support mask_unique_key (such as Amazon DynamoDB).
• Using from_unique in a mask_table task where worker_count > 1 is only supported for Oracle and Amazon DynamoDB connections.
• from_unique does not support directly masking partitioning tables (child tables of partitions) in Postgres 10+. Please mask the parent table instead.
• from_unique does not support directly masking hybrid tables in Snowflake.
• from_unique cannot be used in the rules of a mask_file task.
• Like other mask functions, from_unique cannot be used to update the key columns of a mask_table task (except for databases that allow mask_table to update key columns, such as Amazon DynamoDB).
• All from_unique masks within a given task will be produced from the same sequence of unique values, such that two from_unique masks with identical format will produce the same value for the same row. This can be useful to ensure values produced by identical from_unique masks in different if branches are jointly unique.

From Unique Imitate (from_unique_imitate)

Deprecated: Consider using imitate_unique instead, which provides deterministic, collision-free masking with configurable character sets.

Note: When used in a mask_table task, this mask type will be skipped in a dry run as it requires writing to the database.

The from_unique_imitate mask is very similar to the imitate mask:

• alphabetical characters a-z and A-Z are replaced by other alphabetical characters of the same case
• digits are replaced by other digits
• all other characters (whitespace, symbols such as %, and so on) are left as-is.

However, there are the following differences:

• Most notably, from_unique_imitate can also be used on primary keys and columns that have a UNIQUE constraint. Indeed, it is designed specifically for data that must take unique values per row.
• The masked values are guaranteed to be consistent for the same given input, and different for different inputs. (The specific values generated will vary across runs unless you control the seeding.)
  • For example, if a column has a UNIQUE constraint and the existing data satisfies that constraint, the resulting masked values will all be unique.
  • If a column contains the same value in every row, the content of the column after masking will also have the same value in every row.

The from_unique_imitate mask will always create a different result to its input, as if it has an implicit force_change: true. The mask may occasionally preserve individual letters and digits in the same position, but it is always guaranteed that at least one (and very likely almost all) of the alphanumeric characters will be different. For example, ABC-123 may be masked to BYC-457.

This mask can only be used on columns of integer or string (char / varchar) type.

null values will always be left as null.

Important! from_unique_imitate does not support IDENTITY columns in Oracle or Microsoft SQL Server.

from_unique_imitate does not support directly masking partitioning tables (child tables of partitions) in Postgres 10+. Please mask the parent table instead.

from_unique_imitate does not support directly masking hybrid tables in Snowflake.

from_unique_imitate will try to enable row movement for partitioned parent tables in Oracle Enterprise Edition if it has not already been enabled.

Note on Sequence Reset Warnings: When masking columns that have default values containing function calls (e.g. gen_random_uuid(), now() in PostgreSQL, or custom functions), you may see a warning: "Unable to reset the next value of the sequence for column… If a sequence exists, it may generate a value that already exists in the column."

This warning occurs because DataMasque cannot determine whether functions in default values internally use sequences. The warning is informational and harmless for columns where the default function does not use a sequence. DataMasque takes this defensive approach to ensure that if a sequence does exist within a function call, you are notified that it could not be reset. For columns with non-sequence function defaults, this warning can be safely ignored and masking will work correctly.

You can apply a from_unique_imitate mask to a primary key column or a column that is used as a foreign key in another table. References will be updated automatically. Composite primary keys are supported.

Parameters

• skip_letters (optional): A boolean to enable or disable the skipping alphabetical characters from being masked. Defaults to false (alphabetical characters will be replaced).
• skip_digits (optional): A boolean to enable or disable the skipping of digits from being masked. Defaults to false (digits will be replaced).
• checksum (optional): A string to specify an algorithm to use to generate unique valid replacements. Options: brazilian_cpf, credit_card, luhn, icp, australian_business_number, australian_company_number, or weighted_dual_checksum.
• on_invalid (optional): A string to specify the action to take if the value fails checksum validation. One of:
  • error (default): Raise an error and stop masking.
  • mask: If the input length and format is valid (ie, the input value is only invalid because of an invalid checksum), the input value will be masked with a new uniquely assigned invalid checksum. Otherwise, the input value will be masked without consideration of a checksum.
  • skip: Skip to the next value, the value remains unchanged.
• retain_prefix_length (optional): The number of characters of the input value to retain. See Retaining Prefixes below. By default, no prefix is retained (i.e. the entire input value is masked). Note that retaining prefixes does not work with a prefix based checksum (australian_business_number)
• include_leading_zeros (optional): Whether to mask leading zeros in input values and allow leading zeros in masked values. See the table below for details and examples of how this option works. One of:
  • true: Leading zeros will be masked.
  • false: Leading zeros will be preserved.
  • warn (default): Like false, but also issues a warning in the run log if a string value with all zero digit(s) is encountered, reminding the user that the digits (or perhaps the entire value) will not be masked.
• min_digits (optional): Only applicable when include_leading_zeros is true. Pads integer-type values to the given number of digits before masking. Valid values for min_digits are between 1 and 18.
• on_too_long (optional): Only applicable when include_leading_zeros is true and min_digits is specified. Determines what action to take when an integer value with more than min_digits digits is to be masked. (The leading - sign in a negative number does not count as a digit.) Valid values are:
  • error (default): Raise an error and stop masking.
  • mask_suffix: The last min_digits digits of the value will be masked; earlier digits will be left as-is.
  • mask_all: The entire value will be masked.
• disable_warning_on_skipped_characters (optional): A boolean which, when set to true, will prevent warnings about characters being skipped being logged during masking. Defaults to false.

The include_leading_zeros option has no effect for integer data unless min_digits is also specified. By default, integer values are always masked to values with the same number of digits as the input.

Because it can create duplicate values, masking negative integers with min_digits is not supported. If this option is set and a negative integer value is encountered, DataMasque raises an error and stops masking.

version: "1.0"
tasks:
  - type: mask_table
    table: '"my_table"'
    key: '"id"'
    rules:
      - column: '"value_as_string"'
        masks:
          - type: from_unique_imitate
            include_leading_zeros: true
      - column: '"value_as_integer"'
        masks:
          - type: from_unique_imitate
            include_leading_zeros: true
            min_digits: 6

Invalid Parameter Combinations

• Setting both skip_digits and skip_letters to true is prohibited as no masking would take place.
• Using the checksum option requires masking digits, so if any checksum is specified then you cannot set skip_digits to true. Further, the icp checksum is an alphanumeric checksum, so you cannot set skip_digits nor skip_letters to true when using this checksum.
• Because it would have no effect, include_leading_zeros cannot be specified if skip_digits is set to true.
• The min_digits option is not compatible with checksum, nor with retain_prefix_length.
• Prefix based checksums cannot be used with retain_prefix_length. This applies to australian_business_number.

Values Requiring Checksums

When the output values must satisfy a checksum, specify the name of the checksum as the checksum parameter. Unique values will be generated that satisfy that checksum algorithm. The available options for checksum are:

• brazilian_cpf
• credit_card
• luhn
• icp
• australian_business_number
• australian_company_number
• weighted_dual_checksum

For each checksum, the input value must contain a certain number of digits 0-9 and no letters A-Z or a-z. Other non-letter characters that are used for formatting are retained in the output. The replacement value will conform to the checksum algorithm, even if the input did not, provided it is of the correct length.

To handle masking of values that may not match the checksum, or that may contain letters, specify the on_invalid parameter. Please refer to Using on_invalid for a detailed explanation of the behavior of each on_invalid option.

brazilian_cpf

Use this checksum type to generate values that satisfy the Brazilian CPF (Cadastro de Pessoas Físicas) number checksum.

For valid CPFs to be generated, the input value must contain 11 digits (and may contain spaces or punctuation). For handling of invalid input values (for example, incorrect length, bad checksum or the presence of letters), please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

credit_card and luhn

The credit_card and luhn checksums both generate values that satisfy the Luhn checksum algorithm. The difference is how they each validate the length of the number:

• luhn may be applied to any number containing two or more digits.
• credit_card is only valid for numbers of length 12 to 19, inclusive.

For values of length 12-19 characters, the behaviour of both checksums is identical.

If you are masking only credit cards, then credit_card should be preferred, as it will also validate the length of existing values. luhn should be used when generating values of other lengths, for example, mobile phone IMEI numbers.

For handling of invalid input values (for example, incorrect length, bad checksum or the presence or letters), please refer to Using on_invalid.

icp

Use this checksum type to generate values that satisfy the New Zealand Installation Control Point (ICP) checksum.

For valid ICPs to be generated, the input value must contain 15 digits and letters (and may contain spaces or punctuation). For handling of invalid input values (for example, incorrect length, or bad checksum), please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

australian_business_number

Use this checksum type to generate values that satisfy the Australian Business Number (ABN) checksum.

For valid ABNs to be generated, the input value must contain 11 digits (and may contain spaces, hyphens, or punctuation), and retain_prefix_length may not be specified. For handling of invalid input values (for example, incorrect length, bad checksum, or when containing letters) please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

australian_company_number

Use this checksum type to generate values that satisfy the Australian Company Number (ACN) checksum.

For valid ACNs to be generated, the input value must contain 9 digits (and may contain spaces or punctuation). For handling of invalid input values (for example incorrect length, bad checksum or letters), please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

Example

This example will apply from_unique_imitate masks to the vehicle_id, license_plate and validation_code columns.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: vehicle_id
        masks:
          - type: from_unique_imitate
      - column: license_plate
        masks:
          - type: from_unique_imitate
      - column: validation_code
        masks:
          - type: from_unique_imitate

weighted_dual_checksum

Use this checksum type to generate values that satisfy the dual weighted checksum, where each digit in the value is given its own weight, but may generate a high or low checksum digit, to match the high/low value on the original checksum.

Using on_invalid

The on_invalid parameter can be used to control how invalid values are handled by from_unique_imitate. It can only be used in combination with the checksum parameter, since without specifying a checksum there is no way of considering a value "invalid".

A value is considered invalid if:

• It is too short for the checksum, or,
• The checkdigit(s) are not valid for the checksum, or,
• It contains letters – this can be controlled with the skip_letters parameter. See Determining and Handling Invalid Values for more detail.

Note that null is a special case and is not considered invalid. null input is masked to null output, regardless of settings.

The following table illustrates the behaviour of from_unique_imitate based on different on_invalid parameters.

Determining and Handling Invalid Values

There two are main ways that a value can be invalid:

• It contains letters.
• It has the wrong length or checksum.

Note that special characters, punctuation, and spaces, do not affect the validity of values. For example, from_unique_imitate considers the values 123 456, 123-456 and 123456 the same.

Determining the validity based on the presence of letters can be controlled with the skip_letters parameter. When set to true, a value will not be invalid if it contains letters. However, if the digits themselves do not satisfy the given checksum, then the value would be considered invalid.

The following table shows the validity of some example values for the brazilian_cpf algorithm, with and without the use of skip_letters: true. The rules are applicable to any checksum.

Once a value is determined to be invalid:

• If using on_invalid: error then the masking task will stop with an error.
• If using on_invalid: skip then the value will be returned unmasked.
• If using on_invalid: mask then masking will continue, and is described in more detail below.

For invalid values, the behaviour of the mask will change based on the reason for it being invalid.

If masking a value that is invalid only because of a bad check checksum, the output will mask uniquely to a new value with a uniquely masked, invalid checksum. If the value is invalid for another other reason, the string will be masked uniquely.

For example, the brazilian_cpf algorithm requires 11-digit values. An 11-digit value that is not a valid CPF number will be masked to an invalid CPF number (an 11-digit number that does not have a valid CPF number checksum).

However, a number that is not 11-digits will not be masked to a valid CPF number, as the number of digits is not correct for that algorithm, but the numbers will be masked.

The following table shows the minimum and maximum value length to which the checksum applies.

If the value is invalid because it contains letters:

• The letters will be masked if skip_digits is false.
• The letters remain unchanged if skip_digits is true.

The following table gives examples of masked outputs for invalid inputs, for different skip_letters options. on_invalid is set to mask, otherwise no masking would occur. The example uses the brazilian_cpf algorithm but the rules are applicable to any checksum.

Warning: When using on_invalid: mask with the brazilian_cpf checksum, 10 of the possible 1,000,000,000 digit combinations are not maskable with this algorithm.

Where the data to mask has 11 digits, and the first nine digits are all the same digit (eg, 111.111.111-xx regardless of the checksum), the number itself is considered to be invalid.

Where this data is required to be masked, it is recommended that the from_unique_imitate rule is wrapped by an if rule that detects this data, and directs the engine to apply an alternative algorithm.

When the masking process is configured to uniquely mask a CPF number that is invalid due to repeated digits, the run log will note a warning, and these entries will not be masked. This does not affect other checksum masking as this situation is specific to Brazilian CPF unique masking.

Example

This example will apply from_unique_imitate masks to the cpf_number and apply a redaction to known invalid numbers by leveraging an if conditional masking rule.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules: 
      - if:
        - column: cpfnumber
          matches: '(?:.*?(\d)\1\1){3}.*'
        rules:
          - column: cpfnumber
            masks:
              - type: from_fixed
                value: 'redacted'
        else_rules:
          - column: cpfnumber
            masks:
              - type: from_unique_imitate
                checksum: brazilian_cpf
                on_invalid: mask

Retaining Prefixes

When masking values with from_unique_imitate, the retain_prefix_length option be used to specify the number of prefix characters of the input to retain in the output.

The length takes into account only characters that would be masked based on the parameters of the mask. Since from_unique_imitate doesn't mask punctuation or spaces, then these are not counted towards the prefix length. Similarly, if using skip_letters/skip_digits, then letters or digits (respectively) won't be counted in the prefix.

If the retain_prefix_length is equal to or longer than the values to be masked, then an error will be raised during masking.

The following table shows the retained prefix for example values, based on different parameters.

The retain_prefix_length parameter can be combined with the checksum parameter with a suffix-based checksum (may not be specified when selecting australian_business_number). The checksum will be generated after combining the prefix with the masked values.

For example, using the luhn checksum, with retain_prefix_length of 5. The value to be masked is 211287932175 (which is valid for the Luhn checksum).

The order of masking is:

• Extract 5 characters as a prefix: 21128
• Apply unique masking to all digits but the checksum digit 793217, giving (for example) 123456
• This value is combined with the prefix, giving 21128123456
• The checksum digit is calculated from this prefix and masked value, which is 0
• The checksum digit is appended to the masked value, giving final output value of 211281234560, which is also valid for the Luhn algorithm.

When retaining the prefix and using a checksum, the number of digits must be less than the number of digits in the original value, minus the length of the check digits. For example, to mask a Brazilian CPF, a maximum of 8 digits can be retained, since the value is made up of 11 digits with the final 2 being checkdigits.

Checksum validity is checked when retaining the prefix, and validation is on the original value (i.e. including the prefix). To control how to handle invalid values, refer to Determining and Handling Invalid Values.

The following example ruleset shows masking a column using the credit_card checksum, retaining the first 4 digits.

version: "1.0"
tasks:
  - type: mask_table
    table: customers
    key: customer_id
    rules:
      - column: credit_card_number
        masks:
          - type: from_unique_imitate
            retain_prefix_length: 4
            skip_letters: true
            checksum: credit_card

RedShift Limitations

Due to the current method in which we mask with RedShift, please note the current limitations:

• PRIMARY KEY constrained columns will be transformed into columns which are UNIQUE constrained instead of PRIMARY KEY constrained.
• NOT NULL constrained columns will lose their NOT NULL constraint. However, this is not just limited to from_unique_imitate masking.