String masks

String masks are masks with useful behaviour for string types.
For simpler masks or masks that work on other types See all mask functions.

• Imitate (imitate)
  Replaces characters with other characters from the same set (letters for letters, numbers for numbers)
• Random Text (from_random_text)
  Generates random strings of letters
• Transform Case (transform_case)
  Transforms the case of characters in a string
• Take Substring (take_substring)
  Extracts a substring from a column's value
• Replace Substring (replace_substring)
  Applies masks to a specific portion of string values
• Replace Regular Expression (replace_regex)
  Applies masks to parts of string values that match a given regular expression

Imitate (imitate)

Replace each character in a string with another random character from its same set. The character sets are:

• Uppercase letters (A-Z).
• Lowercase letters (a-z).
• Digits (0-9).

Characters not in these sets (such as punctuation and symbols) are not replaced.

This mask is designed to be easy to drop in place to mask values that must have a specific format, but whose value is not important. For example, it could be used to mask:

• Phone numbers (e.g. +1 (555) 867-5309 to +2 (938) 123-8372)
• License plates (e.g. BZF123 to LMA191)
• Bank accounts (e.g. 10-9282-9478563-00 to 23-1840-6492817-01)
• Passport numbers (e.g. FD194845 to CZ858584)

and so on.

imitate is a good, simple and safe default for many data types. However, it is not intended to generate perfect replacements for columns that must have special rules. For example, if a value must always start with the letter C, followed by 6 random numbers and letters, then imitate is not suitable as the C might be replaced with another letter.

The uppercase, lowercase and digits arguments can be used to disable the replacement of each of these character sets. No errors are raised if a character set is enabled but those characters are not in the string, for example, it's safe to try to replace letters in a phone number field.

imitate also accepts integer, decimal (NUMERIC/DECIMAL) and floating-point values, including numbers in JSON documents. Only the digits (0-9) are replaced. With digits: false, a number would be returned unmasked, so imitate refuses it and the run fails. Boolean values are refused too. The sign, the decimal point, the scale of a decimal and the exponent of a floating-point value are kept, and so is the type of the value. When the integer part has more than one digit, its first digit is never 0, so the number keeps its number of digits. For example, -1234.50 might become -8305.27.

A masked integer stays within the range of the narrowest of SMALLINT, INTEGER and BIGINT that holds the original value, so 2147483647 becomes at most 2147483647, and -30000 at least -32768. A column narrower than SMALLINT, or an unsigned column, can still be exceeded: for example a TINYINT holding 200 can become 873, and the write then fails. An integer beyond the range of BIGINT keeps only its number of digits. A masked floating-point value that fits a 4-byte float (REAL, or FLOAT in MySQL), that is, whose magnitude is at most about 3.4e38, stays within that range, so it still fits such a column. A larger floating-point value stays within the range of an 8-byte float (DOUBLE PRECISION). Near the largest value of its type, a digit may keep its value, even with force_change, so that the number still fits: 1e308 is returned unchanged. A masked decimal keeps only its number of digits, not its range. A single-digit negative number can become zero, so -5 may become 0. NaN and infinite floating-point values have no digits, and are returned unchanged.

Because each digit is replaced at random, masked floating-point values may also lose precision. DataMasque logs a warning, once per run, the first time imitate is applied to a decimal or floating-point value. Integers do not raise this warning. See Imitate Numeric Warning for how to disable it.

Parameters

• force_change (optional): Since characters are chosen randomly, it is possible that a character might be randomly replaced with the same one (for example, A is chosen as a replacement for A). Set force_change to true to make sure the replacement character differs. Defaults to false. Note that this makes the output slightly less random as the number of possible replacements is reduced by one.
• uppercase (optional): A boolean to enable or disable the replacement of uppercase characters. Defaults to true (uppercase characters will be replaced).
• lowercase (optional): A boolean to enable or disable the replacement of lowercase characters. Defaults to true (lowercase characters will be replaced).
• digits (optional): A boolean to enable or disable the replacement of digits. Defaults to true (digits will be replaced).

Example

This example will apply imitate masks to the phone, license_plate and validation_code.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: phone
        masks:
          - type: imitate
      - column: license_plate
        masks:
          - type: imitate
      - column: validation_code
        masks:
          - type: imitate

Show result

Before	After
phone license_plate validation_code (09) 8198822 BA981 aFec9-LIZN7 +64 (21) 0917762 GL1748 77HG8-bbA9 1-800 GET-MASQUE CDF345 Lm85-gC5D	phone license_plate validation_code (29) 01691548 BV912 bZwh0-NCZY9 +91 (45) 54173964 XP9165 01MV0-kqC7 2-975 JDV-PLASHE LCU788 Ys04-wL9V

Random text (from_random_text)

This mask replaces the column's value with randomly generated a-z characters.

Parameters

• max (required): The generated character string will be this length at maximum. The maximum length must be between 1 and 100.
• min (optional): The generated character string will be this length at minimum. If no value is supplied here, the generated string's length will always be equal to the max value.
• case (optional): The case (upper or lower) of the text generated. Mixed case will be generated if this field is left blank. Must be one of: upper, lower

Example

This example replaces the values in the name column with a random string of lower case characters between 5 and 10 characters in length.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: name
        masks:
          - type: from_random_text
            min: 5
            max: 10
            case: lower

Show result

Before	After
name Bill Chris Anastasia Judith Gordon Joel	name fjggrw bjoquazqit pljfrey sdnbomx wpoieut yptrf

Transform case (transform_case)

A mask to perform a transformation to the case/capitalisation of a string.

Parameters

• transform (required): The transformation to apply. Must be one of: uppercase, lowercase, capitalize_words (capitalizes first letter of each word), capitalize_string (capitalizes first letter only).

Example

This example will convert all values in the name column into uppercase.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: name
        masks:
          - type: transform_case
            transform: uppercase

Show result

Before	After
name Bill Chris Anastasia Judith Gordon Joel	name BILL CHRIS ANASTASIA JUDITH GORDON JOEL

Substring (take_substring)

A mask to select a substring from a column value. You may wish to use this to select or remove a subset of characters from the beginning, end, or middle of a string.

Parameters

• start_index (optional): The index of the first character to include in the selected substring, with 0 being the index of the first character in the string. Defaults to 0.
• end_index (optional): The index of the character immediately AFTER the selected substring (i.e. the end_index is exclusive). If omitted, the selection will continue until the end of the string.

Positive and negative indices can be used, i.e. the first character in a string is at index 0, the second character is at index 1, the last character is at index -1, and the second-to-last character is at index -2.

Example

This example will return only the first 3 characters of each value in the name column. The final result will return the characters at positions 0, 1, and 2. This is because the end_index is exclusive; the characters starting from the end_index value of 3 onwards are omitted from the final result.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: name
        masks:
          - type: take_substring
            start_index: 0
            end_index: 3

Show result

Before	After
name Bill Chris Anastasia Judith Gordon Joel	name Bil Chr Ana Jud Gor Joe

Replace substring (replace_substring)

A mask for transforming a selected substring of a string value. The transformation is defined by a nested sequence of masks. Matched substrings are transformed in-place, leaving the unmatched sections intact. For more complex use cases, replace_regex may be helpful.

Parameters

• masks(required): A list of masks (or dictionary mapping keys to masks) that define the transformation to apply to the selected substring. The selected substring is provided as the input to the first mask.
• start_index (optional): The index of the first character to include in the selected substring, with 0 being the index of the first character in the string. Defaults to 0.
• end_index (optional): The index of the character immediately AFTER the selected substring (i.e. the end_index is exclusive). If omitted, the selection will continue until the end of the string.
• preserve_length (optional): If set to true, then the output of the masks will be truncated or repeated until it has the same length as the original substring. This ensures the length of the entire string is unchanged. Defaults to false.

Positive and negative indices can be used, i.e. the first character in a string is at index 0, the second character is at index 1, the last character is at index -1, and the second-to-last character is at index -2.

Example

This example will replace the last 3 characters of each value in the name column with a # symbol. The start_index value of -3 indicates that the third to last character is the beginning of the substring. Because the end_index is not specified, all characters starting from the third to last character of the string until the end of the string are masked. The final result will take the characters at index position -3, -2 and -1, and replace those values with '#', leaving the rest of the string unchanged.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: name
        masks:
          - type: replace_substring
            start_index: -3
            masks:
              - type: from_fixed
                value: '###'

Show result

Before	After
name Bill Chris Anastasia Judith Gordon Joel	name B### Ch### Anasta### Jud### Gor### J###

Replace regular expression (replace_regex)

A mask for transforming sections of a string that match a certain regular expression. The transformation that is applied to each matched substring is defined by a nested sequence of masks. The matched substrings are transformed in-place, leaving the unmatched sections intact.

Parameters

• masks(required): A list of masks (or dictionary mapping keys to masks) defining the transformation to apply to each substring that matches the pattern specified in regex. The entire sequence of masks will be applied to each substring that is matched, with the matched value being provided as the input to the first mask.
• regex (required): The regular expression that will be used to search for substrings to mask. For more details on how to use regular expressions, see Common regular expression patterns.
• preserve_length (optional): If set to true, then each output of the masks will be truncated or repeated until it has the same length as the original matched substring. This ensures the length of the entire string is unchanged. Defaults to false.

Example

This example replaces all numeric characters in the driversLicence column with #. Please note that it is also best practice to wrap the regular expression in quotes to avoid special characters being misinterpreted as YAML syntax:

version: '1.0'
tasks:
  - type: mask_table
    table: '"DriversLicence"'
    key: id
    rules:
      - column: driversLicence
        masks:
          - type: replace_regex
            regex: '[0-9]'
            masks:
              - type: from_fixed
                value: '#'

Show result

Before	After
driversLicence AB123456 CD987654 EF135790 GH246802 IJ112358	driversLicence AB###### CD###### EF###### GH###### IJ######

Substitute (substitute) (deprecated)

The substitute mask was renamed to imitate in DataMasque 2.12. Unless noted in the changelog, substitute masks will continue to function with the same behaviour and options as the imitate mask. Refer to the imitate mask documentation for usage details and configuration. Backwards compatibility may be removed in a future DataMasque version, so substitute should be replaced with imitate in rulesets, when possible.