Tabular file masks (`mask_tabular_file`)

Each mask_tabular_file task defines masking operations to be performed on a file or set of files. mask_tabular_file tasks support the following file types:

Comma separated (CSV), Tab delimited (TBV), and other character-delimited files
Parquet
Fixed-width columns
Parameters
Tabular File Type Detection
Quoting options
Parquet INT96 Timestamp Handling
- When to use this parameter
Fixed Width File Masking Parameters
Supported Parquet column data types
- Scalar types
- Complex types

Parameters

Each task with type mask_tabular_file is defined by the following parameters:

rules (required): A list of masking rules (or dictionary mapping arbitrary keys to rules) to apply to the table. Rules are applied sequentially according to the order they are listed. Regular (non-conditional) masking rules are defined by the following attributes:
- column (required): A column within the tabular file intended for masking. This is the header row of the column (for CSVs) or name of column (for parquet files).
- masks (required): A list of masks (or dictionary mapping arbitrary keys to masks) to apply to the column. Masks are applied sequentially according to the order they are listed, so the output value from each mask is passed to the next mask in the list. Each type of mask has a different set of required attributes, as described for each type in Mask functions.
recurse (optional): A boolean value, when set to true any folders in the Data Source will be recursed into and the files contained will also be masked. Defaults to false.
workers (optional): The number of parallel workers to use for this masking task. Defaults to 1.
skip (optional): Specifies files to not include in the masking run.
- regex (optional): Files that match this regular expression will not be included for masking.
- glob (optional): Files that match this glob will not be included for masking.
- match_against (optional): Choose if the skip rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.
include (optional): Specifies files to include in the masking run. If not set, all files will be masked (except those skipped by a skip rule).
- regex (optional): Files that match this regular expression will be masked.
- glob (optional): Files that match this glob will be masked.
- match_against (optional): Choose if the include rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.
encoding (optional): The encoding to use when reading and writing files. Defaults to UTF-8. Refer to Python Standard Encodings for a list of supported encodings.
coerce_int96_timestamp_unit (optional): Unit for INT96 timestamp coercion when processing Parquet files. Valid values are ms (milliseconds), us (microseconds), ns (nanoseconds), or null. Defaults to ns if unspecified or null. Set to ms to prevent timestamp corruption for extreme dates like 9999-12-31. See Parquet INT96 Timestamp Handling for more details.
delimiter (optional): The character that is used to delimit columns in character-delimited files (e.g. , for CSVs or \t for TDVs [Tab Delimited Values]). Defaults to ,.
quoting (optional): Controls how the masked file will be quoted. Valid values are all,minimal and none. Defaults to minimal. See Quoting options.
disable_warning_on_glacier_storage (optional): Suppress the warning for files archived with Amazon S3 Glacier storage. See Skipping files archived with Amazon S3 Glacier storage.
column_names (optional): An array of string defining the names of the fixed-width columns or non-header character-delimited file and used to refer to them in masking rules. For masking fixed-width files, it is required if fixed_width_extension is specified, and must match the length of fixed_width_columns_indexes.
fixed_width_extension (optional): The file extension that fixed-width files have. Not required if no fixed-width files are to be masked. Should not include a leading . (e.g. specify txt not .txt).
fixed_width_columns_indexes (optional): An array of two-element arrays of start and end indexes of the fixed width columns. Required if fixed_width_extension is specified.
fixed_width_column_names (optional, deprecated): This option was renamed to column_names option. However, it is still supported for backward compatibility.
fixed_width_too_wide_action (optional): The action to take if masked data exceeds the width of the column, can be either truncate to truncate the value to fit in the column or error to raise an error and stop the masking run. Defaults to truncate.
fixed_width_line_ending (optional): The line ending to use when writing out the fixed width data. Will attempt to be detected from the input file, otherwise defaults to \n.
trim_leading_whitespace (optional): Whether to trim leading whitespace from all columns in the fixed width file during reading. Defaults to false. Set to true to remove leading spaces from columns.
trim_trailing_whitespace (optional): Whether to trim trailing whitespace from all columns in the fixed width file during reading. Defaults to true. Set to false to preserve trailing spaces in columns.

For more information about:

How tabular file types are detected, see Tabular File Type Detection.
Parameters for fixed-width file masking, see Fixed Width File Masking Parameters.
The ordering of skip and include, please refer to Include/Skip.
Table joins are not supported in tabular file masking.

Note: regex/glob will match to the path from the base directory specified in the source connection, consider adding .*(regex) or *(glob) to the beginning of the expression for matching. For example: If the structure is /path1/path2/target_file.json, if the base directory is path1/ and recurse: true in the ruleset, the regex/glob will try to match path2/target_file.json. When including a path by specifying a glob such as target_path/* the recurse option needs to be set to true otherwise the included path won't be entered and the files it contains will not be masked.

version: "1.0"
tasks:
  - type: mask_tabular_file
    recurse: true
    skip:
      - regex: '.*\.json'
      - glob: "input/*"
    include:
      - glob: "other_inputs/*"
    rules:
      - column: name
        masks:
        - type: from_fixed
          value: REDACTED

Tabular File Type Detection

DataMasque uses file extensions to determine how tabular files are loaded for masking. The extension comparison is not case-sensitive. Files with extension parquet are treated as Apache Parquet files. If a fixed_width_extension is specified (see Fixed Width File Masking Parameters below) then the file is loaded as fixed-width. All other file types are loaded as character-delimited. If the delimiter parameter is unspecified then it defaults to , (i.e. CSV).

Note Delimiters must be a single character, and should be quoted with double quotes. For example: delimiter: " " for a space or delimiter: ";" for a semicolon. Tabs or other invisible delimiters should use the correct escape sequence with a backslash. For example, delimiter: "\t" for a tab.

The skip, include and delimiter options should be used together to specify the correct delimiter for each file type that is to be masked.

Note CSV files require header columns for tabular masking as the header columns are used as column names during masking. CSV and fixed-width files are all string based files, therefore values should be cast to other types if being used with masks that require specific types (e.g. numeric_bucket). To do this use a typecast mask, for more information please refer to Typecast

DataMasque will only attempt to load fixed width files if fixed_width_extension is specified, and will treat any files with this extension as fixed-width. See also Fixed Width File Masking Parameters.

Once files are loaded they are all masked in the same way, that is, rules are executed and applied on a per row/column basis regardless of the original source type. Data will be written back out in the same format as it was read.

Quoting options

Character delimited files (such as CSVs) handle special characters by quoting the field when written to a file. Special characters include the delimiter (e.g. , for CSV), double quotes (") or new line characters.

The field content is wrapped in double quotes, and double quotes are replaced by two double quotes.

For example, the text:

She said, suddenly, "Yes, that's right!"

would be written to a CSV file as:

"She said, suddenly, ""Yes, that's right!"""

This quoting will happen automatically if the field requires quoting, but the behavior can be modified based on the quoting option.

all: quote all fields.
minimal (default): only quote those fields which contain special characters such as delimiter, double quote or newlines.
none: never quote fields. If the delimiter occurs in the output data then an error will be raised.

Parquet INT96 Timestamp Handling

Parquet files may contain columns of INT96 timestamp type, which is an older deprecated timestamp format. When processing these files, DataMasque may experience precision issues with extreme dates (such as dates in the year 9999, e.g. 9999-12-31 used to represent an open-ended date, or date with no expiry) that can result in the timestamps being changed even if these columns are not being masked.

The coerce_int96_timestamp_unit parameter allows you to control how these timestamps are processed:

null: Equivalent to 'ns'.
'ms': Coerces timestamps to millisecond precision. Prevents timestamp changes for extreme dates.
'us': Coerces timestamps to microsecond precision. Generally prevents timestamp changes for extreme dates.
'ns' (default): Coerces timestamps to nanosecond precision. May cause timestamp changes for extreme dates.

When to use this parameter

If you encounter timestamp changes when masking Parquet files containing extreme dates, (particularly dates like 9999-12-31), set coerce_int96_timestamp_unit: ms to resolve the issue. Or if you know the precision your Parquet files use, specify that.

Example:

version: "1.0"
tasks:
  - type: mask_tabular_file
    coerce_int96_timestamp_unit: ms
    rules:
      - column: customer_name
        masks:
          - type: from_fixed
            value: MASKED_NAME

Note: This parameter only affects Parquet files with INT96 timestamp columns. It has no effect on other file types or Parquet files using standard timestamp formats.

Fixed Width File Masking Parameters

Masking of fixed-width files is only attempted if fixed_width_extension is specified. If fixed_width_extension is present in the ruleset without fixed_width_columns_indexes and column_names, then an error will be raised. However, it is valid to have fixed_width_columns_indexes and column_names missing if fixed_width_extension is also absent.

If fixed_width_extension is set, then DataMasque will treat any files with that extension as fixed-width and load them based on the other fixed-width options. To assist in explaining the rules, consider an example file called users.txt. This is the content:

Adam    2010-01-01  AAA-1111
Brenda  2010-01-01  EEE-5555
Charlie 2010-02-02  GGG-7777

It has 3 columns, the first containing a name, which is from index 0 to 8. The second column contains a date and spans from 8 to 19. The final column contains a transaction ID and spans from index 19 to 27.

Note these indexes are specified to be contiguous as some fixed-width formats require contiguous columns, therefore a trailing space is included in the first and second columns. By default, DataMasque preserves leading spaces and trims trailing spaces when the data is read. This behavior can be controlled using the trim_leading_whitespace and trim_trailing_whitespace options. Contiguous columns are not required though, so the same result could be achieved with indexes.

Since fixed-width files do not have column headers, the ruleset must also specify these. They can be any arbitrary valid column identifier (i.e. alphanumeric string without special characters) and are used to identify the columns in the masking rules. In this case they will be named name, date and transaction_id.

Considering these rules will yield a ruleset like this:

version: "1.0"
tasks:
  - type: mask_tabular_file
    recurse: true
    fixed_width_extension: txt
    fixed_width_columns_indexes:
      - [0, 8]
      - [8, 19]
      - [19, 27]
    column_names:
      - name
      - date
      - transaction_id
    rules:
      - column: name
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed
      - column: date
        masks:
          - type: from_random_date
            min: '1950-01-01'
            max: '2000-12-31'
          - type: typecast
            typecast_as: string
            date_format: '%Y-%m-%d'
      - column: transaction_id
        masks:
          - type: imitate

Note that when this ruleset is executed, DataMasque will still load any CSVs or Parquet files it encounters, however it will use the standard loaders instead of applying the fixed width rules. In this case fixed-width rules will only be used for txt files.

This can be useful if CSV or Parquet files exist in the source with the same columns and need to be masked in the same manner. If these files exist, and they shouldn't be masked, then skip rules should be added to skip them.

Supported Parquet column data types

Scalar types

The following scalar Parquet column data types are supported for masking, and likewise for use with seed_filter_column in a from_file mask, from_column, if, or hash_columns.

Null type: null (sometimes called na)
Integer types: int8, int16, int32, int64
Unsigned integer types: uint8, uint16, uint32, uint64
Floating-point types: float, double, decimal128
Boolean type: bool
String types: string, large_string
Binary types: binary, large_binary
Date and time types: date32, date64, timestamp, time32, time64, duration

Note: Writing masked values of decimal128 type requires decimal-format data. Attempting to write integer or float values to a column of decimal128 type causes an error. Use a typecast mask with typecast_as: decimal to convert the masked value to decimal.

Complex types

DataMasque supports the complex Parquet data types struct, map, and list, which are sometimes referred to as nested columns.

Note: Map keys must be strings. DataMasque does not support maps with keys of other data types. Map values, and the contents of lists and structs, can be of any supported type: either another struct, map, or list, or any of the supported scalar types listed above.

For masking struct, map, and list columns, use the json mask. In the ruleset, specify the name of the column, and then the JSON path to the field to be masked, treating lists as JSON arrays, and structs and maps as JSON objects. For example, suppose you have the following Parquet column of map type:

column name: staff_by_office
type: map<string, list<struct<first_name: string, surname: string>>>

i.e. each key in the map is an office name, and each value is a list of staff members' names, stored as a struct with fields first_name and surname.

Then the following ruleset will mask the surname of all staff in the north office:

version: "1.0"
tasks:
  - type: mask_tabular_file
    rules:
      - column: staff_by_office
        masks:
          - type: json
            transforms:
              - path:
                  - north
                  - '*'
                  - surname
                masks:
                  - type: from_fixed
                    value: REDACTED

You can also copy one complex column to another of the same type using the from_column mask, and use a value within a complex column as a hash_columns source by specifying a json_path as detailed above.

Complex columns cannot be used as the seed_filter_column in from_file masks.

Conditional masking for Parquet

Complex columns can be referenced in if conditions. Specify the json_path to the field to be compared.

Other data types

Attempts to mask or reference a column of type not mentioned above will cause the task to fail. However, you can still mask other columns in a file containing one or more columns with an unsupported data type.

Tabular file masks (mask_tabular_file)