DataMasque Portal

Object file masks (mask_file)

Each mask_file task defines masking operations to be performed on a file or set of files. mask_file tasks support the following file types:

In addition, any file type can be used with from_blob, which will replace the entire file with another file.

Note The file type will be determined solely by the file extension, not by its content.

Parameters

Each task with type mask_file is defined by the following parameters:

  • rules (required): A list of masking rules (or dictionary mapping arbitrary keys to rules) to apply to the table. Rules are applied sequentially according to the order they are listed. Regular (non-conditional) masking rules are defined by the following attributes:
    • masks (required): A list of masks (or dictionary mapping arbitrary keys to masks) to apply to the file. Masks are applied sequentially according to the order they are listed, so the output value from each mask is passed to the next mask in the list. Each type of mask has a different set of required attributes, as described for each type in Mask types.
  • recurse (optional): A boolean value, when set to true any folders in the Data Source will be recursed into and the files contained will also be masked. Defaults to false.
  • workers (optional): The number of parallel workers to use for this masking task. Defaults to 1.
  • skip (optional): Specifies files to not include in the masking run.
    • regex (optional): Files that match this regular expression will not be included for masking.
    • glob (optional): Files that match this glob will not be included for masking.
    • match_against (optional): Choose if the skip rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.
  • include (optional): Specifies files to include in the masking run. If not set, all files will be masked (except those skipped by a skip rule).
    • regex (optional): Files that match this regular expression will be masked.
    • glob (optional): Files that match this glob will be masked.
    • match_against (optional): Choose if the include rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.
  • encoding (optional): The encoding to use when reading and writing files. Defaults to UTF-8. Refer to Python Standard Encodings for a list of supported encodings.
  • disable_warning_on_glacier_storage (optional): Suppress the warning for files archived with Amazon S3 Glacier storage. See Skipping files archived with Amazon S3 Glacier storage.

For more information about the ordering of skip and include please refer to Include/Skip.

Note: Increasing the number of workers can reduce masking time by masking files in parallel, but extra workers will not mask different parts of the same file. This is useful if there are a large number of files to be masked, but will not reduce the amount of time required to mask a single large file. For files that can be split into multiple parts (for example, tabular files or multi-record files), manually chunking and recombining the files can improve masking speed.

Note: regex/glob will match to the path from the base directory specified in the source connection, consider adding .*(regex) or *(glob) to the beginning of the expression for matching. For example: If the structure is /path1/path2/target_file.json, if the base directory is path1/ and recurse: true in the ruleset, the regex/glob will try to match path2/target_file.json. When including a path by specifying a glob such as target_path/* the recurse option needs to be set to true otherwise the included path won't be entered and the files it contains will not be masked.

Supported file types

In general, mask_file has been designed to mask XML, JSON, NDJSON, and Apache Avro files. Each file is loaded as a string and passed to the masks. Therefore, to mask a JSON file, a json mask would be implemented, like the following example:

version: "1.0"
tasks:
  - type: mask_file
    recurse: true
    skip:
      - regex: '.*\.json'
      - glob: "input/*"
    include:
      - glob: "other_inputs/*.json"
    rules:
      - masks:
          - type: json
            transforms:
            - path: ['name']
              masks:
                - type: from_fixed
                  value: REDACTED

This would replace the root name attribute in the JSON with the text REDACTED.

Multirecord Files (NDJSON/Avro)

NDJSON (Newline Delimited JSON) (.ndjson) files can be masked by specifying a json mask in the ruleset which will be applied to each line of JSON in the NDJSON file.

For NDJSON files, a similar ruleset can be specified which will be applied to each record of the file, rather than being applied to the entire file.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - masks:
        - type: json
          transforms:
            - path: ['name']
              masks:
                - type: from_fixed
                  value: REDACTED

This would replace the root name attribute for each record in the NDJSON with the text REDACTED.

Apache Avro (.avro) files can be masked in the same way as NDJSON files. When masking Avro files there also needs to be an Avro Schema file (.avsc) in the same directory and with the same name as the .avro file, as the schema file is required to write the masked .avro file to ensure data type consistency. For example, if a file is called data.avro DataMasque will expect to find a schema file called data.avsc in the same directory.

Note: The Avro Schema file (.avsc) will be copied unmasked into the output directory. Deterministic Masking is also supported for multirecord files, for more information please refer to Deterministic Masking with Files.

Similarly, for XML files, use an xml mask:

version: "1.0"
tasks:
  - type: mask_file
    recurse: true
    skip:
      - regex: '^(.*).xml'
      - glob: "input/*"
    include:
      - glob: "other_inputs/*.xml"
    rules:
      - masks:
          - type: xml
            transforms:
            - path: 'User/Name'
              node_transforms:
                - type: text
                  masks:
                    - type: from_fixed
                      value: REDACTED

This would replace the content of the node(s) at User/Name with the text REDACTED.

To mask other types of files, basic redaction is possible. For example, to replace the contents of every txt file with the text REDACTED:

version: "1.0"
tasks:
  - type: mask_file
    recurse: true
    include:
      - glob: "*.txt"
    rules:
      - masks:
          - type: from_fixed
            value: REDACTED

It is possible to use any mask that accepts text input (or no input), although their effectiveness will depend on the size and content of the input file.

Note also, that files that have not been processed will not be copied from the source to the destination. That is, DataMasque will either load a file (based on skip/include rules), mask it, then copy it to the destination, or it will ignore the file. Unmasked files will not be copied to the destination.