File tasks

Each mask_file or mask_tabular_file task specifies the masking rules to apply to each file in the base directory and/or any subdirectories. As well as any files/directories intended to be skipped or included, and any conditionals required to define which data to mask in the masking process. Masking rules and masks are applied sequentially in the order they are listed. When multiple masks are combined in sequence, the output value from each mask is passed as the input to the next mask in the sequence.

Note If the source connection and destination connection are of the same type and have the same base directory, the files will be overwritten. The list of files is read at the start of the masking run, so new files added during the masking run will not be masked and will not be present in the destination.

---

A ruleset consists of a list of tasks to be performed in sequence on the target data source after the file is masked, it is then written to the selected data destination.

---

The two file masking tasks have their own pages: Object file masks for mask_file, and Tabular file masks for mask_tabular_file.

• File sensitive data discovery
  • Parameters
  • Examples
• Choosing files to mask with include/skip
  • include/skip examples and common scenarios
• Skipping files archived with Amazon S3 Glacier storage

File sensitive data discovery

The run_file_data_discovery task type inspects the metadata and optionally the contents of your files, searching for columns which are likely to contain sensitive data. On completion, a report is generated containing a summary of all identified columns, and their current masking coverage.

See the Sensitive Data Discovery guide for more information on this feature.

Parameters

• in_data_discovery (required): Enable In-Data discovery to examine the contents of discovered files for sensitive data matches. See In-Data Discovery.
• custom_keywords (optional): Additional custom keywords and to facilitate sensitive data discovery.
• disable_built_in_keywords (optional): Disable DataMasque's built-in discovery keywords. See Built-in data discovery keywords.
• disable_global_custom_keywords (optional): Disable any global custom keywords configured on the settings page.
• disable_global_ignored_keywords (optional): Disable any global ignored keywords configured in the settings page.
• disable_warning_on_glacier_storage (optional): Suppress the warning for files archived with Amazon S3 Glacier storage. See Skipping files archived with Amazon S3 Glacier storage.
• ignored_keywords (optional): Additional ignored keywords to facilitate sensitive data discovery.
• encoding (optional): The encoding to use when reading files. Defaults to UTF-8. Refer to Python Standard Encodings for a list of supported encodings.
• include (optional): Specifies files to include in the discovery run. If not set, all files will be discovered (except those skipped by a skip rule).
  • regex (optional): Files that match this regular expression will be discovered.
  • glob (optional): Files that match this glob will be discovered.
  • match_against (optional): Choose if the include rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.
• recurse (optional): A boolean value, when set to true any folders in the Data Source will be recursed into and the files contained will also be discovered. Defaults to false.
• skip (optional): Specifies files to not include in the discovery run.
  • regex (optional): Files that match this regular expression will not be included for discovery.
  • glob (optional): Files that match this glob will not be included for discovery.
  • match_against (optional): Choose if the skip rule is applied to the whole path or just the filename. Valid choices are path or filename. Defaults to path.

Examples

The following shows an example ruleset that will execute only the run_file_data_discovery task and no masking tasks. This example usage may be useful to determine a starting point for developing your masking rules.

version: '1.0'
tasks:
  - type: run_file_data_discovery
    include: [{"glob": "*.csv"}]
    in_data_discovery: {"enabled": False}

This example enables In-Data discovery:

version: '1.0'
tasks:
  - type: run_file_data_discovery
    include: [{"glob": "*.csv"}]
    in_data_discovery: {"enabled": True}

---

Choosing files to mask with include/skip

When specifying which files to include or skip for a mask_file/mask_tabular_file task, the order of checking needs to be considered. The skip rules are checked first, followed by the include rules, which means skip takes precedence. Warning! If a file matches both the skip and include rules, that item will not be included in the masking task.

Rules are applied with "or" logic, that is, as long as the file matches any rule in skip or include, it will be skipped/included, respectively.

In comparisons, the file's path does not include a leading /.

By default, include and skip rules are applied to the file's path, relative to the base_directory. By specifying match_against: filename on the include/skip, the rule will be applied to just the file's name.

Filtering is performed only on files, not on directories. If recurse is true, a directory will always be entered into, even if it would be skipped based on filters. If a directory has no files to be masked (they are all skipped, or none are included) then an empty directory will not be created on the target.

glob matching does not support the ** syntax for multi-level directory matching. If the depth of directories is unknown, or more advanced path matching is required regex filtering should be used.

These rules are explained in more detail below, with some examples and common file filtering scenarios.

include/skip examples and common scenarios

In general, glob is used for simple matching, and regex should be considered when more advanced rules are required. Note that some of these examples use a mask_file task and others use a mask_tabular_file task, however there is no difference in the way the file matching rules are executed between these different task types.

Mask all files with a certain extension

The ruleset will include all files that end with .xml inside the base directory and any subdirectories.

version: "1.0"
tasks:
  - type: mask_file
    recurse: true
    include:
    - glob: "*.xml"

The recurse option will enter every subdirectory. Since the match_against option has not been specified the default path option is used, so the glob is applied against the file's path. However since a path like dir1/dir2/file.xml and a filename like file.xml both end with .xml they both match the specified glob; in this case specifying match_against: filename would have no effect.

Mask all files with a specific start and end

This ruleset will mask any file that starts with user and has a .json extension. For example, user_1.json, users_file.json, etc. These files can be anywhere inside the source. The match_against option is set to filename.

version: "1.0"
tasks:
  - type: mask_file
    recurse: true
    include:
    - regex: "^user.*\.json$"
      match_against: filename

Explaining the entire regular expression syntax is beyond the scope of these examples, but since this pattern is quite common we will briefly explain how it works:

• ^ means the expression must match start of the path, it can't just be found somewhere in the middle.
• user has no special meaning, so it matches the literal user part of the path.
• .* has two parts, . means match any character and * means match the preceding expression (the .) any number of times. So together they mean match any number of any characters.
• \. matches a literal .. The \ escapes the . so it doesn't behave as matching any character.
• json has no special meaning, so it matches the literal json extension.
• $ means the expression must also match at the end of the path, .json can't just appear somewhere in the middle.

match_against: filename is specified so that the regex pattern does not match user in a directory name. For example, we do not want it to match the path user_data/file_1.json, but that full path does satisfy the regex. By specifying match_against: filename, the regex is applied only to the filename part (file_1.json) which does not match.

Notes:

• Regex patterns only match at the beginning of the path (or filename, if using match_against: filename). So the pattern above will not match some_user_data.json as the word user is not at the start of the filename, even if the leading ^ start anchor is omitted from the regex.
• When using match_against: filename, a glob or regex with a / in it will cause the pattern to never match anything, since filenames never contain a /. DataMasque does not prevent you from using a / in a pattern when matching against filename, so if there are files you expect to be included, but they are not, this could be the reason why.
• match_against: filename has no effect when using only glob patterns. Globs always match at the end of the path, and wildcards do not match across directory separators (/). It follows that the above example can be rewritten more succinctly as glob: user*.json with no match_against parameter. However, were the regular expression more complex, it may not be expressible as a glob pattern.

Mask files of a certain type only in the base directory

To mask CSV files only in the base directory, disable the recurse option.

version: "1.0"
tasks:
  - type: mask_tabular_file
    include:
    - glob: "*.csv"

Without recurse: true, subdirectories will not be entered into, so only .csv files in the base directory will be masked.

Mask files in nested directories of known depth

In this example, there is a directory layout like this:

• users1.csv
• users2.csv
• staff/users1.csv
• staff/users2.csv
• staff/management/users1.csv
• staff/management/users2.csv
• staff/contractors/users1.csv
• staff/contractors/users2.csv

(Assume there are also some files of other types that are not to be masked).

All users*.csv files should be masked, except those in the base directory. Since we know the depth of the directories (staff/ is one level; staff/management/ and staff/contractors/ are two levels) two globs can be added to match these.

version: "1.0"
tasks:
  - type: mask_tabular_file
    include:
    - glob: "staff/*.csv"
    - glob: "staff/*/*.csv"

Since the matching is performed on the path, none of the files in the base directory match. Files in the staff/ directory match the first glob and files in staff/management/ or staff/contractors/ match the second. Since include use OR logic then as long as at least one include rule matches a file will be included.

Note: Globs also match at the end of a more deeply-nested path. For example, with the above globs, folder1/folder2/staff/file.csv also matches. If you want to avoid this behaviour, use a regex starting with ^staff/ to ensure that the pattern only matches a folder named staff when it is in the base directory - see the following section.

Mask files in all nested directories

If the depth of the directories is unknown, or could be very deep, so that a lot of glob rules need to be created, then using regex to apply regular expressions to the path can be a better option.

Following on from the previous example, CSV files might be nested inside a deep directory tree. The following ruleset uses a regex to match any .csv file inside staff/ or any of its child directories.

version: "1.0"
tasks:
  - type: mask_tabular_file
    include:
    - regex: "^staff/.*\.csv$"

Any path that starts with staff/ and ends with .csv will be matched by this regex.

Often you will be able to apply a slightly modified version of this regex, by just changing the start and end to match your particular paths and file extensions.

For more information on the regular expression syntax used by DataMasque, check the Python Regex Howto.

Mask all files with a given extension, except those with a certain prefix

In this example all .avro files in the base directory will be masked, except the ones starting with users*. This means files like sales_1.avro, data.avro, orders-for-2020.avro will be masked, but users-list.avro and users-exported.avro would not be. It uses an include rules to include all .avro files and a skip rule to skip any starting with users.

version: "1.0"
tasks:
  - type: mask_file
    include:
    - glob: "*.avro"
    skip:
    - glob: "users*.avro"
      match_against: filename

The file users-exported.avro matches both *.avro and users*.avro, but since skip takes precedence the file will not be included. The file data.avro only matches *.avro so it will be included.

Mask files in all nested directories, excluding those with a given prefix

This example will include all .csv files in the staff/ directory, except those that start with data. For example, it matches staff/sales.csv and staff/management/users.csv but not staff/data01.csv or staff/management/data-list.csv.

version: "1.0"
tasks:
  - type: mask_tabular_file
    include:
    - regex: "^staff/.*\.csv$"
    skip:
    - glob: "data*.csv"
      match_against: filename

This ruleset show that glob and regex can be combined to produce flexible and simple matching rules.

Mask files that match a particular pattern.

This example includes XML files that start with users_, then have at least one digit (the \d+ parameter). But since it uses match_against: filename it will not match against the path, therefore files inside a directory starting with users_ won't match.

version: "1.0"
tasks:
  - type: mask_tabular_file
    include:
    - regex: "^users_\d+.xml$"
      match_against: filename

Some example results are:

• users_5.xml: match
• users_10.xml: match
• users_list.xml: no match (the filename doesn't match the regex)
• users_5/file.xml: no match (the path matches the regex but comparison is performed against the filename)
• users_5/users_10.xml: match (the filename users_10.xml matches the regex)

Skipping files archived with Amazon S3 Glacier storage

If a file is archived with Amazon S3 Glacier storage, DataMasque cannot read it and will show a warning that it is skipping the file.

• disable_warning_on_glacier_storage (optional): Set this to true on a mask_file, mask_tabular_file or run_file_data_discovery task to suppress the warning and skip the file silently. For a run_file_data_discovery task this also excludes Glacier-archived files from the Files Could Not Be Scanned Warning shown with the end-of-run discovery summary. Defaults to false.