DataMasque Warnings

• Multiple Rules Warning
• Value Truncation Warning
• Row Limit Reached Warning
• Value Limit Reached Warning
• Files Not Masked Warning
• Glacier Files Warning
• Files Could Not Be Scanned Warning
• No Files Scanned Warning
• Use Of Conditionals Warning
• Missing Unique Index Warning
• Skipped Character Classes Warning
• Snowflake Time Travel Warning/Error
• Materialised Float Column Warning

This document describes warnings that DataMasque may generate in the course of normal masking runs, and, how to disable them (if they can be disabled).

Multiple Rules Warning

This warning is displayed if a table or column is masked by more than one rule in a single ruleset, which may indicate that a rule was mistakenly included more than once.

Example

The following tables are configured to be masked multiple times by `mask_table` tasks in the ruleset: …

The following columns are configured to be masked multiple times in the ruleset: …

To Disable

Add disable_warning_on_duplicate_rules: true at the top level of the ruleset.

Value Truncation Warning

This warning is displayed when auto_truncate_value is enabled and a masked value is wider than its target string column, so DataMasque truncates the value to fit. Only string-type columns are truncated. The truncated value may no longer be valid, for example a shortened email address.

Example

One or more masked values for column `email` exceeded its maximum width of 25 and were truncated. The truncated values may no longer be valid.

To Disable

Add disable_warning_on_auto_truncate_value: true at the top level of the ruleset, or on the specific rule where you want to suppress the warning.

Row Limit Reached Warning

This warning is displayed when a license imposes a row limit and the data was larger than it.

In a database run the limit applies per table, to the table the ruleset names, and the rows past it are left in place unmasked; rows in child tables reached through a cascading mask are updated in full. In a file run it applies per file, to each tabular file a mask_tabular_file task masks. What happens to the rows past a file's limit depends on the license: a Sandbox license writes them to the output file unmasked, while a Free Trial license leaves them out of the output file. The message names which of the two happened.

It cannot be disabled: partly masked data still holds real values, so the run completes with warnings to make that unmissable. A max_rows you set yourself is reported the same way but without the warning, since bounding the run was your own choice.

A dry run reports the limit it would reach without this warning, because nothing is written and no partly masked data exists.

Some columns cannot be masked part-way without producing duplicate values or stranding rows in tables that reference them. Those are exempt from the limit and masked in every row, so a table named here may still have columns masked throughout; the run log lists them on a separate Row limit exempt line. See Licensing for exactly which columns qualify.

Where every column masked on a table is exempt, nothing is cut and this warning is not shown: the run reports that the limit did not apply and finishes without warnings.

Example

Row limit reached: masked the first 10,000 rows of `public.customers`. The remaining rows are unmasked.

For a file run:

Row limit reached: masked the first 10,000 rows of file "bucket/customers.csv", which contains 25,000. The remaining 15,000 rows are written through unmasked.

Value Limit Reached Warning

This warning is displayed when a Sandbox license's per-file value limit of 10,000 is reached and the ruleset matched more than that. Only the values a ruleset masks are counted: the rest of the document is neither counted nor changed.

A Free Trial license sets no value limit. It bounds a document by its size instead and skips it whole, so a trial file is never masked in part this way. See Licensing for the limits each license carries.

It applies to the JSON and XML files a mask_file task masks, including the NDJSON and Avro files that hold many records. These are masked as documents rather than row by row, so they are bounded by the values they mask instead of by a row count. The values past the limit are written to the output file unmasked, so the file comes back whole.

The limit applies per file. A file holding many records gives them one allowance between them rather than one each. PDF, DOCX, PPTX and other generic files are not covered: they are masked as a single item, with no values to count, and are bounded by the file limit instead.

It cannot be disabled, for the same reason the row limit's warning cannot: a partly masked file still holds real values.

A dry run reports the limit it would reach without this warning, because nothing is written and no partly masked file exists.

Example

The count of remaining values is the number the ruleset would have masked, not the number the file holds.

Value limit reached: masked the first 10,000 values of file "bucket/patients.json". The remaining 4,312 values are written through unmasked.

Files Not Masked Warning

At the end of a file masking run, DataMasque lists the files it did not mask, with the number of files for each reason:

• files past the license's limit on the number of files or file size
• files archived with S3 Glacier
• tabular files that are empty or have no rows

This information is logged as a warning when some files were unmasked due to license limits, to draw attention to the fact some files were left unmasked. This warning cannot be disabled.

Files archived with S3 Glacier also have a Glacier Files Warning of their own, which can be disabled.

Example

7 files were not masked:
- 1 file past the Trial license limit of 5 files: "customers_6.json"
- 3 files larger than the Trial license file size limit of 1MB
- 2 files archived in S3 Glacier storage
- 1 file empty or with no rows

Glacier Files Warning

On Amazon S3, when masking files or performing a file discovery run, this warning will be shown if a file has been archived with S3 Glacier.

For file masking, this means the file will not be copied to the destination connection. For discovery, it means that DataMasque will not be able to determine if the file contains sensitive information.

Example

Skipping masking file "archived_with_glacier.csv": 
files archived with Glacier can not be masked.

To Disable

Add disable_warning_on_glacier_storage: true to the specific mask_file, mask_tabular_file or run_file_data_discovery task where you want to disable the warning. For a run_file_data_discovery task this also excludes Glacier-archived files from the Files Could Not Be Scanned Warning.

Files Could Not Be Scanned Warning

At the end of a file data discovery run, this warning is displayed if any files could not be scanned: the file could not be read (including files archived with S3 Glacier), a Parquet file contained a column with an unsupported data type, or an unexpected error occurred during discovery. The discovery results may therefore be incomplete. The count is an aggregate; the file discovery summary above it breaks the files down by reason, and the run report lists every affected file.

Files that were not scanned for any of the following reasons do not trigger this warning, because nothing went wrong when DataMasque considered them. The reason names are the ones used in the run report and in the file discovery summary:

• File type unsupported by data discovery
• File type could not be determined
• File was empty
• Did not match any include filter
• Matched a skip filter

Example

3 files could not be scanned. See the messages above for details on each file, or download the run report from the Run Logs page for the full list.

To Disable

This warning cannot be disabled in general. The exception is files archived with S3 Glacier: add disable_warning_on_glacier_storage: true to the run_file_data_discovery task to exclude those files from the warning and its count.

No Files Scanned Warning

At the end of a file data discovery run, this warning is displayed if the source contained files but none of them were scanned: for example, a source holding only unsupported file types, or include/skip filters that matched nothing scannable. The file discovery summary above the warning shows, per file extension, why each file was not scanned.

Example

No files were scanned. All 243 files in the source were skipped for the reasons shown in the summary above.

To Disable

This warning cannot be disabled. To resolve it, adjust the task's include/skip filters, or point the connection at a source containing file types supported by discovery: CSV, JSON, NDJSON or Parquet.

Use Of Conditionals Warning

Displayed when using conditional masking features that may leave some data unmasked. For example, skip, where and if/else.

Example

Use of the conditional masking features `where`, `skip`, or `if`/`else`, 
may mean your masking rules are not applied to some database rows or values.

To Disable

Add disable_warning_on_use_of_conditionals: true to the specific mask_table task where you want to disable the warning.

Missing Unique Index Warning

For mask_unique_key tasks, the key column(s) being masked are expected to have a unique index. If a unique index does not exist, then DataMasque will temporarily create one for the duration of the masking run, and will issue a warning that one was created.

Example

Unique requirement for specified key could not be validated, 
as no adequate unique index is found for key column: … 

To Disable

Add disable_warning_on_missing_unique_index: true to the mask_unique_key task to disable the warning for that task. DataMasque will silently add the unique index during masking, without showing a warning.

Alternatively, add a unique key/index to the keys(s) before beginning the masking task.

Skipped Character Classes Warning

When using a from_unique_imitate mask, if either of the skip_letters or skip_digits parameters are specified true, a warning is shown to let you know that the configuration will leave some data unmasked.

Example

No letters will be masked.
If the column to be masked only contains letters, nothing will be masked.

No digits will be masked.
If the column to be masked only contains digits, nothing will be masked.

To Disable

Add disable_warning_on_skipped_characters: true as a parameter to the from_unique_imitate mask.

Snowflake Time Travel Warning/Error

When masking data on Snowflake databases, if a table has Snowflake Time Travel enabled, the unmasked data can be recovered by restoring a snapshot of the table from a previous point in time. As such, DataMasque disallows masking until Time Travel is turned off for that table.

Refer to the Snowflake documentation for more information about Time Travel, including how to disable it at the table, schema, database or account level.

Example Error

Snowflake Time Travel is enabled for table `myschema.mytable` (configured explicitly on the table).
When Snowflake Time Travel is enabled, the unmasked data can be recovered. As such, masking is not secure.
Please disable Snowflake Time Travel on the table `myschema.mytable` before masking.
Refer to https://docs.snowflake.com/en/user-guide/data-time-travel for more information.

To Disable

If you want to mask the table without turning off Time Travel, specify allow_masking_tables_with_time_travel: true at the top level of the ruleset. The above message will appear in the run logs as a warning instead of an error, and masking will proceed as normal.

Materialised Float Column Warning

When materialising a table reference (copying the table reference into a temporary table), each column is recreated with a matching type. A floating-point column is materialised like any other, but the target database may store and compare it slightly differently than DataMasque does. The risk is specific to using the column as a join or match key: a match that succeeds when using table references directly can behave differently once the value is materialised and compared by the database. Using the column only as a copied value — for example a from_column mask that carries it across unchanged — is unaffected. The warning is shown once for each floating-point column.

Example

Materialising floating-point column `balance` of table reference "customers": floating-point values do not compare exactly, so joining or matching on this column may be unreliable.

To Disable

This warning cannot be disabled. A floating-point value is a poor join or match key on any database, materialised or not. If you match on this data, key the join on an exact-typed column — an integer id or text — rather than the float. If you only copy the column's value, no action is needed.