DataMasque Warnings
- Multiple Rules Warning
- Value Truncation Warning
- Imitate Numeric Warning
- Worker Sizing Warning
- Row Limit Reached Warning
- Value Limit Reached Warning
- Files Not Masked Warning
- Glacier Files Warning
- Files Could Not Be Scanned Warning
- No Files Scanned Warning
- AI Engine Batch Warning
- Use Of Conditionals Warning
- Missing Unique Index Warning
- Skipped Character Classes Warning
- Snowflake Time Travel Warning/Error
- Materialised Float Column Warning
- TLS Fallback Warning
- Memory Limit Warning
This document describes warnings that DataMasque may generate in the course of normal masking runs, and, how to disable them (if they can be disabled).
Multiple Rules Warning
This warning is displayed if a table or column is masked by more than one rule in a single ruleset, which may indicate that a rule was mistakenly included more than once.
Example
The following tables are configured to be masked multiple times by `mask_table` tasks in the ruleset: …
The following columns are configured to be masked multiple times in the ruleset: …
To Disable
Add disable_warning_on_duplicate_rules: true at the top level of the ruleset.
Value Truncation Warning
This warning is displayed when auto_truncate_value is enabled and a masked value is wider
than its target string column, so DataMasque truncates the value to fit. Only string-type
columns are truncated. The truncated value may no
longer be valid, for example a shortened email address.
Example
One or more masked values for column `email` exceeded its maximum width of 25 and were truncated. The truncated values may no longer be valid.
To Disable
Add disable_warning_on_auto_truncate_value: true at the top level of the ruleset,
or on the specific rule where you want to suppress the warning.
Imitate Numeric Warning
This warning is displayed, once per run, when the imitate mask is applied to a decimal (NUMERIC/DECIMAL) or floating-point value.
imitate replaces each digit at random,
so the masked values may fall outside the range of the original values,
and floating-point values may lose precision.
Integer values do not raise this warning.
Example
The `imitate` mask was applied to a decimal or floating-point value. Each digit is replaced at random, so masked values may fall outside the original range and floating-point values may lose precision. Set `disable_warning_on_imitate_numeric: true` at the top level of the ruleset to disable this warning.
To Disable
Add disable_warning_on_imitate_numeric: true at the top level of the ruleset.
Worker Sizing Warning
This message is displayed at the start of a run whose ruleset asks for more concurrent workers
than the CPUs or memory available to the agent can support.
It is logged at the Info level.
The CPU warning applies to database and file runs.
Discovery runs started from the Ruleset Generator, File Ruleset Generator or Unstructured Discovery pages, or their API endpoints,
do not run this check, because DataMasque writes their ruleset.
A ruleset you write, including one with a run_file_data_discovery or run_file_unstructured_discovery task, is checked.
The memory warning applies to database runs only.
A database run counts no more workers than its worker budget,
so it warns only when even the budget's minimum of 1 worker exceeds a limit.
Example
CPU, for a file run:
This run asks for up to 24 concurrent workers, more than twice the 4 CPUs available to the agent. Throughput stops rising at about 1 worker per CPU, so lower `workers` or the number of parallel tasks, or give the agent more CPUs. Set `disable_warning_on_worker_sizing: true` at the top level of the ruleset to disable this warning.
Memory, for a database run:
This run asks for up to 1 concurrent worker, which may need about 5.5 GiB of memory: a 5 GiB base plus 128 MiB and 8 KiB per `batch_size` row for each concurrent worker. The agent has 5.0 GiB (host). Lower `workers` or `batch_size`, or give the agent more memory, so the run is not stopped for lack of memory. Set `disable_warning_on_worker_sizing: true` at the top level of the ruleset to disable this warning.
To Disable
Add disable_warning_on_worker_sizing: true at the top level of the ruleset.
The run log then has one info line: "The worker sizing check is off: the ruleset sets disable_warning_on_worker_sizing: true."
This changes no worker count, license limit or masked output.
Row Limit Reached Warning
This warning is displayed when a license imposes a row limit and the data was larger than it.
In a database run the limit applies per table, to the table the ruleset names, and the rows
past it are left in place unmasked; rows in child tables reached through a cascading mask are
updated in full. In a file run it applies per file, to each tabular file a mask_tabular_file
task masks. What happens to the rows past a file's limit depends on the license: a Sandbox
license writes them to the output file unmasked, while a Free Trial license leaves them out of
the output file. The message names which of the two happened.
It cannot be disabled: partly masked data still holds real values, so the run completes with
warnings to make that unmissable. A max_rows you set yourself is reported the same way but
without the warning, since bounding the run was your own choice.
A dry run reports the limit it would reach without this warning, because nothing is written and no partly masked data exists.
In a copy, the rows past the limit are not copied to the destination,
including from the tables copy_scope copies without masking.
In a copy without a filter, a foreign key that references a table the limit cuts short is created
ENABLE NOVALIDATE on Oracle, and left out on Db2 for i, because the copied rows can reference
rows that were not copied. A filtered copy does the same for a table that the filter does not reach.
In a filtered copy, the read of each table that the filter reaches
stops once it has kept the limit's number of rows, counting the parent rows it pulls in for kept rows.
The foreign keys that reference such a table stay validated, because the copy changes the child
rows instead: a nullable foreign key to a row that was not copied is set to NULL, and a row whose
NOT NULL foreign key references it is removed. The run log says how many pulled-in rows were not copied.
Some columns cannot be masked part-way without producing duplicate values or stranding rows in
tables that reference them. Those are exempt from the limit and masked in every row, so a table
named here may still have columns masked throughout; the run log lists them on a separate
Row limit exempt line. See Licensing for exactly which columns qualify.
Where every column masked on a table is exempt, nothing is cut and this warning is not shown: the run reports that the limit did not apply and finishes without warnings.
Example
Row limit reached: masked the first 10,000 rows of `public.customers`. The remaining rows are unmasked.
For a file run:
Row limit reached: masked the first 10,000 rows of file "bucket/customers.csv", which contains 25,000. The remaining 15,000 rows are written through unmasked.
Value Limit Reached Warning
This warning is displayed when a Sandbox license's per-file value limit of 10,000 is reached and the ruleset matched more than that. Only the values a ruleset masks are counted: the rest of the document is neither counted nor changed.
A Free Trial license sets no value limit. It bounds a document by its size instead and skips it whole, so a trial file is never masked in part this way. See Licensing for the limits each license carries.
It applies to the JSON and XML files a mask_file task masks, including the NDJSON and Avro
files that hold many records. These are masked as documents rather than row by row, so they are
bounded by the values they mask instead of by a row count. The values past the limit are written
to the output file unmasked, so the file comes back whole.
The limit applies per file. A file holding many records gives them one allowance between them rather than one each. PDF, DOCX, PPTX and other generic files are not covered: they are masked as a single item, with no values to count, and are bounded by the file limit instead.
It cannot be disabled, for the same reason the row limit's warning cannot: a partly masked file still holds real values.
A dry run reports the limit it would reach without this warning, because nothing is written and no partly masked file exists.
Example
The count of remaining values is the number the ruleset would have masked, not the number the file holds.
Value limit reached: masked the first 10,000 values of file "bucket/patients.json". The remaining 4,312 values are written through unmasked.
Files Not Masked Warning
At the end of a file masking run, DataMasque lists the files it did not mask, with the number of files for each reason:
- files past the license's limit on the number of files or file size
- files archived with S3 Glacier
- tabular files that are empty or have no rows
This information is logged as a warning when some files were unmasked due to license limits, to draw attention to the fact some files were left unmasked. This warning cannot be disabled.
Files archived with S3 Glacier also have a Glacier Files Warning of their own, which can be disabled.
Example
7 files were not masked:
- 1 file past the Trial license limit of 5 files: "customers_6.json"
- 3 files larger than the Trial license file size limit of 1MB
- 2 files archived in S3 Glacier storage
- 1 file empty or with no rows
Glacier Files Warning
On Amazon S3, when masking files or performing a file discovery run, this warning will be shown if a file has been archived with S3 Glacier.
For file masking, this means the file will not be copied to the destination connection. For discovery, it means that DataMasque will not be able to determine if the file contains sensitive information.
Example
Skipping masking file "archived_with_glacier.csv":
files archived with Glacier can not be masked.
For a run_file_unstructured_discovery task:
Could not read "archived_with_glacier.pdf": it is archived with Glacier.
To Disable
Add disable_warning_on_glacier_storage: true to the specific mask_file, mask_tabular_file,
run_file_data_discovery or run_file_unstructured_discovery task where you want to disable the warning.
For a run_file_data_discovery or run_file_unstructured_discovery task,
this also excludes Glacier-archived files from the
Files Could Not Be Scanned Warning.
Files Could Not Be Scanned Warning
At the end of a file data discovery run,
this warning is displayed if any files could not be scanned:
the file could not be read (including files archived with S3 Glacier),
a Parquet file contained a column with an unsupported data type,
or an unexpected error occurred during discovery.
For a run_file_unstructured_discovery task,
it is also displayed when a file could not be read from the connection or decoded,
or a matcher or the AI Engine failed on it.
The discovery results may therefore be incomplete.
The count is an aggregate;
the file discovery summary above it breaks the files down by reason,
and the run report lists every affected file.
Files that were not scanned for any of the following reasons do not trigger this warning, because nothing went wrong when DataMasque considered them. The reason names are the ones used in the run report and in the file discovery summary:
File type unsupported by data discoveryFile type unsupported by unstructured discoveryFile type could not be determinedFile was emptyDid not match any include filterMatched a skip filterFile path too longExceeds the configured file size limitOver the per-directory file limitOver the run's file limit
Example
3 files could not be scanned. See the messages above for details on each file, or download the run report from the Run Logs page for the full list.
For a run_file_unstructured_discovery task:
3 files could not be read. See the messages above for details on each file, or download the run report from the Run Logs page for the full list.
To Disable
This warning cannot be disabled in general.
The exception is files archived with S3 Glacier:
add disable_warning_on_glacier_storage: true to the run_file_data_discovery
or run_file_unstructured_discovery task
to exclude those files from the warning and its count.
No Files Scanned Warning
At the end of a file data discovery run,
this warning is displayed if the source contained files but none of them were scanned:
for example, a source holding only unsupported file types,
or include/skip filters that matched nothing scannable.
The file discovery summary above the warning shows,
per file extension, why each file was not scanned.
Example
No files were scanned. All 243 files in the source were skipped for the reasons shown in the summary above.
For a run_file_unstructured_discovery task:
No files were read. See the summary above for why each of the 243 files was not read.
To Disable
This warning cannot be disabled.
To resolve it, adjust the task's include/skip filters,
or point the connection at a source containing file types the task reads:
CSV, JSON, NDJSON or Parquet files for a run_file_data_discovery task,
and PDF, DOCX and PPTX files, or files with an extension listed in plain_text_extensions,
for a run_file_unstructured_discovery task.
For a run_file_unstructured_discovery task, also check max_bytes_per_file.
AI Engine Batch Warning
This warning is displayed when a run_file_unstructured_discovery task
cannot send files to the AI Engine in batches.
The task then sends one file at a time, so the run takes longer.
Example
Sending files to the AI Engine in batches failed, so each file is sent on its own.
To Disable
This warning cannot be disabled. To check that the AI Engine is running, see Verification.
Use Of Conditionals Warning
Displayed when using conditional masking features that may leave some data unmasked.
For example, skip, where and if/else.
Example
Use of the conditional masking features `where`, `skip`, or `if`/`else`,
may mean your masking rules are not applied to some database rows or values.
To Disable
Add disable_warning_on_use_of_conditionals: true to the specific mask_table task where you want to disable the warning.
Missing Unique Index Warning
For mask_unique_key tasks, the key column(s) being masked are expected to have a unique index.
If a unique index does not exist, then DataMasque will temporarily create one for the duration of the masking run,
and will issue a warning that one was created.
Example
Unique requirement for specified key could not be validated,
as no adequate unique index is found for key column: …
To Disable
Add disable_warning_on_missing_unique_index: true to the mask_unique_key task to disable the warning for that task.
DataMasque will silently add the unique index during masking, without showing a warning.
Alternatively, add a unique key/index to the keys(s) before beginning the masking task.
Skipped Character Classes Warning
When using a from_unique_imitate mask,
if either of the skip_letters or skip_digits parameters are specified true,
a warning is shown to let you know that the configuration will leave some data unmasked.
Example
No letters will be masked.
If the column to be masked only contains letters, nothing will be masked.
No digits will be masked.
If the column to be masked only contains digits, nothing will be masked.
To Disable
Add disable_warning_on_skipped_characters: true as a parameter to the from_unique_imitate mask.
Snowflake Time Travel Warning/Error
When masking data on Snowflake databases, if a table has Snowflake Time Travel enabled, the unmasked data can be recovered by restoring a snapshot of the table from a previous point in time. As such, DataMasque disallows masking until Time Travel is turned off for that table.
Refer to the Snowflake documentation for more information about Time Travel, including how to disable it at the table, schema, database or account level.
Example Error
Snowflake Time Travel is enabled for table `myschema.mytable` (configured explicitly on the table).
When Snowflake Time Travel is enabled, the unmasked data can be recovered. As such, masking is not secure.
Please disable Snowflake Time Travel on the table `myschema.mytable` before masking.
Refer to https://docs.snowflake.com/en/user-guide/data-time-travel for more information.
To Disable
If you want to mask the table without turning off Time Travel,
specify allow_masking_tables_with_time_travel: true at the top level of the ruleset.
The above message will appear in the run logs as a warning instead of an error, and masking will proceed as normal.
Materialised Float Column Warning
When materialising a table reference
(copying the table reference into a temporary table),
each column is recreated with a matching type.
A floating-point column is materialised like any other,
but the target database may store and compare it slightly differently than DataMasque does.
The risk is specific to using the column as a join or match key:
a match that succeeds when using table references directly
can behave differently once the value is materialised and compared by the database.
Using the column only as a copied value — for example a from_column
mask that carries it across unchanged — is unaffected.
The warning is shown once for each floating-point column.
Example
Materialising floating-point column `balance` of table reference "customers": floating-point values do not compare exactly, so joining or matching on this column may be unreliable.
To Disable
This warning cannot be disabled. A floating-point value is a poor join or match key on any database, materialised or not. If you match on this data, key the join on an exact-typed column — an integer id or text — rather than the float. If you only copy the column's value, no action is needed.
TLS Fallback Warning
This warning is written to the run log when a PostgreSQL, MySQL, MariaDB or Microsoft SQL Server connection
that uses the default TLS (or Encryption) choice ended up connected without TLS,
because DataMasque tried TLS and could not make a TLS connection
(the server does not offer TLS, or does not support encryption for SQL Server;
for PostgreSQL, also when the TLS handshake or the login over TLS failed).
The choice was left at Prefer (Optional for SQL Server), so DataMasque fell back.
A connection set to Require, Required or Off never produces this warning.
SQL Server Optional against a server that supports encryption does not warn either,
even though only the login is encrypted.
Amazon Redshift, Databricks Lakebase, Apache Cassandra, Db2 LUW and Informix connections never produce it.
The warning is written once per connection per run, and the run ends as finished_with_warnings.
Connection tests do not write it.
DataMasque checks for the fallback each time it opens a connection for a task.
A PostgreSQL, MySQL or MariaDB connection pool can open more connections later in a task;
if those connect without TLS while the first one used TLS, no warning is written.
Example
The PostgreSQL connection "prod_db" connected without TLS because a TLS connection to the server could not be made. To stop this warning, edit the connection and set TLS to Require (connect only over TLS) or Off (connect without TLS).
The SQL Server connection "prod_sql" connected without encryption because the server does not support encryption. To stop this warning, edit the connection and set Encryption to Required (connect only with encryption) or Off (connect without encryption).
For MySQL and MariaDB connections the text starts The MySQL connection or The MariaDB connection.
To Disable
Edit the connection and set TLS to Require or Off (Encryption to Required or Off for SQL Server).
Require is the safer choice, because the run then fails instead of connecting without TLS.
See the PostgreSQL,
MySQL,
MariaDB and
Microsoft SQL Server pages.
Memory Limit Warning
During every run, DataMasque checks how much of its memory limit is in use. The limit is the container's memory limit when one is set, and otherwise the host's total memory. When memory use first reaches 80% of the limit, a message is logged at the Info level, which does not change the run's status. When it first reaches 90%, a warning is logged, so a run that then completes has the Finished With Warnings status. If the run is killed because it ran out of memory, these messages show it was nearing the limit. Without a container memory limit, the figures include every other process on the host, so both messages are logged at the Info level and do not change the run's status. Runs that share a container share its memory limit, so each of them logs these messages.
Example
Agent memory use has reached 90% of its limit: 9.1 GiB of 10.0 GiB (cgroup v2 memory.max). The run may be killed if memory runs out.
Memory use on the host has reached 90% of the host's memory: 14.4 GiB of 16.0 GiB. The run may be killed if memory runs out.
To Disable
This warning cannot be disabled. To avoid it, give DataMasque more memory or reduce the memory the run needs.