Performance Optimisation
In this document, you'll discover guidelines focused on performance optimisation. It provides guidance on utilising multiple workers, allowing for parallel execution and, consequently, enhanced performance.
- Masking using parallelism and workers
- Masking a table with multiple workers
- Performing tasks in parallel
- Worker and parallel task count selection
- If performance does not increase
- Masking large text data
- Snowflake recommendations
Masking using parallelism and workers
When planning how to parallelize tasks, always bear in mind the total number of workers that could be running concurrently. DataMasque runs every worker as a thread inside the agent process that runs the task. Each run that starts at the same time uses its own agent process. The total number of concurrent workers is equal to the number of parallel tasks times the number of workers per task.
For example:
1 task in parallel * 2 workers = 2 concurrent workers
or
2 tasks in parallel * 2 workers = 4 concurrent workers
Worker budget
A database masking run never uses more concurrent workers than its worker budget. DataMasque sizes the budget at the start of each run from the CPUs and memory available to the agent, as the lower of two limits:
- CPU: twice the number of CPUs available to the agent.
- Memory: the memory available to the agent minus a base, divided by the memory one worker needs:
128 MiB plus 8 KiB for each row of the largest batch the run reads:
the
batch_sizerun option, or a task's ownbatch_sizewhen that is larger. The base is 5 GiB when the agent uses the machine's memory, and 1.5 GiB when the agent's container has a memory limit of its own.
The budget is never less than 1. The run log states the budget and how it was calculated, for example:
Worker budget: 4 workers at once, the lower of the CPU limit (2 x 2 CPUs = 4) and the memory limit ((4.0 GiB - 1.5 GiB base) / 519 MiB per worker for batches of 50,000 rows = 4).
The budget is shared by every task in the run, including the tasks in nested parallel tasks:
- When the tasks that can run at the same time ask for more workers than the budget, each task waits until enough workers are free, and tasks start in the order they are ready. The run log says when a ruleset can ask for more workers than the budget.
- A task whose
workersis more than the whole budget runs with the budget's number of workers, and the run log says so, naming the task. Its masked values are the same as with theworkersit asks for, so a ruleset masks the same values on every agent, whatever its budget. - The unmasked tables of a copy load up to 10 at a time, or as many as the budget allows when it is lower.
The budget keeps a run within what the agent's CPUs and memory can hold,
however the ruleset's tasks are nested. It is not a setting:
to use fewer workers, lower workers or the number of tasks in parallel tasks;
to allow more, give the agent more CPUs or memory.
- Each worker holds its own database connection (two in a copy: one to the source and one to the destination),
so a run holds at most about as many source connections as its budget, plus one for the run itself.
If the database allows fewer connections than that, lower
workersor the number of parallel tasks. - Each run has its own budget. Runs that start at the same time on one agent do not share it.
- A
mask_tabletask starts on one worker while it reads its table. It then waits until all the workers it will start are free, even when some are free already, and a task that is ready later waits behind it. It starts no more than itsworkers, and no more than one per batch, so a small table never holds workers that other tasks could use. - The CPU limit is rounded down: 1.2 CPUs allow 2 workers. If the agent cannot read its memory, the CPU limit alone sets the budget.
- The memory limit allows about 8 KiB for each row of a batch. A row with large text or LOB values
takes far more, so give such tables a low
workersand keep them out of wideparalleltasks. The budget only limits how many workers run at once; it never starts more than the ruleset asks for.
File masking runs have no worker budget: they run one task at a time.
The guidance below is for choosing workers, parallel tasks and the size of the machine.
A ruleset sized this way runs without waiting for its budget.
The total number of concurrent workers should not exceed twice the number of CPUs available to the DataMasque instance. For example, if your virtual machine has four CPUs, the total number of concurrent workers should not exceed eight. Each worker uses up to about one CPU, so allow 1 vCPU per concurrent worker for full speed. Workers beyond one per CPU share the CPUs and add little throughput. These limits apply per worker, not per task: a task with four workers counts as four.
For Linux installation on a EC2 or similar VM, the workers of one run share the memory of one agent process. Runs that start at the same time do not share that memory. Add the allowance for each concurrent run. Each concurrent worker holds its own database connection and up to two batches of rows in memory: it masks one batch while it writes the batch before it. The memory that a worker adds therefore depends on the batch size and on the width of the masked rows. Allow, for each concurrent worker:
128 MiB + batch size × 8 KiB
in addition to a base of 5 GiB of RAM for the operating system and DataMasque.
For example, a worker with a batch size of 50,000 needs about 520 MiB, and one with a batch size of 10,000 needs about 210 MiB.
If the agent container has its own memory limit, the base within that limit is 1.5 GiB instead of 5 GiB.
These figures come from benchmarks on a 6 vCPU, 16 GiB virtual machine, not on production hardware.
Rows with wide text columns need more memory per row than this allowance.
Measure the actual memory use of your own workload with
diagnostic logging, then adjust the worker count.
A mask_table task that masks a column whose masks begin with imitate_unique or email
masks that column faster, and can also use more memory per worker,
only when no more workers mask at once than there are CPUs available to the DataMasque instance.
A task in a parallel block counts together with the workers of the tasks that the block can run at the same time as it.
The count never exceeds the run's worker budget, because no more workers than that mask at once.
Each worker then prepares the masked values of its whole batch on one thread per CPU, at most 8.
In internal benchmarks with a batch size of 50,000, this raised peak memory up to about 2.8 times,
so allow such a task more than the allowance above, and with a larger batch size, measure before you add workers.
When more workers than CPUs mask at once, these columns get neither the speed-up nor the extra memory use.
Because the count is held to the budget, a task whose workers is above the CPUs still gets both
when the budget is at or below the CPU count, for example when the memory limit sets it.
To see which columns were prepared this way, set the Display Log Level dropdown of the run log to Debug:
for each such column, each task logs one line that begins The warm pass ran for the
and names the mask, the column and the number of threads.
Thus, for example, if your machine has 16 GiB of RAM and every task uses a batch size of 50,000:
- 16 GiB total memory
- Deduct 5 GiB for the OS and DataMasque
- 11 GiB remaining, divided by about 520 MiB per worker, yields a maximum of 21 workers.
DataMasque logs a sizing message at the start of a run whose concurrent workers exceed the CPU limit above.
For database runs, it also logs a sizing message when they exceed the memory available to it.
These messages are logged at the Info level.
Discovery runs started from the Ruleset Generator, File Ruleset Generator or Unstructured Discovery pages, or their API endpoints,
do not run this check, because DataMasque writes their ruleset.
A ruleset you write, including one with a run_file_data_discovery or run_file_unstructured_discovery task, is checked.
A database run counts no more workers than its worker budget,
so it warns only when even the budget's minimum of 1 worker exceeds a limit.
To turn these messages off, set disable_warning_on_worker_sizing: true
at the top level of the ruleset.
The run log then notes that the check is off.
This changes no worker count, license limit or masked output.
For example, this file run asks for 32 workers, which logs the CPU sizing message on an agent with fewer than 16 CPUs
unless the check is off:
version: "1.0"
disable_warning_on_worker_sizing: true
tasks:
- type: mask_tabular_file
workers: 32
include:
- glob: "*.csv"
rules:
- column: last_name
masks:
- type: from_fixed
value: 'redacted last name'
Masking a table with multiple workers
To improve masking performance on a single table, you can enable parallelism which allows multiple workers to work
together simultaneously to mask a single table. This can be achieved simply by specifying a number of
workers greater than 1 for a task.
It is also recommended to increase the batch size in addition to increasing the number of workers to achieve optimal performance. Increasing the batch size increases the number of rows that are fetched, masked, and updated in a single operation. This will reduce the database operation overhead at the cost of increasing the memory usage of DataMasque. More details on batch size can be found under the Database Run Options guide.
Note: DataMasque ships engine-specific batch-size defaults based on internal benchmarking (see Database Run Options). The recommendations on this page describe values that may improve throughput for users who have profiled their own workload and have memory headroom available; raising the batch size beyond the default trades memory for speed.
When using multiple workers (workers > 1),
each worker operates on a separate batch of rows and these workers will run simultaneously.
This can result in reduced masking run time,
and hence improved performance as more rows of the table are masked at once.
Note: Increasing the number of workers will increase the amount of memory used (as well as CPU consumption). It is recommended to monitor resource usage when using parallelism.
In the ruleset specification below workers: 4 is specified,
therefore four workers will be used to mask the users table simultaneously.
version: "1.0"
tasks:
- type: mask_table
table: users
workers: 4
key: id
rules:
- column: last_name
masks:
- type: from_fixed
value: 'redacted last name'
The following diagram describes how multiple workers work in the example ruleset above.

Notes:
- Number of rows in each buffer is set by the batch size parameter/run option.
- As each worker finishes masking a batch of rows, it will move on to the next unmasked batch of rows.
Performing tasks in parallel
When using the parallel task type, DataMasque performs masking using multiple workers which allows masking
to run in parallel across multiple tables at once. Parallel tasks can reduce the time needed to mask a
database when compared to performing masking on individual tables sequentially.
Below is an example ruleset of how mask_table table tasks can be set up in to run in parallel.
Three tables are masked simultaneously in each parallel task block.
Once the first three tables are masked,
the next parallel task block is executed,
until finally all three parallel task blocks are complete and all tables in the ruleset are masked.
version: "1.0"
tasks:
- type: parallel
tasks:
- type: mask_table
table: table_1
...
- type: mask_table
table: table_2
...
- type: mask_table
table: table_3
...
- type: parallel
tasks:
- type: mask_table
table: table_4
...
- type: mask_table
table: table_5
...
- type: mask_table
table: table_6
...
- type: parallel
tasks:
- type: mask_table
table: table_7
...
- type: mask_table
table: table_8
...
- type: mask_table
table: table_9
...
Note:
mask_unique_keytasks are not allowed to be run in parallel.
The following diagram describes how parallel execution works in the example ruleset shown above.

Worker and parallel task count selection
As a general guideline, DataMasque should run at most two concurrent workers per CPU (or vCPU),
and one concurrent worker per vCPU for full speed.
Count workers, not tasks: the limit is the total of the workers of all tasks that run at the same time.
Each concurrent worker also needs memory; see Masking using parallelism and workers.
For example, a virtual machine with two vCPUs could run at most:
- A single
mask_tabletask with four workers; or, - Two
mask_tabletasks in parallel, each with two workers; or, - Four
mask_tabletasks in parallel, each with one worker, etc.
Increasing the worker or parallel count will also increase the amount of memory that DataMasque consumes. Therefore, it is crucial to monitor memory usage carefully when adjusting these settings.
We recommend experimenting with different worker counts to find the optimal configuration for your environment, while monitoring memory usage if increasing worker counts.
Memory consumption will increase if any of the following increase:
- Worker count.
- The number and types of columns being masked (database masking).
- Batch size (database masking).
- The size of individual files being masked (file masking).
Memory consumption does not increase based on the number of rows in a table, or the number of files being masked.
Once you have a stable number of workers for a particular table or file structure, the worker count should not need to be adjusted as the number of rows in the table grows or number of files to mask increases.
If performance does not increase
If you find that the performance of DataMasque does not improve as you add more workers, then you will need to identify the bottleneck. The tuning report at the end of each run's Debug log names the tables whose timings show what held them back.
First, make sure the DataMasque instance itself has enough free resources. Some masking configurations or certain tables/files may use more CPU and memory than others. Even if you are under the recommended number of workers per vCPU, check that the DataMasque instance has not exhausted its resources.
If the DataMasque instance has CPU, memory and IOPS capacity, it may indicate that the bottleneck lies elsewhere.
Consider the following possibilities:
- Database Server Limitations: Your database server might need additional resources, such as more CPU power, higher IOPS, or increased memory, to handle the increased load efficiently.
- Network Speed: For file masking tasks, particularly when dealing with cloud storage, performance is heavily dependent on your network speed. To a lesser extent, network speed can also affect database masking tasks, especially if there is high latency. Check for network saturation, and try to host your database(s) and DataMasque instance in the same region and availability zone if possible.
- Disk I/O: The speed of the mounted file share can also impact file masking tasks. If you are working with on-premise or mounted storage, consider upgrading to faster disks or optimizing the storage configuration.
In these cases, adjusting the resources allocated to your database server or network infrastructure may yield better results. We recommend continuing to monitor resource usage closely and experimenting with these settings to find the optimal configuration for your environment.
Masking large text data
To mask data, DataMasque loads data into memory in batches. The memory consumed depends on the size of each row and the number of rows processed at a time.
For smaller data types, such as integers, floats, dates, and small VARCHAR columns,
memory usage is typically minimal.
Since these data types result in small row sizes, usually only a few kilobytes,
they generally do not cause memory issues.
However, when masking larger data types, memory usage becomes more significant.
Large text fields, for instance, can use up to 4 times their size in memory when loaded into DataMasque.
For example, a single TEXT row of 100 MB would consume 400 MB of memory.
If you are masking 20 rows, each 100 MB in size, the memory requirement becomes: 20 rows x 100 MB x 4 = 8000 MB.
This would exceed the available memory in an 8 GB DataMasque instance.
To avoid memory exhaustion, you can reduce the batch_size to a lower value
based on the size of the rows being processed.
For example, masking 5 rows instead of 20 would significantly reduce memory usage.
Additionally, certain mask types, such as xml, json, and imitate, require extra memory during masking operations.
These types can consume up to 20 times the size of the data being masked.
When working with large text data, be cautious to avoid memory overload.
Snowflake Recommendations
Introduction
Snowflake differs from other RDBMS systems like Postgres in two significant ways.
- It is designed for very large volumes of data: writing more data in fewer requests is very efficient. For example, writing twenty times the amount of data might only take five times as long.
- It has a major limitation: due to table locking, only one batch of masked data can be written back to the target table at a time.
The above imply that using multiple workers to mask one table will only have a limited effect - the workers will sit idle, waiting for their turn to write back the masked data. Using larger batch sizes is more effective: it reduces the average write time per row, reduces the number of batches to mask, and means each worker spends more time masking rather than waiting.
Provided the machine running DataMasque has sufficient RAM, it is strongly recommended to use larger batch sizes. As an example, during internal testing, 8 workers did not perform faster than 4 workers with a batch size of 50,000 rows. However, large performance gains were achieved with 8 workers by increasing the batch size to 500,000 rows.
Recommendations
The following is a list of recommendations for masking Snowflake databases.
- Where possible, create the table(s) to be masked as a zero-copy clone of the original production data
(using
CREATE TABLE <table to be masked> CLONE <production table>). - We strongly recommend creating a warehouse solely for use by DataMasque. This not only avoids contention, but also allows you to clearly see the compute costs incurred by DataMasque.
- A warehouse size of Small or Medium is more than adequate for masking. Unless you are masking a lot of tables in parallel, you are unlikely to get faster performance with a larger warehouse, as the bottleneck is not the warehouse, but the machine running DataMasque.
- Use large batch sizes. A good value is on the order of 250,000 to 1,000,000 rows, depending on the number of columns, their data types, and available RAM.
- Mask multiple tables in parallel to avoid table lock contention. See Parallel tasks.
- Use between 4 and 12 workers per table. Fewer than 4 workers is not efficient, while too many workers results in lock contention.
Note: The DataMasque UI defaults the batch size to 250,000 when a Snowflake database is selected as the target for masking.
Note: Use of very large batch sizes with multiple workers and/or parallel tasks can exhaust the RAM on the machine running DataMasque and render it unresponsive. Select a batch size and number of workers that maximises masking speed while retaining at least 1GiB of free memory.
Note: With larger batch sizes, the workers are mostly CPU-bound performing the actual masking, rather than waiting for Snowflake or performing I/O. This maximises masking efficiency, but means CPU usage will be much higher than for other databases. The worker budget is a ceiling, not a target: it never lets a run hold more than twice the number of vCPUs, and at large batch sizes its memory limit is usually lower still. For Snowflake, size the ruleset below that ceiling: aim for at most 1.5x the number of vCPUs across all parallel tasks. For example, on a machine with 16 vCPUs the budget allows up to 32 workers, but a Snowflake ruleset should ask for no more than 24 at once.