Parallel and serial tasks
A parallel or serial task nests other tasks inside it. Use parallel for performance and
serial for dependency management.
Parallel tasks
Using the parallel task type, you can specify a block of tasks to be executed
in parallel, spread across as many workers as are available.
How many tasks run at once is limited by the run's
worker budget, which DataMasque sizes from the CPUs and memory
available to the agent. Each running task uses as many workers from the budget as its workers
(one for a task without workers), and a task waits until enough workers are free.
Nested parallel tasks share the same budget.
It is recommended to begin testing parallelisation with at most 4 tasks in parallel, then
increase parallelisation if the database has more capacity.
Parallel tasks can be nested inside other serial/parallel tasks.
Not every license type includes parallel tasks.
A run is refused if its ruleset contains a parallel task and the license type does not include parallel tasks.
See Parallel tasks availability.
Parallel task parameters
tasks(required): A set of tasks to perform in parallel.
Parallel task example
tasks:
- type: parallel
tasks:
- type: mask_table
table: employees
key: id
rules:
- column: 'name'
masks:
- type: from_fixed
value: 'REDACTED'
- type: mask_table
table: customers
key: id
rules:
- column: 'address'
masks:
- type: from_fixed
value: 'REDACTED'
...
Note: DataMasque disallows masking the same table in multiple tasks (including
mask_table,run_sqltasks) in parallel, as this could result in data being incorrectly masked. The ruleset editor warns when aparallelblock masks the same table name more than once. It compares table names as written, sousersandpublic.usersare not reported as the same table.Note: Currently, the
paralleltask type is not supported for file masking. To reduce masking time, setworkerson themask_fileormask_tabular_filetask instead, so that each worker masks a different file in parallel.
Serial tasks
Although tasks are performed serially in the order they are listed in the ruleset by default, you can specify a block of tasks to be performed in serial within a parallel block. This is useful when a subset of parallelisable tasks have dependencies that mean they must be executed in sequence.
Serial tasks can be nested inside other serial/parallel tasks.
Serial task parameters
tasks(required): A set of tasks to perform in series.
Serial task example
tasks:
- type: parallel
tasks:
- type: serial
tasks:
- type: run_sql
sql_file: pre_employees_script.sql
- type: mask_table
table: 'employees'
key: id
rules:
- column: 'name'
masks:
- type: from_fixed
value: 'REDACTED'
- type: mask_table
table: 'customers'
key: id
rules:
- column: 'address'
masks:
- type: from_fixed
value: 'REDACTED'
...
Note: Currently, the
serialtask type is not supported for file masking.