DataMasque Portal

Parallel and serial tasks

A parallel or serial task nests other tasks inside it. Use parallel for performance and serial for dependency management.

Parallel tasks

Using the parallel task type, you can specify a block of tasks to be executed in parallel, spread across as many workers as are available.

How many tasks run at once is limited by the run's worker budget, which DataMasque sizes from the CPUs and memory available to the agent. Each running task uses as many workers from the budget as its workers (one for a task without workers), and a task waits until enough workers are free. Nested parallel tasks share the same budget. It is recommended to begin testing parallelisation with at most 4 tasks in parallel, then increase parallelisation if the database has more capacity.

Parallel tasks can be nested inside other serial/parallel tasks.

Not every license type includes parallel tasks. A run is refused if its ruleset contains a parallel task and the license type does not include parallel tasks. See Parallel tasks availability.

Parallel task parameters

  • tasks (required): A set of tasks to perform in parallel.

Parallel task example

tasks:
  - type: parallel
    tasks:
      - type: mask_table
        table: employees
        key: id
        rules:
          - column: 'name'
            masks:
              - type: from_fixed
                value: 'REDACTED'
      - type: mask_table
        table: customers
        key: id
        rules:
          - column: 'address'
            masks:
              - type: from_fixed
                value: 'REDACTED'
 ...

Note: DataMasque disallows masking the same table in multiple tasks (including mask_table, run_sql tasks) in parallel, as this could result in data being incorrectly masked. The ruleset editor warns when a parallel block masks the same table name more than once. It compares table names as written, so users and public.users are not reported as the same table.

Note: Currently, the parallel task type is not supported for file masking. To reduce masking time, set workers on the mask_file or mask_tabular_file task instead, so that each worker masks a different file in parallel.

Serial tasks

Although tasks are performed serially in the order they are listed in the ruleset by default, you can specify a block of tasks to be performed in serial within a parallel block. This is useful when a subset of parallelisable tasks have dependencies that mean they must be executed in sequence.

Serial tasks can be nested inside other serial/parallel tasks.

Serial task parameters

  • tasks (required): A set of tasks to perform in series.

Serial task example

tasks:
  - type: parallel
    tasks:
      - type: serial
        tasks:
          - type: run_sql
            sql_file: pre_employees_script.sql
          - type: mask_table
            table: 'employees'
            key: id
            rules:
              - column: 'name'
                masks:
                  - type: from_fixed
                    value: 'REDACTED'
      - type: mask_table
        table: 'customers'
        key: id
        rules:
          - column: 'address'
            masks:
              - type: from_fixed
                value: 'REDACTED'
 ...

Note: Currently, the serial task type is not supported for file masking.