Parallel and serial tasks

A parallel or serial task nests other tasks inside it. Use parallel for performance and serial for dependency management.

• Parallel tasks
  • Parallel task parameters
  • Parallel task example
• Serial tasks
  • Serial task parameters
  • Serial task example

Parallel tasks

Using the parallel task type, you can specify a block of tasks to be executed in parallel, spread across as many workers as are available.

Each parallel task distributes to a maximum of 10 sub-tasks. It is recommended to begin testing parallelisation with at most 4 tasks in parallel, then increase parallelisation if the database has more capacity.

Parallel tasks can be nested inside other serial/parallel tasks.

Parallel task parameters

• tasks (required): A set of tasks to perform in parallel.

Parallel task example

tasks:
  - type: parallel
    tasks:
      - type: mask_table
        table: employees
        key: id
        rules:
          - column: 'name'
            masks:
              - type: from_fixed
                value: 'REDACTED'
      - type: mask_table
        table: customers
        key: id
        rules:
          - column: 'address'
            masks:
              - type: from_fixed
                value: 'REDACTED'
 ...

Note: DataMasque disallows masking the same table in multiple tasks (including mask_table, run_sql tasks) in parallel, as this could result in data being incorrectly masked.

Note: Currently, the parallel task type is not supported for file masking. To reduce masking time, set workers on the mask_file or mask_tabular_file task instead, so that each worker masks a different file in parallel.

Serial tasks

Although tasks are performed serially in the order they are listed in the ruleset by default, you can specify a block of tasks to be performed in serial within a parallel block. This is useful when a subset of parallelisable tasks have dependencies that mean they must be executed in sequence.

Serial tasks can be nested inside other serial/parallel tasks.

Serial task parameters

• tasks (required): A set of tasks to perform in series.

Serial task example

tasks:
  - type: parallel
    tasks:
      - type: serial
        tasks:
          - type: run_sql
            sql_file: pre_employees_script.sql
          - type: mask_table
            table: 'employees'
            key: id
            rules:
              - column: 'name'
                masks:
                  - type: from_fixed
                    value: 'REDACTED'
      - type: mask_table
        table: 'customers'
        key: id
        rules:
          - column: 'address'
            masks:
              - type: from_fixed
                value: 'REDACTED'
 ...

Note: Currently, the serial task type is not supported for file masking.