Ruleset YAML Specification for `unstructured_text`

matchers (required): A dictionary defining which matchers to use for entity detection. Each Matcher Type is a key containing a list of matcher configurations.
- context_sources (optional): Detect entities by matching values from other columns.
  - label (required): The entity label (must be UPPER_CASE).
  - column (required): The source column containing entity values.
  - case_sensitive (optional): Whether to match case-sensitively. Defaults to false.
  - context_filters (optional): List of Context Filters to refine matches.
  - overlap_priority (optional): Whole number from -999 to 999. The higher number wins when overlapping entities start at the same position and are the same length. Defaults to 0.
- seed_files (optional): Detect entities by matching values from seed files.
  - label (required): The entity label (must be UPPER_CASE).
  - seed_file (required): The name of a user-provided seed file (see Files guide).
  - seed_column (required): The name of the seed file column that will provide entity values.
  - case_sensitive (optional): Whether to match case-sensitively. Defaults to false.
  - context_filters (optional)
  - overlap_priority (optional): Whole number from -999 to 999. The higher number wins when overlapping entities start at the same position and are the same length. Defaults to 0.
- ai_detect (optional): AI-powered entity detection (requires a DataMasque AI Engine).
  - label (required): The entity label (must be UPPER_CASE).
  - use_preset (optional): Use a built-in preset (e.g., "FULL_NAME").
  - guidelines (optional, deprecated): Not sent to the AI Engine. A run of a ruleset that sets it finishes with warnings. Give the label a self-explanatory name instead.
  - context_filters (optional)
  - overlap_priority (optional): Whole number from -999 to 999. The higher number wins when overlapping entities start at the same position and are the same length. Defaults to 0.
- regex (optional): Detects entities matching regular expressions.
  - label (required): The entity label (must be UPPER_CASE).
  - pattern (required): The regular expression used to detect entities.
  - context_filters (optional)
  - overlap_priority (optional): Whole number from -999 to 999. The higher number wins when overlapping entities start at the same position and are the same length. Defaults to 0.
- checksum (optional): Detects entities matching checksum algorithms.
  - label (required): The entity label (must be UPPER_CASE).
  - type (required): Checksum type. See set_checksum for supported types.
  - context_filters (optional)
  - overlap_priority (optional): Whole number from -999 to 999. The higher number wins when overlapping entities start at the same position and are the same length. Defaults to 0.
masks (required): A list of label-to-mask mappings defining how to mask each detected entity.
- label (required): The entity label (must match a label from matchers).
- masks (required): A list of masks to apply to entities with this label.
- date_format_preservation (optional): Overrides date format preservation settings for this label. Defaults to enabled for compatible date masks.
overlap_handling (optional): Controls how overlapping matches are resolved into one span.
- matcher_priorities (optional): The five matcher types listed exactly once each, highest priority first, replacing the default order.
hash_sources (optional): List of Hash Source configurations for deterministic masking.
- self: entity (optional): Hash on the detected entity text itself.
  - case_transform (optional): upper/lower - Allows for case transforms on the entity text before hashing, ensuring consistent hashed values irrespective of case.
- label (optional): Hash on a detected entity.
  - match (optional): Zero-based index of which detected entity to hash on (defaults to 0).
  - match_until (optional): Hash on a range of entities from [match:match_until].
  - case_transform (optional): upper/lower - Allows for case transforms on the entity text before hashing, ensuring consistent hashed values irrespective of case.
  - state (optional): original (default) hashes on the referenced label's pre-masked text; masked hashes on its post-masked text. See Hashing on masked results.
fallback_masks (optional): List of masks to apply to the entire text if AI Engine processing fails.
image_handling (optional): Controls how images are handled, both inside Word documents, PowerPoint documents and PDF documents and as standalone PNG and JPEG images, whether as files or in binary columns. Has no effect on text columns.
- "redact" (default): Replaces all images with a black rectangle.
- "retain": Preserves original images, and strips the metadata of standalone PNG and JPEG images, whether as files or in binary columns.
- "mask": Uses optical character recognition (OCR) to read the text in PNG and JPEG images, whether as files or in binary columns, and masks it with the configured matchers and masks. An image inside a Word, PowerPoint or PDF document falls back to "redact", with a warning.
Note: Image properties (alt text, title, description) are always removed regardless of this setting (Word documents only). Non-image drawings (charts, SmartArt, shapes, etc.) in Word documents are always redacted.
drawing_handling (optional): Controls how vector drawings (lines, shapes, paths) are handled in PDF documents. Has no effect on Word documents or text columns.
- "redact" (default): Removes all vector drawings from the output.
- "retain": Preserves original vector drawings.

Important: Every label defined in matchers must have a corresponding entry in masks. Use do_nothing if you want to detect without masking.

Note: When multiple matchers detect overlapping entities, DataMasque merges them by extending boundaries to cover all overlapping text. See Overlap Handling for examples.

Basic Example

The following ruleset matches on customer names in the description column. After matching, each name is deterministically masked with a first name from the DataMasque_firstNames_mixed.csv seed file.

version: "1.0"
tasks:
  - type: mask_table
    table: support_tickets
    key: ticket_id
    rules:
      - column: description
        masks:
          - type: unstructured_text
            hash_sources:
              - self: entity

            matchers:
              context_sources:
                - label: "CUSTOMER_NAME"
                  column: customer_name

            masks:
              - label: "CUSTOMER_NAME"
                masks:
                  - type: from_file
                    seed_file: DataMasque_firstNames_mixed.csv
                    seed_column: firstname-mixed

description (before)	description (after)
Caleb Jones reported issue with login	Marnie Taren reported issue with login
Follow-up for Sally Reefton	Follow-up for Korrie Marwood
Caleb Jones escalated to management	Marnie Taren escalated to management

In the above example, hash_sources deterministically masks repeated names found using the context_sources matcher.

For more on other matchers see Matchers and Labels.
For more detailed examples, see Examples.

Documents and Images in Database Columns

Masking PDF, Word, PowerPoint, PNG or JPEG files in a mask_table task is memory intensive. For these columns:

Set batch_size to 1 on the mask_table task.

Ruleset YAML Specification for unstructured_text

Basic Example

Documents and Images in Database Columns

Ruleset YAML Specification for `unstructured_text`