Matchers and Labels

• Matcher Types
  • context_sources
  • seed_files
  • ai_detect
  • regex
  • checksum
• Overlap Handling
  • Same-Label Overlap
  • Cross-Label Overlap
  • Matcher Priorities
• Context Filters
  • match_whole_words_only
  • exclude_pattern
  • include_pattern
  • add_prefix
  • add_suffix
  • Combining filters
• Configuring Masks for Labels
  • Supported masks
• Fallback Masks

Matcher Types

context_sources

Note: context_sources can only be used with mask_table or mask_tabular_file tasks.

The context_sources matcher references values from other columns or fields.

first_name	clinical_notes
Sarah	Patient Sarah presented with acute migraine.
Bradley	Following the consultation, Bradley reported improved mobility.

Use context_sources if you have structured data alongside each text entry you want masked.

matchers:
  context_sources:
    - label: "PATIENT_FIRST_NAME"
      column: first_name

Case Sensitivity

By default, case_sensitive: false.

first_name	case_sensitive	clinical_notes
Thomas	false	THOMAS arrived
Thomas	true	THOMAS arrived

matchers:
  context_sources:
    - label: "PATIENT_FIRST_NAME"
      column: first_name
      case_sensitive: true

seed_files

The seed_files matcher references values from seed files loaded into DataMasque. Given a medications.csv seed file, you can detect the following:

clinical_notes
Prescribed Aspirin 100mg daily.
Bradley allergic to Morphine.

Use seed_files if you have a list of entities you want masked within each text entry.

matchers:
  seed_files:
    - label: "MEDICATION_NAME"
      seed_file: "medications.csv"
      seed_column: "medication_names"

Case Sensitivity

By default, case_sensitive: false.

case_sensitive	clinical_notes
false	THOMAS arrived
true	THOMAS arrived

matchers:
  seed_files:
    - label: "FIRST_NAME"
      seed_file: "DataMasque_firstNames_mixed.csv"
      seed_column: "firstname-mixed"
      case_sensitive: true

ai_detect

Note: ai_detect is only available if a DataMasque AI Engine is configured. DataMasque supports two AI Engines: the Local AI Engine and the Bedrock AI Engine. See AI Engine for differences.

The ai_detect matcher uses a language model to detect entities.

Text	PATIENT_NAME	DOCTOR_NAME
Jones: Patient Amir reported chest pain.	Amir	Jones
Emil&y reports knee pain to Dr. Jones.	Emil&y	Dr. Jones

Note: Emil&y is deliberately mis-spelt to showcase ai_detect's handling of typos.

Use ai_detect if you want to detect:

• novel or ambiguous entities requiring clear guidelines,
• entities in texts containing misspellings or OCR artefacts,
• entities not detectable by the other matchers such as regex

DataMasque includes built-in entity definitions for common entity types. The matcher below uses the FIRST_NAME preset:

matchers:
  ai_detect:
    - label: "FIRST_NAME"
      use_preset: "FIRST_NAME"

For entities not covered by presets, define custom detection rules with guidelines:

matchers:
  ai_detect:
    - label: "FRIEND_FIRST_NAME"
      guidelines:
        - First names of the patient's friends and family
        - Exclude patients themselves and medical staff

If neither use_preset nor guidelines is specified, the label itself is passed to the AI Engine. This is often sufficient for self-explanatory labels:

matchers:
  ai_detect:
    - label: "FIRST_NAME"
    - label: "LAST_NAME"
    - label: "STREET_ADDRESS"
    # ...

Note: use_preset and guidelines are mutually exclusive.

What the underlying model sees for detection:

• Label only: your label
• use_preset: the preset's label (and preset guidelines if present); your label is not shown to the model
• guidelines: your label and your guidelines

In every case, the AI Engine returns detected entities tagged with your label, which the masks and hash_sources blocks then match against.

If a simple guideline is insufficient for downstream masking, multiple guidelines may be added.

In the guideline below, we are excluding titles because they will be masked separately (with the TITLE preset). Likewise, we are more explicit about excluding friends and family because we want to reduce false matches.

matchers:
  ai_detect:
    - label: "PATIENT_FIRST_NAME"
      guidelines:
        - First names only of patients
        - Exclude titles, and names of family/friends
        - 'Valid examples: "Patient [Smith] reports pain"'
        - 'Invalid examples: "Surgery was performed on [Ms.Smith] Thompson" (Contains title)'

Note: ai_detect processes rows slower than the other matchers.

Use the max_rows run option to validate configuration before running on full datasets.

Writing effective guidelines

0. Do you need a guideline?

• Guidelines are not necessary if the label is self-explanatory. For example, PATIENT_FIRST_NAME is clear and specific.
• Reach for guidelines when the label is ambiguous, or when you need to constrain what the model should and should not match.

1. Start simple

• Language models have a lot of pre-built world knowledge.
• A small simple guideline is usually enough to match on 95%+ of entities.
• For the Local AI Engine, the label alone (no guidelines) is often sufficient because it already recognises the supported labels.

2. Work iteratively

• If you are able to access the data after masking, it helps to view the results both before and after. Tweak the guidelines after each attempt. Use max_rows to mask only a subset of your data.

• If you find that certain entities tend not to be matched or that other entities are unintentionally matched, add clarifying examples to your guidelines (see the PATIENT_FIRST_NAME example above).

3. Consider downstream masks

• Lastly, consider how your guidelines apply to downstream masks and the realism of your masked data.

• Be specific. In the PATIENT_FIRST_NAME example above for example, TITLE was excluded because we wanted to mask it separately.

regex

The regex matcher finds text matching regular expression patterns.

Pattern	Text
\b\d{5}\b	ZIP Code: 90210
\b[A-Z]{3}-\d{4}\b	Here: ABC-1234 is my reference
\b\d{3}-\d{3}-\d{4}\b	Call 555-867-5309 for details

Use regex if you want to detect data following well-defined patterns.

matchers:
  regex:
    - label: "ZIP_CODE"
      pattern: "\b\d{5}\b"

checksum

The checksum matcher validates numbers using checksum algorithms.

Number	Valid Credit Card?
4532-1234-5678-9010	Yes
The cc: <1234-5678-9012-3456>	No
\n5105105105105100\n	Yes

Use checksum if you want to match only those entities which pass validation algorithms.

matchers:
  checksum:
    - label: "CREDIT_CARD"
      type: "credit_card"

For a list of supported checksums, see the set_checksum documentation.

Overlap Handling

When matches overlap, DataMasque merges them by extending boundaries to cover all overlapping text.

The final label is determined by considering:

• the start positions,
• the match lengths,
• and finally the matcher priority.

Note: Adjacent entities that touch but don't overlap are masked separately.

Same-label overlap

Multiple matchers may be assigned to the same label:

matchers:
  context_sources:
    - label: "LAST_NAME"
      column: last_name
  ai_detect:
    - label: "LAST_NAME"
      use_preset: "LAST_NAME"

matcher	label	clinical_notes
context_sources	LAST_NAME	Patient Rebecca de Santis arrived.
ai_detect	LAST_NAME	Patient Rebecca de Santis arrived.
combined	LAST_NAME	Patient Rebecca de Santis arrived.

As both detected entities have the same label, the combined LAST_NAME is the extension of both.

Cross-label overlap

Multiple matchers may overlap across different labels:

matchers:
  context_sources:
    - label: "ORGANISATION"
      column: org_name
  ai_detect:
    - label: "LOCATION"
      use_preset: "STREET_ADDRESS"

matcher	label	text
context_sources	ORGANISATION	Visit Acme Bank, London …
ai_detect	LOCATION	Visit Acme Bank, London …
combined	LOCATION	Visit Acme Bank, London …

If entities with different labels overlap, then the winning label is the one with the left-most start position.

If entities have the same left-most start position, the winning label is the longest one. In the above example, the LOCATION label is used because it is the longest.

When both start positions and lengths are equal, then matcher priorities apply.

Matcher Priorities

From highest to lowest:

1. context_sources
2. seed_files
3. ai_detect
4. checksum
5. regex

Context Filters

Context filters refine matches and can be applied to any matcher.

match_whole_words_only

By default, matchers will match partial words. match_whole_words_only prevents matching within words.

first_name	Without filter	With match_whole_words_only
Tom	Tom arrived	Tom arrived
Tom	Tomorrow	Tomorrow

Use match_whole_words_only to exclude partial word matches.

matchers:
  context_sources:
    - label: "PATIENT_FIRST_NAME"
      column: first_name
      context_filters:
        - type: match_whole_words_only

exclude_pattern

Excludes matches which match a regex pattern.

Without filter	Exclude "@example.com"
Contact: john.smith@example.com	Contact: john.smith@example.com
Email: jose.garcia@email.com	Email: jose.garcia@email.com

Use exclude_pattern to filter out applying regex patterns.

matchers:
  regex:
    - label: "EMAIL"
      pattern: '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
      context_filters:
        - type: exclude_pattern
          pattern: '@example\.com$'

include_pattern

Includes only matches which match a regex pattern.

Without filter	Include " Hospital"
Royal Melbourne Hospital treats…	Royal Melbourne Hospital treats…
Royal Melbourne clinic is closed	Royal Melbourne clinic is closed

Use include_pattern to only include applying regex patterns.

matchers:
  seed_files:
    - label: "HOSPITAL_NAME"
      seed_file: "hospitals.csv"
      seed_column: "hospital_name"
      context_filters:
        - type: include_pattern
          pattern: " Hospital$"

add_prefix

Extends matches to include prefixes.

Without filter	Extended with "Dr. "
Dr. Jones consulted	Dr. Jones consulted
Jones arrived	Jones arrived

Use add_prefix to include leading context.

matchers:
  context_sources:
    - label: "DOCTOR_NAME"
      column: last_name
      context_filters:
        - type: add_prefix
          prefix: "Dr. "

add_suffix

Extends matches to include a suffix if present.

hospital_name	Without filter	Extended with " Hospital"
Royal Melbourne	Royal Melbourne treats	Royal Melbourne treats
Royal Melbourne	Royal Melbourne Hospital	Royal Melbourne Hospital

Use add_suffix to include trailing context.

matchers:
  seed_files:
    - label: "HOSPITAL_NAME"
      seed_file: "hospitals.csv"
      seed_column: "hospital_name"
      context_filters:
        - type: add_suffix
          suffix: " Hospital"

Combining filters

Context Filters are applied in two phases:

1. Validation filters (include_pattern, exclude_pattern, match_whole_words_only) are applied first
2. Extension filters (add_prefix, add_suffix) are applied second

matchers:
  seed_files:
    - label: "DOCTOR_NAME"
      seed_file: "DataMasque_lastNames.csv"
      seed_column: "lastnames"
      context_filters:
        - type: match_whole_words_only
        - type: add_prefix
          prefix: "Dr. "

Text input	Match	Step 1: match_whole_words_only	Step 2: add_prefix	Final result
"Dr. Smith consulted"	Smith ✓	whole word ✓	extends to Dr. Smith ✓	Dr. Smith masked
"Smithson arrived"	Smith ✓	part of word ✗	-	Not masked
"Jones arrived"	Jones ✓	whole word ✓	no prefix found	Jones masked

Configuring Masks for Labels

After defining which labels to match for, a mask must be defined for each label.

Important: DataMasque will raise an error if a matcher defines a label without an associated mask. If you wish to detect entities without masking, you can use the do_nothing mask

Note: Labels must be UPPER_CASE.

Detected label	Mask Used	Original text	Masked text
PHONE	from_fixed	Call 555-123-4567.	Call XXX-XXX-XXXX.
REFERENCE_ID	imitate	Reference ID: AC-891234	Reference ID: BK-472851

matchers:
  ...

masks:
  - label: "PHONE"
    masks:
      - type: from_fixed
        value: "XXX-XXX-XXXX"

  - label: "REFERENCE_ID"
    masks:
      - type: imitate

For a complete list of masks, see Masking Functions Overview.

Supported masks

The following masks can be used to mask detected entities inside unstructured_text:

• brazilian_cpf
• chain
• concat
• credit_card
• do_nothing
• from_choices
• from_column
• from_file
• from_fixed
• from_format_string
• from_random_boolean
• from_random_date
• from_random_datetime
• from_random_number
• from_random_text
• from_unique
• from_unique_imitate
• imitate (alias substitute)
• imitate_nz_ird
• imitate_unique
• imitate_uuid
• replace_regex
• replace_substring
• retain_age
• retain_date_component
• retain_year
• set_checksum
• social_security_number
• take_substring
• transform_case
• typecast

Other masks are not supported when nested inside unstructured_text and are rejected during ruleset validation.

Note on masks that return non-string values: Nested masks inside unstructured_text replace matched text in the source, so their output must be a string.

Integers, floats, boolean values, and UUIDs are converted to a string automatically, so a chain ending in from_random_boolean (outputs 1 or 0, not True/False), from_random_number, typecast to integer, float, decimal, or boolean, or imitate_uuid works as-is.

Dates and datetimes need a format hint. If your chain ends in from_random_date or from_random_datetime, add typecast to string (or any mask that returns a string) to control the output format:

matchers:
  regex:
    - label: DOB
      pattern: '\d{2}/\d{2}/\d{4}'
masks:
  - label: DOB
    masks:
      - type: from_random_date
        min: "1970-01-01"
        max: "2000-12-31"
      - type: typecast
        typecast_as: string
        date_format: "%d/%m/%Y"   # match the source-text date shape

Without a mask that returns a string at the end, the masking run fails at runtime with an error naming the unsupported type.

fallback_masks replace the entire database column or field in the file being masked, so their output is not converted automatically. Add typecast to string at the end of fallback_masks if you need string output.

Note on date masks: retain_age, retain_date_component and retain_year parse the entity using the configured date_format (default %Y-%m-%d). ai_detect passes the detected string straight to the mask, so detected entities which do not match the expected date_format (e.g. 24 May 2024 against %Y-%m-%d) will fall through to the on_invalid action. Set date_format to match your source data, or expect the fallback behaviour.

Note on imitate_uuid and imitate_nz_ird: imitate_uuid and imitate_nz_ird default to on_invalid: error. The run fails if a matched entity isn't a valid UUID or NZ IRD, so set on_invalid: skip to leave non-matching entities unchanged.

Note on brazilian_cpf and social_security_number: brazilian_cpf and social_security_number default to on_invalid: mask: they generate a fresh valid value when the entity contains usable digits, and otherwise pass the entity through unchanged.

Fallback Masks

Recommendation: Always configure fallback_masks when using ai_detect, so a single document failure doesn't abort the masking run. If you'd rather the run fail fast on a failure, omit fallback_masks.

If the ai_detect matcher fails to process text (e.g. a network timeout or an out-of-memory error on the Local AI Engine), fallback_masks can be applied to the entire text instead of raising an error.

This following ruleset uses the ai_detect matcher to match on first names. If processing fails, the entire text is replaced.

matchers:
  ai_detect:
    - label: "FIRST_NAME"
      use_preset: "FIRST_NAME"

masks:
  - label: "FIRST_NAME"
    masks:
      - type: from_file
        seed_file: DataMasque_firstNames_mixed.csv
        seed_column: firstname-mixed

fallback_masks:
  - type: from_fixed
    value: "[REDACTED]"

AI Engine Status	clinical_notes (before)	clinical_notes (after)
Online	Patient Heiko arrived	Patient Marnie arrived
Network Error	Patient Heiko arrived	[REDACTED]

The fallback replaces the entire text, not individual entities.