Examples
- Example 1: Medical Notes
- Example 2: AI/ML Fine-Tuning Dataset
- Example 3: Customer Support Logs in S3
- Example 4: HIPAA-Compliant Healthcare Records
Example 1: Medical Notes
In this example, we will see how to mask "FIRST_NAME" consistently in both
structured and unstructured data
within a medical_records table.
version: "1.0"
tasks:
- type: mask_table
table: medical_records
key: row_id
rules:
- column: clinical_notes
hash_columns:
- patient_id
masks:
- type: unstructured_text
matchers:
context_sources:
- label: "FIRST_NAME"
column: first_name
masks:
- label: "FIRST_NAME"
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
- column: first_names
hash_columns:
- patient_id
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
| patient_id | first_name (before) | first_name (after) | clinical_notes (before) | clinical_notes (after) |
|---|---|---|---|---|
| pt-001 | Bradley | Korrie | Patient Bradley still unwell. | Patient Korrie still unwell. |
| pt-001 | Bradley | Korrie | Follow-up for Bradley. | Follow-up for Korrie. |
| pt-002 | Amir | Janah | Amir has surgery scheduled. | Janah has surgery scheduled. |
Key ideas:
- Matchers detect labels.
- Each label is assigned a mask.
context_sourcesassociates values fromfirst_namestoclinical_notes.hash_columnsmaintains consistency across patients.
Next steps: See Deterministic Masking for more patterns.
Example 2: AI/ML Fine-Tuning Dataset
In this example, we will see how to mask sensitive data in a fine-tuning dataset using the ai_detect matcher
while preserving consistency with hash_sources.
Note: Using the
ai_detectmatcher requires a connection with the DataMasque AI Engine.
version: "1.0"
tasks:
- type: mask_table
table: training_data
key: row_id
rules:
- column: conversation_logs
masks:
- type: unstructured_text
hash_sources:
- self: entity
matchers:
ai_detect:
- label: "FIRST_NAME"
use_preset: "FIRST_NAME"
- label: "BIRTH_CITY"
guidelines:
- "Cities/towns matching a user's place of birth"
- 'Valid examples: "I was born in [Sydney]"'
- 'Invalid examples: "calling from our [Sydney] office", Reason: not birth city'
masks:
- label: "FIRST_NAME"
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
- label: "BIRTH_CITY"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: city
| conversation_logs (before) | conversation_logs (after) |
|---|---|
| Hi, I'm Sara&h. For verification, I was born in Sydney | Hi, I'm Jalisha. For verification, I was born in Albion |
| Thank you Sara&h. I've verified your birth city of Sydney | Thank you Jalisha. I've verified your birth city of Albion |
Note: Sara&h is deliberately mis-spelt to showcase
ai_detect's handling of typos.
Key ideas:
ai_detectcan match on entities with typos or ambiguities.hash_sourcesensures consistent masking within a document.
Next steps: See ai_detect for more on AI-powered entity detection.
Example 3: Customer Support Logs in S3
In this example, we will see how to match on only customer account codes with the help of a context filter.
version: "1.0"
tasks:
- type: mask_file
recurse: true
include:
- glob: "*.txt"
rules:
- masks:
- type: unstructured_text
matchers:
seed_files:
- label: "ACCOUNT_CODE"
seed_file: customer_account_codes.csv # Contains AL, TK, MS, ...
seed_column: account_code
context_filters:
- type: match_whole_words_only
masks:
- label: "ACCOUNT_CODE"
masks:
- type: from_fixed
value: "XX"
| File content | File content (masked, without context_filters) | File content (masked, with context_filters) |
|---|---|---|
| ALERT!! Account AL reported invoice discrepancy. Contact by end of day? | XXERT!! Account XX reported invoice discrepancy. Contact by end of day? | ALERT!! Account XX reported invoice discrepancy. Contact by end of day? |
Key ideas:
context_filterscan refine how matchers identify entities in text.- Specifically,
match_whole_words_onlyprevents partial matches within larger words.
Next steps: See Context Filters for all filter types.
Example 4: HIPAA-Compliant Healthcare Records
In this example, we will see how to mask a medical_records table against the
Local AI Engine tested labels
covering HIPAA Safe Harbor identifiers.
Note: Using the
ai_detectmatcher requires a connection with the DataMasque AI Engine.
version: "1.0"
tasks:
- type: mask_table
table: medical_records
key: row_id
rules:
- column: clinical_notes
masks:
- type: unstructured_text
hash_sources:
- self: entity
matchers:
ai_detect:
- label: "FIRST_NAME"
- label: "MIDDLE_NAME"
- label: "LAST_NAME"
- label: "NATIONALITY"
- label: "STREET_ADDRESS"
- label: "STATE"
- label: "CITY"
- label: "COUNTY"
- label: "SUBURB"
- label: "ZIPCODE"
- label: "COUNTRY"
- label: "LOCATION"
- label: "CREDIT_DEBIT_CARD"
- label: "DATE"
- label: "PHONE_FAX_NUMBER"
- label: "EMAIL"
- label: "URL"
- label: "IP_ADDRESS"
- label: "GENDER"
- label: "ORGANIZATION"
- label: "OTHER_ID"
masks:
- label: "FIRST_NAME"
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
- label: "MIDDLE_NAME"
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
- label: "LAST_NAME"
masks:
- type: from_file
seed_file: DataMasque_lastNames.csv
seed_column: lastnames
- label: "NATIONALITY"
masks:
- type: from_choices
choices:
- "American"
- "British"
- "Canadian"
- "Australian"
- "German"
- "French"
- "Japanese"
- "Brazilian"
- label: "STREET_ADDRESS"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: street_name
- label: "STATE"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: state_long
- label: "CITY"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: city
- label: "COUNTY"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: county
# `DataMasque_US_addresses_large.csv` has no suburb column,
# so we reuse the city column as a stand-in.
- label: "SUBURB"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: city
- label: "ZIPCODE"
masks:
- type: imitate
- label: "COUNTRY"
masks:
- type: from_choices
choices:
- "United States"
- "United Kingdom"
- "Canada"
- "Australia"
- "Germany"
- "France"
- "Japan"
- "Brazil"
- label: "LOCATION"
masks:
- type: from_file
seed_file: DataMasque_US_addresses_large.csv
seed_column: city
- label: "CREDIT_DEBIT_CARD"
masks:
- type: credit_card
- label: "DATE"
masks:
- type: from_fixed
value: "[XX-XX-XXXX]"
- label: "PHONE_FAX_NUMBER"
masks:
- type: imitate
- label: "EMAIL"
masks:
- type: concat
masks:
- type: from_random_text
min: 8
max: 12
- type: from_fixed
value: "@example.com"
- label: "URL"
masks:
- type: concat
masks:
- type: from_fixed
value: "https://"
- type: from_random_text
min: 6
max: 12
- type: from_fixed
value: ".example.com"
- label: "IP_ADDRESS"
masks:
- type: imitate
- label: "GENDER"
masks:
- type: from_choices
choices:
- "Male"
- "Female"
- "Non-binary"
- "Other"
- label: "ORGANIZATION"
masks:
- type: from_choices
choices:
- "General Hospital"
- "City Medical Center"
- "Regional Health Network"
- "Community Care Clinic"
- "Memorial Healthcare"
- "University Medical Group"
- "Riverside Medical"
- "Northside Health"
- label: "OTHER_ID"
masks:
- type: imitate
| clinical_notes (before) | clinical_notes (after) |
|---|---|
| Patient Marcus Chen (Korean-American, MRN WG-2024-08412) admitted to Westbrook General Hospital on 24 May 2024. Phone 206-555-0142, m.chen@example.com. | Patient Dagmar Yvorel (Brazilian, MRN GM-7791-17654) admitted to Riverside Medical on [XX-XX-XXXX]. Phone 358-188-9120, UCbMLvEgd@example.com. |
Key ideas:
- 21
ai_detectlabels covering the HIPAA Safe Harbor identifiers supported by the Local AI Engine. - A different mask type is chosen per label:
from_file,from_choices,imitate,credit_card,from_fixed, andconcatfor synthetic emails and URLs. hash_sourceswithself: entitykeeps each entity's masked value consistent within a document.
Next steps: See Local AI Engine Tested Labels for the full list.