DataMasque Portal

AI Engine

The DataMasque AI Engine unlocks the ai_detect matcher for AI-powered entity detection in unstructured text.

DataMasque supports two AI Engines:

  • the Local AI Engine, which runs on your own infrastructure (on-premise or on a cloud instance such as AWS EC2); and
  • the Bedrock AI Engine, which runs against AWS Bedrock.

Both engines are configured with the same dm_ai_engine_url setting, so this guide applies to either engine unless stated otherwise.

Note: The DataMasque AI Engines are currently preview features.

For a preview build of either engine please contact support@datamasque.com.

Configuration

Setting up the DataMasque AI Engine

Important: DataMasque must be installed before installing a DataMasque AI Engine.

Follow the instructions in the README.md bundled with the DataMasque AI Engine to set it up.

Connecting to the DataMasque AI Engine

Getting a User Token

First, obtain a User Token by logging in via the API. See POST /api/auth/token/login/ for details.

Setting the AI Engine URL

Send a PATCH request to /api/settings/ with the dm_ai_engine_url field:

curl -X PATCH "https://<your-datamasque-host>/api/settings/" \
     -H "Authorization: Token <your-user-token>" \
     -H "Content-Type: application/json" \
     -d '{"dm_ai_engine_url": "http://<ai-engine-host>:9021"}'

Note: Add -k for self-signed certificates.

Replace:

  • <your-datamasque-host> with your DataMasque instance hostname
  • <your-user-token> with the token obtained from the login step
  • <ai-engine-host> with your AI Engine's hostname

The default port for the DataMasque AI Engine is 9021.

This setting applies to all masking runs by default. For more information about the settings API, see the API Endpoints documentation.

Verification

You can verify that the setting is configured via the DataMasque web interface:

  1. Navigate to the Settings page
  2. Scroll to the DataMasque AI Engine section. Note that this section will only appear if the AI Engine URL has been configured
  3. Confirm your URL matches the one you entered

You can check that the AI Engine is running via its GET /health endpoint. See the DataMasque AI Engine's README.md for further instructions.

AI Engine Behaviour

The two AI Engines share the same ai_detect configuration schema but differ in a few behaviours.

Supported languages

ai_detect has been tested on English documents. It may still work for other languages, but accuracy is not guaranteed and should be evaluated on a sample before use.

Processing failures

Both engines can fail on individual documents. Always configure fallback_masks unless you want the run to abort on the first failure.

  • Local AI Engine can fail on out-of-memory errors under load, or on input containing lone surrogates or U+FFFD characters.
  • Bedrock AI Engine may raise ContentFiltered warnings (AWS Bedrock guardrails) or EntityMapping failures when the model's output cannot be reconciled with the input text.

Speed

Since ai_detect uses an AI model it can be slower than other matchers. To increase AI Engine speed, two settings can help:

  • File masking: increase the task workers (e.g. 64) to allow masking multiple files in parallel.
  • Database masking: increase the task workers (e.g. 64) and reduce the batch_size run option (e.g. 1)

These settings maximise the quantity of text sent to the GPU (Local AI Engine) or Bedrock API (Bedrock AI Engine).

See Performance Optimisation for general tuning advice.

Cost

Local AI Engine has no per-token cost. Its only running cost is the uptime of the underlying compute (e.g. an AWS EC2 instance or on-premise hardware).

Bedrock AI Engine costs are billed per token by AWS Bedrock. Assuming Claude Sonnet 4.5 is used as the underlying model, processing 100 documents (1,000 characters each, 20 labels) costs approximately $1 USD. Costs scale linearly with document length, the number of labels, and the length of any custom guidelines.

For more details on Bedrock token usage, see the README.md provided with the DataMasque Bedrock AI Engine.

Presets and Tested Labels

Each AI Engine ships with its own list of supported entities.

Local AI Engine Tested Labels

The Local AI Engine has been validated against the labels below. The resulting detected entities are designed to map cleanly to downstream masks, and are aligned with HIPAA Safe Harbor identifiers where applicable.

Note: The Local AI Engine may detect more than necessary to minimise the risk of PII leakage. If you do not want to mask certain entities (e.g. DOCTOR_FIRST_NAME), add them to the ruleset and use the do_nothing mask.

Specify a tested label directly:

matchers:
  ai_detect:
    - label: "PATIENT_FIRST_NAME"

For a worked ruleset using these tested labels, see Example 4: HIPAA-Compliant Healthcare Records.

Note: use_preset: "PATIENT_FIRST_NAME" is also accepted and is equivalent.

Personal identifiers

Label Description Example
FIRST_NAME Given name [Marcus] arrived for his appointment
MIDDLE_NAME Middle name Sofia [Marie] Hernandez
LAST_NAME Surname Patient Marcus [Chen]
GENDER Gender (where identifying) Patient is [female], 34 yo
NATIONALITY Country adjective or ethnicity Patient is [Korean-American]

Locations

Label Description Example
ZIPCODE Postal/ZIP code Lives in [98109]
STREET_ADDRESS Street-level address Mailed to [1247 Magnolia Lane]
SUBURB Suburb name Resides in [Capitol Hill]
CITY City name Treated in [Seattle]
COUNTRY Country name Travelled to [Singapore]
COUNTY County name Filed in [King County]
STATE State or province Lives in [California]
LOCATION Catch-all not covered by other location labels Lives near [Pike Place Market]

Other

Label Description Example
CREDIT_DEBIT_CARD Card numbers, CVVs, partials (expiry dates fall under DATE) Charged to [4532-7890-1234-5678]
DATE All dates regardless of type; holidays e.g. "Christmas" are not tagged In town [24 May]-[28 May]
EMAIL Email address Contact [m.chen@example.com]
IP_ADDRESS IPv4 or IPv6 address Logged from [10.42.118.205]
ORGANIZATION Organisation name (sub-departments excluded; e.g. "Tesla Call Center" tags only Tesla) Works at [Westbrook General Hospital]
OTHER_ID Catch-all identifier: SSN, MRN, account number, license plate, etc. MRN [WG-2024-08412]
PHONE_FAX_NUMBER Phone or fax number (the two are not differentiated) Called [206-555-0142]
URL Web URL See [https://portal.westbrook.example.com]

Medical domain

These replace the generic FIRST_NAME / LAST_NAME / MIDDLE_NAME labels in medical contexts.

Note: Combining these with FIRST_NAME, LAST_NAME, or MIDDLE_NAME may lead to unexpected results. If combining, please verify the behaviour you require is the behaviour you're getting.

Label Description Example
DOCTOR_FIRST_NAME Doctor's first name [Eleanor] Whitaker, MD
DOCTOR_MIDDLE_NAME Doctor's middle name Eleanor [Margaret] Whitaker, MD
DOCTOR_LAST_NAME Doctor's surname Dr. [Whitaker] reviewed labs
PATIENT_FIRST_NAME Patient's first name [Marcus] arrived for surgery
PATIENT_MIDDLE_NAME Patient's middle name Marcus [Wei] Chen
PATIENT_LAST_NAME Patient's surname Patient Marcus [Chen]

Custom labels

The Local AI Engine also accepts custom labels. Review the model's output on a sample of your data to confirm it captures the entities you want.

matchers:
  ai_detect:
    - label: "EDUCATION_STATUS"

If a custom label is ambiguous, add guidelines to describe what it means. guidelines and use_preset are mutually exclusive.

matchers:
  ai_detect:
    - label: "EDUCATION_STATUS"
      guidelines:
        - "Highest level of education attained"

Bedrock AI Engine Presets

FIRST_NAME

Guidelines:

  • Given name only
  • Excludes titles, middle names, initials
  • 'Valid examples: "Student [Alice] Walker submitted the assignment"'
  • 'Invalid examples: "Manager [J.] approved the request" (Initial)'
LAST_NAME

Guidelines:

  • Surname only
  • Excludes titles, middle names, initials
  • 'Valid examples: "Ms. [Smith] will lead the meeting", "Professor [van der Berg] published the paper" (compound surname as one unit)'
FULL_NAME

Note: Use FIRST_NAME and LAST_NAME if this preset is not behaving as expected.

Guidelines:

  • First and last name together, optionally includes middle names
  • Excludes titles, initials
  • 'Valid examples: "Award winner [Mary Elizabeth Johnson] will speak"'
INITIAL_NAME

Guidelines:

  • Single or multi-letter person-related initials only
  • Excludes titles e.g. Mr, Miss
  • 'Valid examples: "[J. K.] Rowling wrote the book"'
  • 'Invalid examples: "The [U.S.A.] is a large country" (Not an identifying individual)'
TITLE

Guidelines:

  • Professional titles and honorifics, including dot
  • 'Valid examples: "[Dr.] Smith will see you now", "Welcome [Ms] Johnson to the stage"'
AGE

Guidelines:

  • Numeric ages and age ranges of individuals
  • 'Valid examples: "Applicant is [45 years old]", "Participants aged [30-35] were selected"'
  • 'Invalid examples: "DataMasque is [5] years old" (Not an individual)'
  • 'Invalid examples: "Targeted at [elderly] population" (Non-numeric)'
DATE_OF_BIRTH

Guidelines:

  • Dates when context clearly indicates birth
  • 'Valid examples: "DOB: [January 1]", "he was born yesterday (today is [01/05/2024])"'
NATIONALITY

Guidelines:

  • National origin, citizenship when describing identity
  • 'Valid examples: "She is a [British] citizen"'
GENDER

Guidelines:

  • Gender identity or biological sex
  • Excludes pronouns
  • 'Valid examples: "The patient (27[M]) is recovering well", "Participant is [non-binary]"'
RACE

Guidelines:

  • Racial or ethnic identity
  • 'Valid examples: "Patient identified as [African American]"'
EMAIL

Guidelines:

  • Email addresses with @ and domain
  • Includes partial/obfuscated emails
  • 'Valid examples: "Contact: [user+tag@company.co.uk]", "Reach out at [bob@gmail] for details"'
FAX

Guidelines:

  • Fax numbers only when context indicates fax
  • 'Valid examples: "Send documents to fax [487.416.6741x713]"'
PHONE_NUMBER

Guidelines:

  • Phone numbers in any format
  • Includes extensions, partial numbers
  • Excludes fax numbers and placeholders
  • 'Valid examples: "Extension [555.123.4567 ext. 890]", "Heya my number is: *** *** *** [1234]" (Excludes placeholder *s)'
USERNAME

Guidelines:

  • Online handles and account identifiers
  • 'Valid examples: "Follow [@johndoe123] on twitter"'
STREET_ADDRESS

Guidelines:

  • Street number and name only, may include unit/apartment
  • Includes postal codes, PO boxes
  • Excludes city, state, country
  • 'Valid examples: "Ship to [195 Main Street], Boston, MA [02101]", "Office located at [456 Oak Avenue Apt 2B]"'
CITY

Guidelines:

  • Cities, towns, villages
  • 'Valid examples: "Traveling to [Melbourne] next week"'
COUNTRY

Guidelines:

  • Country names, includes abbreviations
  • 'Valid examples: "Visiting the [UK] next month", "Born in [Canada]"'
STATE

Guidelines:

  • States, provinces, territories, includes abbreviations
  • 'Valid examples: "Shipping to [NY]", "Property in [New South Wales]"'
ZIPCODE

Guidelines:

  • ZIP codes and postal codes in any format
  • Can be part of an address or standalone
  • 'Valid examples: "Postcode [SW1A 1AA]"'
JOB_TITLE

Guidelines:

  • Specific, professional roles and job titles
  • 'Valid examples: "Promoted to [Senior Vice President of Marketing]"'
  • 'Invalid examples: "He was a good [manager]" (Generic, non-specific)'
ORGANIZATION

Guidelines:

  • Specific organizations, including appropriate prefixes/suffixes
  • 'Valid examples: "Enrolled at [Stanford University] for fall"'
  • 'Invalid examples: "Works at [a hospital]" (Generic, non-specifying)'
MEDICAL_CONDITION

Guidelines:

  • Diseases, illnesses, diagnoses
  • 'Valid examples: "Diagnosed with [hypertension]", "Treatment for [Type 2 diabetes]"'