AI Engine

The DataMasque AI Engine unlocks the ai_detect matcher for AI-powered entity detection in unstructured text.

DataMasque supports two AI Engines:

the Local AI Engine, which runs on your own infrastructure (on-premise or on a cloud instance such as AWS EC2); and
the Bedrock AI Engine, which runs against AWS Bedrock.

Both engines are configured with the same dm_ai_engine_url setting, so this guide applies to either engine unless stated otherwise.

Note: The DataMasque AI Engines are currently preview features.

For a preview build of either engine please contact support@datamasque.com.

Configuration
AI Engine Behaviour
AI Detection Cache
Presets and Tested Labels
- Local AI Engine Tested Labels
- Bedrock AI Engine Presets

Configuration

Setting up the DataMasque AI Engine

Important: DataMasque must be installed before installing a DataMasque AI Engine.

Follow the instructions in the README.md bundled with the DataMasque AI Engine to set it up.

Connecting to the DataMasque AI Engine

Set the AI Engine URL on the Settings page, as described in Choosing the model, or through the API as described below.

Getting a User Token

First, obtain a User Token by logging in via the API. See POST /api/auth/token/login/ for details.

Setting the AI Engine URL

Send a PATCH request to /api/settings/ with the dm_ai_engine_url field:

curl -X PATCH "https://<your-datamasque-host>/api/settings/" \
     -H "Authorization: Token <your-user-token>" \
     -H "Content-Type: application/json" \
     -d '{"dm_ai_engine_url": "http://<ai-engine-host>:9021"}'

Note: Add -k for self-signed certificates.

Replace:

<your-datamasque-host> with your DataMasque instance hostname
<your-user-token> with the token obtained from the login step
<ai-engine-host> with your AI Engine's hostname

The default port for the DataMasque AI Engine is 9021.

This setting applies to all masking runs by default. A PATCH request saves the URL without checking it against the AI Engine. To check the URL first, or to choose the model, use the Settings page or the PUT request in Choosing the model. For more information about the settings API, see the API Endpoints documentation.

Choosing the model

The Local AI Engine installs two models and serves one at a time:

Model	Description
Precision (the default)	Very accurate. Reads text with broken spacing or non-Latin characters well.
Express	About 10 times faster than Precision. Misses more in text with broken spacing or non-Latin characters. At most 40 labels per mask.

Express accepts at most 40 ai_detect labels in one unstructured_text mask, with roughly 800 characters of label names in total. DataMasque checks the labels before masking and fails a run whose labels do not fit.

To compare the models on your own data, mask a few texts in the Unstructured Ruleset Builder, switch models, and mask them again.

To choose the model, go to the DataMasque AI Engine section of the Settings page:

Enter the AI Engine URL.
Click Test Connection. DataMasque reads the AI Engine at that URL and shows its status under Model. The test saves nothing.
Choose the model. The model the engine serves is marked Ready. The options are disabled until a test, and editing the URL disables them again.
Click Save. If you chose a model other than the one marked Ready, click Save and Switch to confirm the restart.

When the page opens with a saved URL, it tests the connection by itself.

Switching restarts the engine with the other model: about a minute for Express, up to five minutes for Precision. Until then, the AI Engine cannot process requests. The next masking run that uses the AI detection cache clears it. A masking run that uses ai_detect waits up to ten minutes for the engine to answer, and fails if it does not, or cannot load its model.

You cannot switch while database or file runs are queued or running. That check covers only this DataMasque's runs.

An option the engine cannot serve is disabled, with the reason under it:

An older Local AI Engine serves only Precision. Update the engine to choose Express.
A Bedrock AI Engine uses the Bedrock model set in its own configuration. Both options are disabled.

The model can also change during a run when another DataMasque that shares the engine switches it, or when someone changes it on the engine's host. Each run's log names the model it started with:

Masking with the AI Engine's Precision model (`base`).

To set the URL and the model through the API, send a PUT request. model_size takes the model's ID: base for Precision, small for Express.

curl -X PUT "https://<your-datamasque-host>/api/ai-engine/" \
     -H "Authorization: Token <your-user-token>" \
     -H "Content-Type: application/json" \
     -d '{"dm_ai_engine_url": "http://<ai-engine-host>:9021", "model_size": "small"}'

Verification

You can verify that the setting is configured via the DataMasque web interface:

Navigate to the Settings page
Scroll to the DataMasque AI Engine section
Confirm your URL matches the one you entered, and that the status under Model starts with Ready

To test a URL through the API, see POST /api/ai-engine/test/. You can check that the AI Engine is running via its GET /health endpoint. See the DataMasque AI Engine's README.md for further instructions.

AI Engine Behaviour

The two AI Engines share the same ai_detect configuration schema but differ in a few behaviours.

Supported languages

ai_detect has been tested on English documents. It may still work for other languages, but accuracy is not guaranteed and should be evaluated on a sample before use.

Processing failures

Both engines can fail on individual documents. Always configure fallback_masks unless you want the run to abort on the first failure.

Local AI Engine can fail on out-of-memory errors under load, or on input containing lone surrogates or U+FFFD characters.
Bedrock AI Engine may raise ContentFiltered warnings (AWS Bedrock guardrails) or EntityMapping failures when the model's output cannot be reconciled with the input text.

Speed

ai_detect uses an AI model, so it will be slower than other matchers. DataMasque sends many texts to the AI Engine together. The AI Engine queue holds at most 10,000 texts, so sending more texts does not make masking faster.

Database masking: set the task workers and the batch_size run option so that workers * batch_size is at least 10,000.
File masking: each worker sends a maximum of 10,000 texts or 64 MiB of files at a time.

The batch processing occurs only when unstructured_text is directly in the rule's masks list:

`masks`	`mask_table`	`mask_file`
`unstructured_text` first	Yes	Yes
Other masks before `unstructured_text`	Yes	No
`unstructured_text` inside a `chain`, `concat`, `json`, `xml`, `replace_regex` or `replace_substring` mask	No	No

For mask_file, batch processing applies only to the first rule that a file reaches.

The following formats do not use batch processing yet: .json, .ndjson, .xml, and .avro. Image masking and mask_tabular_file tasks also do not use batch processing. mask_table rules with column: "*" (Amazon DynamoDB, MongoDB and Apache Cassandra) send one text at a time.

Without batch processing, a worker sends one text at a time, so increasing the number of workers increases the speed as more texts are sent to the AI Engine. The recommended settings for these are:

Database masking: increase the task workers (e.g. 64) and reduce the batch_size run option (e.g. 1).
File masking: increase the task workers (e.g. 64) to allow masking multiple files in parallel.

For repeat runs over mostly unchanged data, set cache_ai_detect_spans to true on the task so that later runs send only new or changed text to the AI Engine.

See Performance Optimisation for general tuning advice.

Cost

Local AI Engine has no per-token cost. Its only running cost is the uptime of the underlying compute (e.g. an AWS EC2 instance or on-premise hardware).

Bedrock AI Engine costs are billed per token by AWS Bedrock. Assuming Claude Sonnet 4.5 is used as the underlying model, processing 100 documents (1,000 characters each, 20 labels) costs approximately $1 USD. Costs scale linearly with document length and the number of labels. With cache_ai_detect_spans set to true, later runs are billed only for new or changed text.

For more details on Bedrock token usage, see the README.md provided with the DataMasque Bedrock AI Engine.

AI Detection Cache

When a mask_table, mask_file or mask_tabular_file task sets cache_ai_detect_spans to true, DataMasque stores the result of each ai_detect detection. A later run that masks the same text with the same ai_detect matchers reuses the stored result, and does not send that text to the AI Engine. This makes repeat runs over mostly unchanged data faster, and lowers the token cost of the Bedrock AI Engine.

The following task caches the ai_detect results for the body column:

version: "1.0"
tasks:
  - type: mask_table
    table: messages
    key: message_id
    cache_ai_detect_spans: true
    rules:
      - column: body
        masks:
          - type: unstructured_text
            matchers:
              ai_detect:
                - label: "FIRST_NAME"
                  use_preset: "FIRST_NAME"
            masks:
              - label: "FIRST_NAME"
                masks:
                  - type: from_file
                    seed_file: DataMasque_firstNames_mixed.csv
                    seed_column: firstname-mixed

cache_ai_detect_spans is a task parameter. Set it on the task, not on the unstructured_text mask or its ai_detect matchers. A mask_file or mask_tabular_file task takes the same parameter.

DataMasque sends a text to the AI Engine again when:

the text changes; or
the ai_detect matchers of the mask change, including their labels, presets or order.

Nothing else in the ruleset affects reuse.

The cache holds where each entity was found in a text, not the text itself. DataMasque encrypts the cache and keeps it on the files volume, with seed files and uploads. All tasks that set cache_ai_detect_spans to true share one cache, across runs and across agent containers that share the files volume.

The cache has no size limit. It keeps an entry for each distinct text and set of ai_detect matchers until one of these events deletes it:

Event	Effect on the cache
You clear the cache on the Settings page	DataMasque deletes the entries you chose.
The AI Engine URL, model, engine version or chunk settings change	The next task that uses the cache deletes every entry first, and the run log says so.
You rotate the instance secret	DataMasque deletes every entry.

To limit the size of the cache, regularly clear the entries that no masking run has used in the last month or year.

If DataMasque cannot create or write the cache directory, the task fails. If the AI Engine health check fails or does not report a model, the task runs without the cache, and the run log shows a warning.

The run log shows a line like this for each task:

AI detection cache for `messages`: 9,800 reused, 200 detected, 0 rejected

reused: texts answered from the cache, with no AI Engine call.
detected: texts the AI Engine detected.
rejected: stored results that DataMasque could not use. Those texts are detected again.

For a mask_file or mask_tabular_file task, the line shows the source directory in double quotes instead of the table.

A warning follows if some results could not be stored, if some cache files could not be opened, or if some cache files could not be deleted when the task cleared the cache. The task carries on in each case.

To see the size of the cache, or to delete entries from it, see AI Detection Cache on the Settings page.

Presets and Tested Labels

Each AI Engine ships with its own list of supported entities.

Local AI Engine Tested Labels are labels the Local AI Engine has been validated against.
Bedrock AI Engine Presets are labels built into the Bedrock AI Engine.

Local AI Engine Tested Labels

The Local AI Engine has been validated against the labels below. The resulting detected entities are designed to map cleanly to downstream masks, and are aligned with HIPAA Safe Harbor identifiers where applicable.

Note: The Local AI Engine may detect more than necessary to minimise the risk of PII leakage. If you do not want to mask certain entities (e.g. DOCTOR_FIRST_NAME), add them to the ruleset and use the do_nothing mask.

Specify a tested label directly:

matchers:
  ai_detect:
    - label: "PATIENT_FIRST_NAME"

For a worked ruleset using these tested labels, see Example 4: HIPAA-Compliant Healthcare Records.

Note: use_preset: "PATIENT_FIRST_NAME" is also accepted and is equivalent.

Personal identifiers

Label	Description	Example
`FIRST_NAME`	Given name	[Marcus] arrived for his appointment
`MIDDLE_NAME`	Middle name	Sofia [Marie] Hernandez
`LAST_NAME`	Surname	Patient Marcus [Chen]
`GENDER`	Gender (where identifying)	Patient is [female], 34 yo
`NATIONALITY`	Country adjective or ethnicity	Patient is [Korean-American]

Locations

Label	Description	Example
`ZIPCODE`	Postal/ZIP code	Lives in [98109]
`STREET_ADDRESS`	Street-level address	Mailed to [1247 Magnolia Lane]
`SUBURB`	Suburb name	Resides in [Capitol Hill]
`CITY`	City name	Treated in [Seattle]
`COUNTRY`	Country name	Travelled to [Singapore]
`COUNTY`	County name	Filed in [King County]
`STATE`	State or province	Lives in [California]
`LOCATION`	Catch-all not covered by other location labels	Lives near [Pike Place Market]

Other

Label	Description	Example
`CREDIT_DEBIT_CARD`	Card numbers, CVVs, partials (expiry dates fall under `DATE`)	Charged to [4532-7890-1234-5678]
`DATE`	All dates regardless of type; holidays e.g. "Christmas" are not tagged	In town [24 May]-[28 May]
`EMAIL`	Email address	Contact [m.chen@example.com]
`IP_ADDRESS`	IPv4 or IPv6 address	Logged from [10.42.118.205]
`ORGANIZATION`	Organisation name (sub-departments excluded; e.g. "Tesla Call Center" tags only `Tesla`)	Works at [Westbrook General Hospital]
`OTHER_ID`	Catch-all identifier: SSN, MRN, account number, license plate, etc.	MRN [WG-2024-08412]
`PHONE_FAX_NUMBER`	Phone or fax number (the two are not differentiated)	Called [206-555-0142]
`URL`	Web URL	See [https://portal.westbrook.example.com]

Medical domain

These replace the generic FIRST_NAME / LAST_NAME / MIDDLE_NAME labels in medical contexts.

Note: Combining these with FIRST_NAME, LAST_NAME, or MIDDLE_NAME may lead to unexpected results. If combining, please verify the behaviour you require is the behaviour you're getting.

Label	Description	Example
`DOCTOR_FIRST_NAME`	Doctor's first name	[Eleanor] Whitaker, MD
`DOCTOR_MIDDLE_NAME`	Doctor's middle name	Eleanor [Margaret] Whitaker, MD
`DOCTOR_LAST_NAME`	Doctor's surname	Dr. [Whitaker] reviewed labs
`PATIENT_FIRST_NAME`	Patient's first name	[Marcus] arrived for surgery
`PATIENT_MIDDLE_NAME`	Patient's middle name	Marcus [Wei] Chen
`PATIENT_LAST_NAME`	Patient's surname	Patient Marcus [Chen]

Custom labels

The Local AI Engine also accepts custom labels. The model decides what to find from the label's name, so name a custom label for what it means:

matchers:
  ai_detect:
    - label: "EDUCATION_STATUS"

Review the model's output on a sample of your data to confirm it captures the entities you want.

Bedrock AI Engine Presets

The Bedrock AI Engine accepts these names in use_preset:

Kind	Presets
Names	`FIRST_NAME`, `LAST_NAME`, `FULL_NAME`, `INITIAL_NAME`, `TITLE`
Personal details	`AGE`, `DATE_OF_BIRTH`, `NATIONALITY`, `GENDER`, `RACE`
Contact details	`EMAIL`, `FAX`, `PHONE_NUMBER`, `USERNAME`
Addresses	`STREET_ADDRESS`, `CITY`, `COUNTRY`, `STATE`, `ZIPCODE`
Work	`JOB_TITLE`, `ORGANIZATION`
Health	`MEDICAL_CONDITION`

Note: Use FIRST_NAME and LAST_NAME if FULL_NAME is not behaving as expected.