AI Engine
The DataMasque AI Engine unlocks the ai_detect matcher
for AI-powered entity detection in unstructured text.
DataMasque supports two AI Engines:
- the Local AI Engine, which runs on your own infrastructure (on-premise or on a cloud instance such as AWS EC2); and
- the Bedrock AI Engine, which runs against AWS Bedrock.
Both engines are configured with the same dm_ai_engine_url setting,
so this guide applies to either engine unless stated otherwise.
Note: The DataMasque AI Engines are currently preview features.
For a preview build of either engine please contact support@datamasque.com.
Configuration
Setting up the DataMasque AI Engine
Important: DataMasque must be installed before installing a DataMasque AI Engine.
Follow the instructions in the README.md bundled with the DataMasque AI Engine to set it up.
To run the Local AI Engine in an EKS or ROSA cluster, see AI Engine on Kubernetes.
Connecting to the DataMasque AI Engine
Set the AI Engine URL on the Settings page, as described in Choosing the model, or through the API as described below.
Getting a User Token
First, obtain a User Token by logging in via the API. See POST /api/auth/token/login/ for details.
Setting the AI Engine URL
Send a PATCH request to /api/settings/ with the dm_ai_engine_url field:
curl -X PATCH "https://<your-datamasque-host>/api/settings/" \
-H "Authorization: Token <your-user-token>" \
-H "Content-Type: application/json" \
-d '{"dm_ai_engine_url": "http://<ai-engine-host>:9021"}'
Note: Add
-kfor self-signed certificates.
Replace:
<your-datamasque-host>with your DataMasque instance hostname<your-user-token>with the token obtained from the login step<ai-engine-host>with your AI Engine's hostname
The default port for the DataMasque AI Engine is 9021.
This setting applies to all masking runs by default. A PATCH request saves the URL without checking it against the AI Engine. To check the URL first, or to choose the model, use the Settings page or the PUT request in Choosing the model. For more information about the settings API, see the API Endpoints documentation.
Choosing the model
The Local AI Engine installs two models and serves one at a time:
| Model | Description |
|---|---|
| Precision (the default) | Very accurate. Reads text with broken spacing or non-Latin characters well. |
| Express | About 10 times faster than Precision. Misses more in text with broken spacing or non-Latin characters. At most 40 labels per mask. |
Express accepts at most 40 ai_detect labels in one unstructured_text mask,
with roughly 800 characters of label names in total.
DataMasque checks the labels before masking and fails a run whose labels do not fit.
To compare the models on your own data, mask a few texts in the Unstructured Ruleset Builder, switch models, and mask them again.
To choose the model, go to the DataMasque AI Engine section of the Settings page:
- Enter the AI Engine URL.
- Click Test Connection. DataMasque reads the AI Engine at that URL and shows its status under Model. The test saves nothing.
- Choose the model. The model the engine serves is marked Ready. The options are disabled until a test, and editing the URL disables them again.
- Click Save. If you chose a model other than the one marked Ready, click Save and Switch to confirm the restart.
When the page opens with a saved URL, it tests the connection by itself.
Switching restarts the engine with the other model: about a minute for Express, up to five minutes for Precision. Until then, the AI Engine cannot process requests. A switch does not clear the AI detection cache.
A masking run that uses ai_detect waits for the engine before it masks anything:
- While the engine answers that it is loading or restarting, the run waits up to ten minutes for the model, and fails if the engine does not serve it by then, or cannot load it.
- While DataMasque cannot reach the engine, the run waits two minutes. The engine may refuse connections for a short time while it restarts, before it answers that it is loading. After two minutes the run writes a warning to the run log and masks with the engine unreachable: see When the AI Engine is unreachable. Each answer starts the two minutes again, and the whole wait never exceeds ten minutes. If DataMasque cannot reach the engine for more than two minutes while it switches models, the runs that start in that time mask with it unreachable. Wait for the status under Model to read Ready before you start a run.
- While the engine accepts connections but gives no answer, the run waits ten minutes, as for a busy engine. That time does not count toward the two minutes. When the engine has given no answer since the run started waiting, the run then writes a warning to the run log that the engine accepted connections but did not answer, and masks with the engine unreachable.
- An answer that the engine is loading or restarting starts both waits again, but not the ten minutes of the whole wait. An engine that answers and then gives no answer can reach the end of the whole wait first: the run then fails, as for an engine that does not serve its model.
You cannot switch while database or file runs are queued or running. That check covers only this DataMasque's runs.
An option the engine cannot serve is disabled, with the reason under it:
- An older Local AI Engine serves only Precision. Update the engine to choose Express.
- A Bedrock AI Engine uses the Bedrock model set in its own configuration. Both options are disabled.
The model can also change during a run when another DataMasque that shares the engine switches it, or when someone changes it on the engine's host. Each run's log names the model it started with:
Masking with the AI Engine's Precision model (`base`).
A run's Run Object records the engine's model in ai_engine_model,
such as dm-ai-ner-small-1.0.0-20260601.tar.enc,
and its version in ai_engine_version.
To set the URL and the model through the API, send a PUT request.
model_size takes the model's ID: base for Precision, small for Express.
curl -X PUT "https://<your-datamasque-host>/api/ai-engine/" \
-H "Authorization: Token <your-user-token>" \
-H "Content-Type: application/json" \
-d '{"dm_ai_engine_url": "http://<ai-engine-host>:9021", "model_size": "small"}'
Verification
You can verify that the setting is configured via the DataMasque web interface:
- Navigate to the Settings page
- Scroll to the DataMasque AI Engine section
- Confirm your URL matches the one you entered, and that the status under Model starts with Ready
To test a URL through the API, see POST /api/ai-engine/test/.
You can check that the AI Engine is running
via its GET /health endpoint.
See the DataMasque AI Engine's README.md for further instructions.
AI Engine Behaviour
The two AI Engines share the same ai_detect configuration schema
but differ in a few behaviours.
Supported languages
ai_detect has been tested on English documents.
It may still work for other languages,
but accuracy is not guaranteed and should be evaluated on a sample before use.
Processing failures
Both engines can fail on individual documents. Always configure fallback_masks unless you want the run to abort on the first failure.
- Local AI Engine can fail on out-of-memory errors under load,
or on input containing lone surrogates or
U+FFFDcharacters. - Bedrock AI Engine may raise
ContentFilteredwarnings (AWS Bedrock guardrails) orEntityMappingfailures when the model's output cannot be reconciled with the input text.
An HTTP error from an engine that answers is a failure on that one text:
the text gets the fallback_masks, and the next text goes to the engine as usual.
When the AI Engine is unreachable
DataMasque cannot reach the AI Engine when it gets no connection to it (the connection is refused or times out, or the host name does not resolve), or when a proxy in front of the engine answers with a 502 or 503 gateway error. A request can also reach the engine, or a proxy in front of it, and get no answer: it times out after DataMasque connected, the connection closes before any answer, or a proxy answers with a 504 gateway timeout.
It is unreachable when DataMasque cannot reach it for two minutes, or gets no answer from it for ten minutes, counting both kinds of failure: at the start of a run, as described in Choosing the model, or partway through one, once DataMasque's requests have waited that long in all for answers that never came, with no answer in between. The time between two requests counts only when the second follows the first within 15 seconds. A longer pause is a time DataMasque did not ask the engine: it does not count, and the time counted before it is kept. Requests that wait at the same time count that time once.
A request that reached the engine and got no answer may be waiting on an engine that is busy: an older Local AI Engine can take minutes over one large batch. Ten minutes is the longest DataMasque waits for one request, so such a request counts toward the ten minutes but not the two. A proxy answers 504, and a load balancer closes the connection, both when the engine is slow and when the engine's host has stopped, so DataMasque cannot tell the two apart. When the engine's host stops:
- With no proxy, the connection is refused or never accepted, and the run masks with the engine unreachable after two minutes.
- Behind a proxy that answers 503 once the engine is gone, such as a load balancer whose health checks have taken the engine out of service, the run masks with the engine unreachable after two minutes.
- Behind a proxy that answers 504 when it cannot reach the engine's host, as nginx and an AWS Application Load Balancer do, the run masks with the engine unreachable after ten minutes.
Once the engine is unreachable, DataMasque sends it no request,
and masks each text of an unstructured_text mask with ai_detect matchers in this order:
- If the task sets
cache_ai_detect_spanstotrueand the cache holds the text, DataMasque masks it with the cached result, exactly as the run that cached it did. - Otherwise DataMasque applies the mask's
fallback_masksto the whole text. - Without
fallback_masks, the task fails at the first such text, with an error that names the AI Engine URL.
While the engine is unreachable, no text waits for its own timeout. When the wait at the start of a run finds the engine unreachable, every task of the run masks with it unreachable, in a database or a file masking run. When it becomes unreachable partway through, the rest of a database masking run masks with it unreachable, or the rest of the file masking task that found it so. The run or task stays so to its end, even if the engine answers again.
When the wait at the start of a run finds the engine unreachable,
an unstructured discovery run with an ai_detect matcher fails.
When the engine becomes unreachable partway through,
the run lists each file it could not process under AI Engine could not process the file.
The fallback warning names the engine:
AI Engine processing failed for labels - [FIRST_NAME, LAST_NAME]: The AI Engine at "http://ai-engine:9021" is unreachable: it does not answer. Applying fallback masks to entire text. This warning will not be repeated for subsequent rows.
Speed
ai_detect uses an AI model, so it will be slower than other matchers.
DataMasque sends many texts to the AI Engine together.
The AI Engine queue holds at most 10,000 texts,
so sending more texts does not make masking faster.
- Database masking:
set the task
workersand thebatch_sizerun option so thatworkers * batch_sizeis at least 10,000. - File masking: each worker sends a maximum of 10,000 texts or 64 MiB of files at a time.
The batch processing occurs only when unstructured_text is directly in the rule's masks list:
masks |
mask_table |
mask_file |
|---|---|---|
unstructured_text first |
Yes | Yes |
Other masks before unstructured_text |
Yes | No |
unstructured_text inside a chain, concat, json, xml, replace_regex or replace_substring mask |
No | No |
For mask_file, batch processing applies only to the first rule that a file reaches.
The following formats do not use batch processing yet:
.json, .ndjson, .xml, and .avro.
Image masking and mask_tabular_file tasks also do not use batch processing.
mask_table rules with column: "*" (Amazon DynamoDB, MongoDB and Apache Cassandra) send one text at a time.
Without batch processing, a worker sends one text at a time, so increasing the number of workers increases the speed as more texts are sent to the AI Engine. The recommended settings for these are:
- Database masking:
increase the task
workers(e.g.64) and reduce thebatch_sizerun option (e.g.1). - File masking:
increase the task
workers(e.g.64) to allow masking multiple files in parallel.
For repeat runs over mostly unchanged data,
set cache_ai_detect_spans to true on the task
so that later runs send only new or changed text to the AI Engine.
See Performance Optimisation for general tuning advice.
Cost
Local AI Engine has no per-token cost. Its only running cost is the uptime of the underlying compute (e.g. an AWS EC2 instance or on-premise hardware).
Bedrock AI Engine costs are billed per token by AWS Bedrock.
Assuming Claude Sonnet 4.5 is used as the underlying model,
processing 100 documents (1,000 characters each, 20 labels) costs approximately $1 USD.
Costs scale linearly with document length and the number of labels.
With cache_ai_detect_spans set to true,
later runs are billed only for new or changed text.
For more details on Bedrock token usage,
see the README.md provided with the DataMasque Bedrock AI Engine.
AI Detection Cache
When a mask_table,
mask_file
or mask_tabular_file task
sets cache_ai_detect_spans to true,
DataMasque stores the result of each ai_detect detection.
A later run that masks the same text with the same ai_detect matchers reuses the stored result,
and does not send that text to the AI Engine.
This makes repeat runs over mostly unchanged data faster,
and lowers the token cost of the Bedrock AI Engine.
The following task caches the ai_detect results for the body column:
version: "1.0"
tasks:
- type: mask_table
table: messages
key: message_id
cache_ai_detect_spans: true
rules:
- column: body
masks:
- type: unstructured_text
matchers:
ai_detect:
- label: "FIRST_NAME"
use_preset: "FIRST_NAME"
masks:
- label: "FIRST_NAME"
masks:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
cache_ai_detect_spans is a task parameter.
Set it on the task, not on the unstructured_text mask or its ai_detect matchers.
A mask_file or mask_tabular_file task takes the same parameter.
DataMasque sends a text to the AI Engine again when:
- the text changes; or
- the
ai_detectmatchers of the mask change, including their labels, presets or order.
DataMasque caches each result under one combination of AI Engine URL, model and engine version. A run with a different combination sends the text to the AI Engine.
The cache holds where each entity was found in a text, not the text itself.
DataMasque encrypts the cache and keeps it on the files volume, with seed files and uploads.
All tasks that set cache_ai_detect_spans to true share one cache,
across runs and across agent containers that share the files volume.
Before a task ends, it writes its new results to the cache, and records the day it reused each stored result. If the task stops before it ends, for example because its container stops, it does not write them, and later runs send those texts to the AI Engine again. Each run keeps at most 128 MiB of unwritten results in memory. When its unwritten results reach that size, the run writes them before the task ends. Clearing unused results while a task runs can delete results that the task reused. To keep them, clear unused results only when no task that caches results is running. Otherwise, the next run sends those texts to the AI Engine again.
When the files volume is a network filesystem, such as NFS or Amazon EFS, a task that looks up many texts in a cache file reads the whole file once and keeps a copy in memory. The tasks of a run that run at the same time share that copy. Each run masks in its own agent process, which holds up to 256 MiB of copies, so each run that masks at the same time on an agent container can use up to 256 MiB more memory. A task that looks up few texts, such as one of many small tables, reads only what it needs from the file. A result that another run or agent container stores after the copy is read is not found by the tasks that use the copy, which send that text to the AI Engine.
The cache has no size limit.
It keeps an entry for each distinct text and set of ai_detect matchers
until one of these events deletes it:
| Event | Effect on the cache |
|---|---|
| You clear the cache on the Settings page | DataMasque deletes the entries you chose. |
| A task starts with the AI Engine unreachable | Nothing: the task only reads the cache. It neither deletes, adds nor updates entries. |
| A task starts while the AI Engine gives no answer, before it is found unreachable | The task only reads the cache until the engine answers again. The later batches of a database masking task then use the cache as usual: they add entries. |
| The AI Engine becomes unreachable partway through a task | From then on the task only reads the cache. It keeps the entries it stored before. |
| You rotate the instance secret | DataMasque deletes every entry. |
After an AI Engine upgrade, the entries of the earlier version stay until a clear deletes them. To limit the size of the cache, regularly clear the entries that no masking run has used in the last month or year. A task that only reads the cache, with the AI Engine unreachable, does not count as a use: an entry it reads keeps the day it was last used, so such a clear can delete it.
If DataMasque cannot create or write the cache directory, the task fails. If the AI Engine answers its health check with an error, or does not report a model, the task runs without the cache, and the run log shows a warning.
With the AI Engine unreachable,
a task reads the results cached for the AI Engine that the last task with the engine reachable used at the same URL,
and masks every other text with fallback_masks.
DataMasque cannot check that the engine still runs the same model,
so the run log names the engine whose results it read,
with the number of texts read from the cache and the number sent to fallback_masks:
AI detection cache for `messages`: the AI Engine at "http://ai-engine:9021" is unreachable, and the cache answered with the results of the AI Engine with model_path "dm-ai-ner-base-0.0.24-20260420.tar.enc", version "1.26.10.0"; chunk_overlap 250, chunk_size 2500. 9,800 texts read from the cache, 200 sent to `fallback_masks`.
When the AI Engine fails on texts the cache does not hold but is not unreachable, as when it stops answering shortly before the task ends, the line says how many texts it failed on instead of "is unreachable".
When a batch of a task starts while the AI Engine gives no answer, and the engine answers again during that batch, DataMasque masks the texts of that batch the cache does not hold with the engine's results but does not store them, since it did not check the engine's model. The run log says how many:
AI detection cache for `messages`: the AI Engine answered again after it gave no answer, and the results of 40 texts it detected then were not stored. Later runs detect those texts again.
A task finds no cached results when no task has used the cache with the engine reachable at that URL since DataMasque was upgraded to this release.
The run log shows a line like this for each task:
AI detection cache for `messages`: 9,800 reused, 200 detected, 0 rejected
reused: texts answered from the cache, with no AI Engine call.detected: texts the AI Engine detected.rejected: stored results that DataMasque could not use. Those texts are detected again.
For a mask_file or mask_tabular_file task, the line shows the source directory in double quotes instead of the table.
A warning follows if the AI Engine was unreachable (see above), if some results could not be stored, if some cache files could not be opened, or if some cache files could not be deleted when the task cleared the cache. The task carries on in each case.
To see the size of the cache, or to delete entries from it, see AI Detection Cache on the Settings page.
Presets and Tested Labels
Each AI Engine ships with its own list of supported entities.
- Local AI Engine Tested Labels are labels the Local AI Engine has been validated against.
- Bedrock AI Engine Presets are labels built into the Bedrock AI Engine.
Local AI Engine Tested Labels
The Local AI Engine has been validated against the labels below. The resulting detected entities are designed to map cleanly to downstream masks, and are aligned with HIPAA Safe Harbor identifiers where applicable.
Note: The Local AI Engine may detect more than necessary to minimise the risk of PII leakage. If you do not want to mask certain entities (e.g.
DOCTOR_FIRST_NAME), add them to the ruleset and use thedo_nothingmask.
Specify a tested label directly:
matchers:
ai_detect:
- label: "PATIENT_FIRST_NAME"
For a worked ruleset using these tested labels, see Example 4: HIPAA-Compliant Healthcare Records.
Note:
use_preset: "PATIENT_FIRST_NAME"is also accepted and is equivalent.
Personal identifiers
| Label | Description | Example |
|---|---|---|
FIRST_NAME |
Given name | [Marcus] arrived for his appointment |
MIDDLE_NAME |
Middle name | Sofia [Marie] Hernandez |
LAST_NAME |
Surname | Patient Marcus [Chen] |
GENDER |
Gender (where identifying) | Patient is [female], 34 yo |
NATIONALITY |
Country adjective or ethnicity | Patient is [Korean-American] |
Locations
| Label | Description | Example |
|---|---|---|
ZIPCODE |
Postal/ZIP code | Lives in [98109] |
STREET_ADDRESS |
Street-level address | Mailed to [1247 Magnolia Lane] |
SUBURB |
Suburb name | Resides in [Capitol Hill] |
CITY |
City name | Treated in [Seattle] |
COUNTRY |
Country name | Travelled to [Singapore] |
COUNTY |
County name | Filed in [King County] |
STATE |
State or province | Lives in [California] |
LOCATION |
Catch-all not covered by other location labels | Lives near [Pike Place Market] |
Other
| Label | Description | Example |
|---|---|---|
CREDIT_DEBIT_CARD |
Card numbers, CVVs, partials (expiry dates fall under DATE) |
Charged to [4532-7890-1234-5678] |
DATE |
All dates regardless of type; holidays e.g. "Christmas" are not tagged | In town [24 May]-[28 May] |
EMAIL |
Email address | Contact [m.chen@example.com] |
IP_ADDRESS |
IPv4 or IPv6 address | Logged from [10.42.118.205] |
ORGANIZATION |
Organisation name (sub-departments excluded; e.g. "Tesla Call Center" tags only Tesla) |
Works at [Westbrook General Hospital] |
OTHER_ID |
Catch-all identifier: SSN, MRN, account number, license plate, etc. | MRN [WG-2024-08412] |
PHONE_FAX_NUMBER |
Phone or fax number (the two are not differentiated) | Called [206-555-0142] |
URL |
Web URL | See [https://portal.westbrook.example.com] |
Medical domain
These replace the generic FIRST_NAME / LAST_NAME / MIDDLE_NAME labels in medical contexts.
Note: Combining these with
FIRST_NAME,LAST_NAME, orMIDDLE_NAMEmay lead to unexpected results. If combining, please verify the behaviour you require is the behaviour you're getting.
| Label | Description | Example |
|---|---|---|
DOCTOR_FIRST_NAME |
Doctor's first name | [Eleanor] Whitaker, MD |
DOCTOR_MIDDLE_NAME |
Doctor's middle name | Eleanor [Margaret] Whitaker, MD |
DOCTOR_LAST_NAME |
Doctor's surname | Dr. [Whitaker] reviewed labs |
PATIENT_FIRST_NAME |
Patient's first name | [Marcus] arrived for surgery |
PATIENT_MIDDLE_NAME |
Patient's middle name | Marcus [Wei] Chen |
PATIENT_LAST_NAME |
Patient's surname | Patient Marcus [Chen] |
Custom labels
The Local AI Engine also accepts custom labels. The model decides what to find from the label's name, so name a custom label for what it means:
matchers:
ai_detect:
- label: "EDUCATION_STATUS"
Review the model's output on a sample of your data to confirm it captures the entities you want.
Bedrock AI Engine Presets
The Bedrock AI Engine accepts these names in use_preset:
| Kind | Presets |
|---|---|
| Names | FIRST_NAME, LAST_NAME, FULL_NAME, INITIAL_NAME, TITLE |
| Personal details | AGE, DATE_OF_BIRTH, NATIONALITY, GENDER, RACE |
| Contact details | EMAIL, FAX, PHONE_NUMBER, USERNAME |
| Addresses | STREET_ADDRESS, CITY, COUNTRY, STATE, ZIPCODE |
| Work | JOB_TITLE, ORGANIZATION |
| Health | MEDICAL_CONDITION |
Note: Use
FIRST_NAMEandLAST_NAMEifFULL_NAMEis not behaving as expected.