DataMasque Portal

Unstructured Discovery

Overview

Use DataMasque's Unstructured Discovery page to find which documents on a file connection hold sensitive data, and which entities are in them. An entity is one sensitive value, such as an email address or a card number. A configuration's matchers find entities and give each one a label, such as EMAIL. Navigate to the Unstructured Discovery page, under Discovery, to use this feature.

The page runs a run_file_unstructured_discovery task in the background. An unstructured discovery run reads the text of PDF, DOCX and PPTX files, and of the files whose extension is listed in the configuration's plain_text_extensions. It counts what the matchers of an unstructured discovery configuration find, for each label in each file. DataMasque stores the counts and the paths of the files. It does not store the values it found, or where in a file it found them.

To find the sensitive columns and JSON paths in tabular files instead, use the File Ruleset Generator.

The page is available to Admin and Mask Builder users. Unstructured Discovery requires a license type that includes it (see Unstructured Discovery availability).

Running unstructured discovery

Select a connection from the File Connection dropdown. If you do not have a connection configured yet, you can create one on the File Masking page. Ensure the connection is configured to be used as Source or Source & Destination.

Select a configuration from the Discovery Configuration dropdown. The dropdown lists the unstructured discovery configurations. To create one, click New configuration beside the page title; the editor opens with an example that counts the email addresses in PDF, DOCX, PPTX and .txt files. Click the pencil icon beside the dropdown to edit the selected configuration.

When Git integration is configured and the selected configuration has been synchronised with Git, a Pull config from latest git branch checkbox is shown below the dropdown; see Auto-pull the referenced config at run start.

Click Run discovery to start a run. Once the selected connection has results, the button reads Rerun discovery. When the run finishes, a message links to its run log, and the page shows what the run found.

A configuration with an ai_detect matcher needs an AI Engine. If no AI Engine is configured, DataMasque does not start the run, and the page says why. See AI Engine.

Reading the results

When a connection is selected, the page shows its newest finished unstructured discovery run. If a run is still in progress, the page says so and shows that run when it finishes. Dry runs, failed runs and cancelled runs are not shown. To download the findings of an earlier run, select it on the Run Logs page; see Findings report.

Below the results, a line summarises the run, such as Last Discovery: 01/09/26 10:30:00 · 340 files found · 20 read (7 sensitive, 11 nothing found, 2 with no text to search) · 1 could not be read · 319 skipped · 5.0 MB opened.

Term Meaning
files found Every file the run found, read or not.
read Files the run opened and ran the matchers on.
sensitive Files read that hold at least one entity.
nothing found Files read whose text holds no entity.
with no text to search Files read that are empty, or hold their text only in images. Discovery does not read text in images, so these files may still hold sensitive data.
could not be read Files the run tried to read and could not, such as an encrypted PDF.
skipped Files the run did not try to read, such as the files past files_per_directory.
opened The total size of the files read.

For a run with an ai_detect matcher, the line also names the AI Engine model and version.

To download the exact configuration the run used, as a .zip file, click Download config snapshot, the download icon beside Download findings.

Filtering the results

  • Directory or file name lists the directories whose path, or the name of a file the run tried to read, contains the text.
  • Labels and Categories list only the directories that hold a selected label, or a label in a selected category. Hover over a label to see its description and categories.
  • Reset clears the search and removes every selected label and category.

While a label or category is selected, Entities found and Sensitive files count only the selected labels, or the labels in the selected categories.

Directories

Each row is a directory the run found:

Column Description
Directory The directory's path. Connection root is the directory the connection points at.
Files found How many files the directory holds.
Files read How many of the directory's files the run read.
Sensitive files How many of the directory's files the run found sensitive data in.
Size The total size of the directory's files, including the files the run did not read.
Entities found How many entities the run found for each label. Nothing found when the run found none in the files it read, and No files read when it read none of the directory's files.

Entities found shows the five most frequent labels; hover over and N more labels to see the rest.

Click a column heading other than Entities found to sort by it, and click it again to reverse the order.

Files in a directory

Click a directory to see its files. Files with entities come first, then files with No text to search, then a line such as 12 files with nothing found, which you can click to list those files. Below them, Could not be read lists each file the run could not read, under its reason, and N files were skipped counts the files the run did not try to read, by reason and file extension. The run report lists every file with its reason.

Download findings

Click Download findings for the run's findings, as a CSV file: a row for each label found in each file the run read, and one row with a blank label for each file where nothing was found. A large report downloads as a zip of CSV files, and the button reads Download findings (ZIP). See Findings report for the columns and the size at which a report becomes a zip.

Limits

  • A run reads at most 20 files from each directory, and at most 100,000 files in all. Change this with files_per_directory and files_per_run, or set either to all.
  • A file larger than 10,000,000 bytes is not read. Change this with max_bytes_per_file.
  • Discovery does not read text in images, such as PNG and JPEG files, scanned PDFs, and pictures inside Word and PowerPoint documents.

Troubleshooting

"My file is not listed in its directory"

  • Is the file one the run reads? The run reads PDF, DOCX and PPTX files, and the files whose extension is listed in plain_text_extensions.
  • Did the run select the file? A run reads only some of a directory's files; see Limits. Open the directory and click N files were skipped to see why those files were skipped.

"A directory is not listed"

  • Does the configuration set files.recurse: true? Without it, the run reads only the directory the connection points at.

If none of the above steps help, check the Run Logs and the Run Report for more detailed information about the run.

  • Run Log: navigate to the Run Logs page and locate a run with a matching Source Connection and with Ruleset set to $auto_unstructured_discovery$. The file discovery summary at the end of the run log counts, per file extension, the files that were read and the reasons any files were not.
  • Run Report: see Run Report for how to download it.