Discovery YAML Specification

• Introduction
  • Creating and Editing Configurations
• Git integration and run provenance
  • Auto-pull the referenced config at run start
  • Config Git audit trail
• Top-Level Structure
• Labels
  • Built-in Labels
  • Custom Labels
  • Label Fields
• Metadata Rules
  • Metadata Rule Fields
  • Field Matchers
  • Data Type Constraints
• IDD Settings
• IDD Rules
  • IDD Rule Fields
  • IDD Matcher Types
• Scope
  • Skip
  • Ignore
  • Non-Sensitive
  • Value-Matching Rules
• Document Discovery Settings
• Files
• Config Libraries and Imports
  • Extending the defaults
  • Importing scope
  • Resolution, validation, and the run snapshot
  • Deleting a library in use
• Worked Examples
  • Example 1: Multilingual Keywords
  • Example 2: Country-Specific IDD Pattern (NZ IRD)
  • Example 3: Custom Label for Compliance Reporting

Introduction

A discovery configuration controls how DataMasque discovers sensitive data in your databases, tabular files, and JSON documents. It defines the labels used to classify columns, the rules that match columns by name or data type (metadata rules), the rules that detect sensitive values by inspecting sampled data (In-Data Discovery, or IDD, rules), and scoping controls that include or exclude specific columns.

Discovery works as a two-stage pipeline: first, rules (metadata and IDD) scan the data and assign labels to each column they match; then, separately, the Ruleset Generator reads those labels and produces a ready-to-use masking ruleset. A discovery configuration controls the first stage only — what labels exist and which rules apply them. Which mask each label produces is controlled by a ruleset generation configuration, chosen when you generate a ruleset.

For an overview of database discovery, see Sensitive Data Discovery. For file discovery, see File Ruleset Generator.

DataMasque ships with a comprehensive built-in default configuration. You can create custom configurations to extend or replace the defaults — for example, to add detection rules for region-specific identifiers, multilingual column names, or organisation-specific compliance labels.

Creating and Editing Configurations

Discovery configurations are managed from the Discovery Configurations page in the DataMasque UI:

1. Navigate to Discovery Configurations in the sidebar.
2. Click New configuration and choose Database or File. The editor opens pre-populated with a copy of the current built-in defaults, which you can edit or clear.
3. Give the configuration a name and edit the YAML in the built-in editor.
4. Click Save to validate and store the configuration.

The editor validates your configuration as you type. Structural errors (invalid YAML, missing required fields) are caught by the built-in schema validation. Cross-section errors — such as a rule referencing an undefined label, duplicate names within a section, or a reference to an unknown mask or seed file — are also highlighted live with inline markers on the relevant line. These checks run entirely in-browser with no save required, so you get immediate feedback while editing.

The server performs the same validation when you click Save and remains the authoritative check. If a mask, seed file, or library is added or removed by another user after you opened the editor, the live checks may not reflect the change until you refresh the page.

Once saved, a configuration can be selected when starting a discovery run.

Git integration and run provenance

Discovery configurations can be version-controlled in the same Git repository as your rulesets. Configurations are written under a discovery-configs/ directory in the configured repository path, and config libraries under discovery-config-libraries/, so they live alongside — but separately from — pushed rulesets and ruleset libraries. For first-time setup, see the Git Integration Setup how-to.

Auto-pull the referenced config at run start

When you start a discovery run from a Git-linked configuration, the Configured tab of the run trigger shows a Pull config from latest git branch checkbox. The checkbox appears only when Git integration is configured and the selected configuration has previously been synchronised with Git (a configuration that has never been pushed or pulled is not Git-linked, so the option is hidden).

When enabled:

1. Selecting the option reveals a Branch dropdown, populated from the repository and defaulting to the repository's default branch.
2. Before the run starts, DataMasque pulls the latest version of the configuration from the selected branch and overwrites the stored configuration with the pulled content. The freshly pulled YAML is what gets resolved, validated, and snapshotted for the run.
3. If the pull fails, the run does not start and no partial changes are applied to the stored configuration — the error is surfaced and you can retry.

Warning: Auto-pull overwrites the stored configuration with the content pulled from Git. Any local edits to the configuration that have not been committed to the selected branch are permanently replaced. Only the pull itself is subject to this — a failed pull leaves the stored configuration untouched (see step 3).

This lets scheduled or automated discovery always run against the latest committed configuration.

Config Git audit trail

DataMasque records a Git snapshot on each configuration whenever its content is synchronised with Git (via auto-pull on run start, or a manual push/pull). The snapshot stores four fields on the configuration — branch, commit SHA, repository URL, and synced-at timestamp — and these are cleared if the configuration is later edited outside a pull or push.

These four fields are snapshotted onto every discovery run at run start, independently of any ruleset Git metadata, giving a per-run record of which commit produced the configuration used. Runs started from configurations that were not Git-managed at run time have null config Git metadata.

When a run auto-pulls its configuration, the run log also records the pull as a single entry:

Auto-pulled discovery config "<name>" from "<repo-url>" branch "<branch>" at commit "<short-sha>" (`content changed`|`content unchanged`)

<short-sha> is the first 12 characters of the commit SHA. The content changed / content unchanged suffix indicates whether the pulled content differed from the configuration's previous state.

Top-Level Structure

A discovery configuration YAML file has the following top-level sections. The YAML below shows an example with representative values for each section:

labels:
  # Label definitions for classifying discovered columns
  - name: first_name
    description: "First name of a person"
    categories: ["PII", "PHI", "PCI"]

metadata_rules:
  # Rules that match columns by name or type
  - name: first_name_keywords
    label: first_name
    column:
      type: keywords
      match: any
      words: [first_name, fname, given_name]

idd_settings:
  # Global settings for in-data discovery
  enabled: false
  row_sample_size: 1000

idd_rules:
  # Rules that detect sensitive data by inspecting values
  - name: email_pattern
    label: email
    matcher:
      type: pattern
      pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'

scope:
  # Subtractive scoping rules
  skip:
    - type: glob
      pattern: "public.audit_log.*"
  ignore:
    values:
      - name: Synthetic test data
        matcher: { type: pattern, pattern: '^TEST-' }
  non_sensitive:
    columns:
      - type: exact
        value: public.config.setting_name
    values:
      - name: Placeholder values
        matcher: { type: pattern, pattern: '^N/A$' }

document_discovery:
  # Per-collection document sampling for document databases (MongoDB); ignored for relational databases
  document_sample_size: 1000

files:
  # File-discovery run options (file configs only; invalid for database configs)
  include: ['*.csv']
  recurse: true

Section	Required	Default	Description
labels	Yes	—	Defines the set of labels that metadata rules and IDD rules can assign to discovered columns.
metadata_rules	Yes	—	Rules that match columns by their name, table, schema, or data type.
idd_settings	No	IDD disabled	Global settings for in-data discovery (sampling, thresholds).
idd_rules	Yes	—	Rules that detect sensitive data by inspecting sampled cell values.
scope	No	No exclusions	Subtractive scoping rules: columns to skip, values to ignore, and non-sensitive designations.
document_discovery	No	document_sample_size: 1000	Per-collection document sampling for document-database (e.g. MongoDB) schema discovery. No effect for relational databases; rejected in file discovery configs.
files	No	See Files	File-discovery run options (include/skip patterns, recurse, encoding, workers). Valid only for file discovery configs; rejected in database configs.
imports	No	None	Names of config libraries this config pulls reusable blocks from via $ref. See Config Libraries and Imports.

Labels

Labels classify discovered columns. Every metadata rule and IDD rule must reference a label defined in the labels section.

Note: This file configures the discovery half only — the labels that discovery assigns to columns. How each label is masked is configured separately in a ruleset generation configuration, which is chosen when you generate a ruleset. Label names are case-sensitive and must match exactly between the discovery configuration and the ruleset generation configuration.

Built-in Labels

The built-in default configuration includes labels for common sensitive data types such as first_name, email, social_security_number, and payment_card_number. To see the full list of built-in labels and their descriptions, download the default configuration from the GET /api/discovery/configs/defaults/ endpoint or view it on the Discovery Configurations page in the UI.

Custom Labels

You can define your own labels alongside or instead of the built-in ones. Custom labels behave identically to built-in labels and can be referenced by any metadata rule or IDD rule in the same configuration.

The following example defines a custom tax_file_number label:

labels:
  - name: tax_file_number
    description: "Australian Tax File Number"
    categories: ["PII"]

To choose the mask that a custom label produces during ruleset generation, add an entry for it to a ruleset generation configuration.

Label Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique label name. Must be non-empty.
description	string	No	null	Human-readable description of this label.
categories	list[string]	No	null	Category tags for grouping labels (e.g. PII, PHI, PCI). Used in discovery reports for filtering and compliance.

When description or categories is null (omitted), the label has no description or no sensitive data categories.

Metadata Rules

Metadata rules match columns to labels based on column metadata — the column name, table name, schema name, or data type — without inspecting actual data values. They are evaluated before IDD rules and are the primary mechanism for detecting sensitive columns.

The following example shows a complete metadata rule that detects email columns by matching column names containing both "e" and "mail" as delimited words:

metadata_rules:
  - name: Email address
    label: email
    column:
      type: keywords
      match: all
      words: [e, mail]
    datatypes:
      - type: string
        min_length: 5

Metadata Rule Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique human-readable name for this rule.
label	string	Yes	—	Label to assign when this rule matches. Must reference a label defined in the labels section.
column	Field matcher	No*	null	Matching criteria for the column name. For database discovery, this matches the column name. For tabular file discovery, this matches the column header. For JSON discovery, this matches the last part of the JSON field path.
table	Field matcher	No*	null	Matching criteria for the table name. Only applies to database discovery.
schema_name	Field matcher	No*	null	Matching criteria for the schema name. Only applies to database discovery.
full_path	Field matcher	No*	null	Matching criteria for the full qualified path (schema.table.column).
datatypes	list[datatype]	No	null	Data type constraints. When null, the rule matches columns of any data type; when present, it only matches columns whose data type matches at least one entry.
allow_idd_override	boolean	No	true	Whether IDD rules can override this metadata match.

* At least one of column, table, schema_name, or full_path must be specified.

Tip: Most users should focus on the column field, which matches the database column name, tabular file column name, or JSON field name (the last part of the JSON path). The table and schema_name fields only apply to database discovery and are useful when the same column name appears in different tables with different sensitivity levels.

Field Matchers

The column, table, schema_name, and full_path fields each accept a field matcher that determines how names are matched. There are four matcher types:

keywords

Matches columns whose name contains specific keywords.

column:
  type: keywords
  match: any      # "any" or "all"
  words: [first_name, fname, given_name, nombre]

Field	Type	Required	Description
type	string	Yes	Must be keywords.
match	string	Yes	any — match if any word appears; all — match if all words appear.
words	list[string]	Yes	Words to match against the field name. At least one word is required.

When match is all, words are matched using delimiter-aware matching (respecting word boundaries such as underscores, spaces, and dashes). When match is any, words are matched as simple substrings, which is useful for abbreviations and aliases.

regex

Matches columns whose name matches a regular expression.

column:
  type: regex
  pattern: '^e_?mail'
  case_sensitive: false

Field	Type	Required	Description
type	string	Yes	Must be regex.
pattern	string	Yes	Regular expression pattern (see Common regular expression patterns).
case_sensitive	boolean	No	Whether matching is case-sensitive. Defaults to false (case-insensitive).

exact

Matches columns whose name exactly equals the given value (case-insensitive). Use this when you know the precise column name. A * in the value is treated as a wildcard matching zero or more characters, not a literal *.

column:
  type: exact
  value: email_address

Field	Type	Required	Description
type	string	Yes	Must be exact.
value	string	Yes	Exact column name to match.

glob

Matches columns whose name matches a glob-style pattern (case-insensitive), where * matches zero or more characters. Use this when you want to match a family of column names with a wildcard.

column:
  type: glob
  pattern: "*_name"

Field	Type	Required	Description
type	string	Yes	Must be glob.
pattern	string	Yes	Glob pattern to match. * matches any chars.

Data Type Constraints

Both metadata rules and IDD rules support a datatypes field to restrict matching to columns of specific data types. Each entry specifies a type and optional length constraints:

datatypes:
  - type: string
    min_length: 3
    max_length: 100
  - type: integer

Type	Supports Length Constraints	Description
string	Yes (min_length, max_length)	String/text columns.
integer	Yes (min_length, max_length)	Integer columns. Length refers to digit count.
number	Yes (min_length, max_length)	Numeric columns (float, decimal).
date	No	Date/datetime columns.

IDD Settings

The idd_settings section controls global settings for in-data discovery (IDD). IDD samples actual cell values to detect sensitive data that cannot be identified by column names alone.

Example configuration with IDD enabled:

idd_settings:
  enabled: true
  row_sample_size: 1000
  row_count_method: estimate
  min_threshold: 50
  safe_data_preview:
    enabled: true
    min_group_size: 5
    string_level: patterns
    numeric_level: histograms
    temporal_level: histograms

Field	Type	Required	Default	Description
enabled	boolean	No	false	Whether in-data discovery is enabled. When false, only metadata rules are used.
row_sample_size	integer	No	1000	Number of rows to sample per column for IDD analysis. Must be between 1 and 10,000.
row_count_method	string	No	estimate	How each table's row count is obtained: estimate uses fast statistics-based estimates where the database supports them (exact counts elsewhere); actual always queries the exact row count, which can be expensive on large tables. The count is shown in the Ruleset Generator and the discovery CSV report.
min_threshold	integer	No	null	Default minimum match percentage (0–100) for IDD rules that don't specify their own threshold. When null, a single match triggers detection.
force	boolean	No	false	When true, every metadata-matched column is also IDD-sampled and data evidence can override the metadata label.
safe_data_preview	mapping	No	Enabled	Safe Data Preview settings. Will be enabled if omitted. See Safe Data Preview Settings.

Safe Data Preview Settings

Safe Data Preview summarises the sampled value statistics without exposing row-level data. It is enabled by default. Set enabled: false to turn it off. See Safe Data Preview for the statistics each level reveals.

Field	Type	Required	Default	Description
enabled	boolean	No	true	Whether Safe Data Preview runs. Omitting the block also enables it.
min_group_size	integer	No	5	Smallest group count shown; smaller groups are suppressed. Floored at 5.
string_level	string	No	patterns	Detail shown for string columns: lengths, patterns, or first_chars.
numeric_level	string	No	histograms	Detail shown for numeric columns: summaries or histograms.
temporal_level	string	No	histograms	Detail shown for temporal columns: summaries or histograms.

IDD Rules

IDD rules detect sensitive data by inspecting sampled cell values. Each rule specifies a matcher that tests individual values and an optional prematcher for pre-filtering.

The following example shows a complete IDD rule that detects email addresses by matching cell values against a regex pattern, requiring at least 60% of sampled rows to match:

idd_rules:
  - name: Email address (IDD)
    label: email
    matcher:
      type: pattern
      pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'
    datatypes:
      - type: string
        min_length: 5
    min_threshold: 60

IDD Rule Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique name for this rule.
label	string	Yes	—	Label to assign when this rule matches. Must reference a label in the labels section.
matcher	IDD matcher	Yes	—	Primary matching criteria (see IDD matcher types below).
prematcher	IDD matcher	No	null	Optional pre-filter matcher. When set, only values that pass the prematcher are tested by the main matcher; when null, every value is tested.
datatypes	list[datatype]	No	null	Data type constraints. When null, the rule applies to columns of any data type; when present, it only applies to columns whose data type matches at least one entry.
min_threshold	integer	No	Inherited	Minimum match percentage (0–100) to trigger detection. See Threshold inheritance below.
test_words	boolean	No	false	If true, split each cell value on whitespace and test each word individually.
min_word_count	integer	No	null	Minimum word count per cell value. Values with fewer words are skipped. When null, no minimum is applied.
max_word_count	integer	No	null	Maximum word count per cell value. Values with more words are skipped. When null, no maximum is applied.

Threshold inheritance

min_threshold sets the minimum percentage of sampled values that must match before a column is flagged. When set on a rule, it overrides idd_settings.min_threshold for that rule; when null, the rule inherits idd_settings.min_threshold. When both the rule's min_threshold and idd_settings.min_threshold are null, any single matching row or file field counts as detection.

IDD Matcher Types

pattern

Matches cell values against a regular expression.

matcher:
  type: pattern
  pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'
  case_sensitive: false

Field	Type	Required	Default	Description
type	string	Yes	—	Must be pattern.
pattern	string	Yes	—	Regex pattern to match values.
must_not_contain	string	No	null	Regex pattern that values must NOT match. Values matching this are excluded.
case_sensitive	boolean	No	true	Whether the pattern match is case-sensitive.

checksum

Validates cell values against a checksum algorithm.

matcher:
  type: checksum
  algorithm: nz_ird
  min_digits: 8
  max_digits: 9
  ignore_non_digit: true

Field	Type	Required	Default	Description
type	string	Yes	—	Must be checksum.
algorithm	string	Yes	—	Checksum algorithm to use. See below.
min_digits	integer	Yes	—	Minimum number of digits for a valid value.
max_digits	integer	Yes	—	Maximum number of digits for a valid value.
ignore_regex	string	No	null	Regex pattern of characters to strip before checksum validation.
ignore_non_digit	boolean	No	true	Whether to strip all non-digit characters before validation.

Available checksum algorithms:

Algorithm	Description
luhn	Luhn algorithm (credit cards, IMEI, etc.)
brazilian_cpf	Brazilian CPF number validation
australian_business_number	Australian Business Number (ABN)
australian_company_number	Australian Company Number (ACN)
icp	ICP validation
weighted_dual_checksum	Weighted dual checksum
vin	Vehicle Identification Number
nz_ird	New Zealand IRD number

credit_card

Matches credit card numbers using Luhn validation combined with issuer-prefix verification.

matcher:
  type: credit_card
  min_digits: 13
  max_digits: 19
  ignore_non_digit: true

Field	Type	Required	Default	Description
type	string	Yes	—	Must be credit_card.
min_digits	integer	Yes	—	Minimum number of digits.
max_digits	integer	Yes	—	Maximum number of digits.
ignore_regex	string	No	null	Regex pattern of characters to strip before validation.
ignore_non_digit	boolean	No	true	Whether to strip all non-digit characters before validation.

ssn

Matches US Social Security Numbers.

matcher:
  type: ssn

Field	Type	Required	Description
type	string	Yes	Must be ssn.

No additional fields are required. The matcher validates the SSN format and area/group number rules.

seed_file

Matches cell values against values loaded from seed CSV files. You can reference both built-in seed files (shipped with DataMasque) and customer-uploaded seed files.

Built-in seed files are selected automatically based on the configured locality. For example, if the locality is set to AU, seed files containing Australian data are preferred where they exist.

To use a custom seed file:

1. Navigate to Files > Seeds in the DataMasque UI.
2. Upload your CSV file (must be a valid CSV with a header row).
3. Reference the filename and column in your IDD rule:

idd_rules:
  - name: custom_first_name_idd
    label: first_name
    matcher:
      type: seed_file
      files:
        - filename: custom_first_names.csv
          column: name
      case_sensitive: false
    min_threshold: 50

Field	Type	Required	Default	Description
type	string	Yes	—	Must be seed_file.
files	list[object]	Yes	—	Seed files and columns to match against. At least one entry required.
files[].filename	string	Yes	—	Filename of the uploaded seed CSV.
files[].column	string	Yes	—	Column name within the seed file to use for matching.
case_sensitive	boolean	No	false	Whether the value lookup is case-sensitive.

Scope

The scope section controls which columns and values are excluded from discovery or marked as non-sensitive. It is organised into three sub-sections by effect.

The following example shows all three sub-sections:

scope:
  skip:
    - type: glob
      pattern: "public.audit_*.*"

  ignore:
    values:
      - name: Synthetic test data
        matcher: { type: pattern, pattern: '^TEST-' }

  non_sensitive:
    columns:  # Use "locators" for file discovery configs.
      - type: exact
        value: public.config.setting_name
    values:
      - name: Placeholder values
        matcher: { type: pattern, pattern: '^N/A$' }

Skip

Columns matching any skip pattern are excluded from discovery entirely. They are not scanned, not IDD-sampled, and do not appear in results.

For database discovery, patterns match against the dot-separated schema.table.column path. All database scope patterns must be fully schema-qualified: exact requires a complete schema.table.column value, and regex/glob must include a schema prefix (for example ^schema\.table\. or schema.table.*).

For file discovery, patterns match against forward-slash-separated locator strings:

• CSV / tabular files: the column name (e.g. name, email)
• JSON files: the full path with forward slashes (e.g. users/0/email, root/nested/field)
• Parquet complex fields: the column name followed by the sub-field path (e.g. companies/company_foo/staff/*/name)

All pattern types (exact, glob, regex) work against these locator strings. Glob wildcards can match across path segments (e.g. users/*/email).

The following example skips columns using all three pattern types:

scope:
  skip:
    - type: exact
      value: public.migrations.id

    - type: regex
      pattern: '^public\.django_.*\.'
      case_sensitive: false

    - type: glob
      pattern: "public.audit_*.*"

Three pattern types are supported:

Field	Type	Required	Default	Description
type	string	Yes	—	exact, regex, or glob.
value	string	Yes (exact)	—	The exact path to match. Only used with type: exact.
pattern	string	Yes (regex/glob)	—	Pattern to match against the path. Used with regex or glob.
case_sensitive	boolean	No	false	Whether matching is case-sensitive. Applies only to regex and glob.

The case_sensitive flag does not apply to exact patterns: for file locators, exact always matches case-sensitively, and for database columns it follows the database's own identifier case rules.

Ignore

Values matching any ignore rule are scanned but dropped from discovery results. Because an ignore match hides the column entirely, these rules always require 100% of sampled values to match — min_threshold is fixed at 100 and cannot be overridden. The ignore section contains a values list of value-matching rules.

The following example drops any values starting with TEST-:

scope:
  ignore:
    values:
      - name: Synthetic test data
        matcher:
          type: pattern
          pattern: '^TEST-'

Each entry in values is a value-matching rule.

Non-Sensitive

The non_sensitive section marks columns or values as explicitly non-sensitive. They appear in discovery results with a non-sensitive designation.

It has two sub-fields:

• columns — column-level patterns (same pattern types as skip). Matched columns are marked non-sensitive regardless of metadata or IDD rules.
• values — value-level rules (same shape as value-matching rules). Matched values are reported as non-sensitive.

Note: File discovery configurations use locators instead of columns. Locator patterns match against the same forward-slash-separated strings described in Skip.

The following example marks a specific column and a value pattern as non-sensitive:

scope:
  non_sensitive:
    columns:  # Use "locators" for file discovery configs.
      - type: exact
        value: public.config.setting_name
      - type: glob
        pattern: "public.system_*.*"
    values:
      - name: Placeholder values
        matcher:
          type: pattern
          pattern: '^N/A$'

Value-Matching Rules

Value rules used inside scope.ignore.values and scope.non_sensitive.values have the same shape as IDD rules but without a label field.

Field	Type	Required	Default	Description
name	string	Yes	—	Unique name for this rule.
matcher	IDD matcher	Yes	—	Primary matching criteria (same IDD matcher types as IDD rules).
prematcher	IDD matcher	No	null	Optional pre-filter matcher.
datatypes	list[datatype]	No	null	Data type constraints.
min_threshold	integer	No	Inherited	Minimum match percentage (0–100). Fixed at 100 for ignore rules; configurable for non-sensitive rules.
test_words	boolean	No	false	If true, split each cell value on whitespace and test each word individually.
min_word_count	integer	No	null	Minimum word count per cell value. Values with fewer words are skipped. When null, no minimum is applied.
max_word_count	integer	No	null	Maximum word count per cell value. Values with more words are skipped. When null, no maximum is applied.

Document Discovery Settings

The document_discovery section controls how the top-level schema of a document database (for example MongoDB) is inferred. Document databases have no fixed schema, so DataMasque samples documents from each collection and unions their top-level field names to build the column list.

This section applies only to document databases. It is accepted but has no effect for relational databases, and a file discovery configuration rejects it.

document_sample_size sets how many documents are sampled per collection. Sampling more than 5% of a collection forces a slower server-side sort, so DataMasque honours the value but logs a warning when it is exceeded.

Example raising the sample size for large collections:

document_discovery:
  document_sample_size: 5000

Field	Type	Required	Default	Description
document_sample_size	integer	No	1000	Number of documents to sample per collection when inferring its top-level schema. Must be at least 1.

Files

The files section contains file-discovery run options. It is only available in file discovery configurations. Including a files section in a database discovery configuration will produce a validation error.

All fields are optional. Unset fields fall back to the built-in defaults: scan all files, no recursing into subdirectories, UTF-8 encoding, one worker.

Example scanning CSV and JSON files recursively, skipping any archive directory:

files:
  include:
    - '*.csv'
    - '*.json'
  skip:
    - '**/archive/**'
  recurse: true
  encoding: utf-8
  workers: 4

Field	Type	Required	Default	Description
include	list[string]	No	All files	Glob patterns selecting which files to scan.
skip	list[string]	No	None	Glob patterns excluding files from the scan.
recurse	boolean	No	false	Whether to descend into subdirectories when looking for files to discover.
encoding	string	No	utf-8	Text encoding used to read files.
workers	integer	No	1	Number of parallel file-discovery workers (≥ 1).

Config Libraries and Imports

A config library is a named, reusable bag of discovery blocks (labels, metadata rules, and IDD rules) that configs share via references, and it may also carry per-type scope (database_scope and file_scope). Libraries are managed under Config Libraries in the sidebar. A library is not tied to a configuration type: the same library can be imported by database and file configurations alike.

Metadata rules that match on table, schema_name, or full_path are database-oriented. Imported into a file configuration, rules with table or schema_name never match (the run log warns when this happens), and a full_path pattern written for one locator family — schema.table.column for databases, /-separated paths for JSON files — will not match the other. Libraries meant to be shared across configuration types should prefer column-only metadata rules.

A config declares the libraries it draws from in the top-level imports list, then pulls blocks in with a $ref. A reference has the form library_name#path/within/the/library, where the path is a /-delimited list of keys and array indices:

imports:
  - common/pii          # a library named "pii" in the "common" namespace
labels:
  - $ref: "common/pii#labels/email"            # pull one block
metadata_rules:
  $ref: "common/pii#metadata_rules"            # pull a whole section
idd_rules: []

Sibling keys placed next to a $ref override the referenced content (a shallow merge), so you can reuse a block and tweak a field:

labels:
  - $ref: "common/pii#labels/email"   # reuse the library's email label...
    categories: ["PII", "PHI"]         # ...but override just its categories

Extending the defaults

The built-in defaults are available under the reserved name datamasque-default-config. To start from the full defaults and customise, import that name and $ref the sections you want:

imports:
  - datamasque-default-config
labels:         { $ref: "datamasque-default-config#labels" }
metadata_rules: { $ref: "datamasque-default-config#metadata_rules" }
idd_rules:      { $ref: "datamasque-default-config#idd_rules" }

Importing scope

Scope differs by configuration type: a database scope's patterns are schema-qualified schema.table.column paths, while a file scope's are free-form locators (see Scope). A library therefore carries scope as two independent, optional blocks, database_scope and file_scope, and may hold one, both, or neither, so a single library can serve database and file configurations alike.

A configuration imports the block matching its own type:

# In a database discovery config
scope:
  $ref: "common/pii#database_scope"

# In a file discovery config
scope:
  $ref: "common/pii#file_scope"

As with any $ref, sibling keys placed next to it override part of the imported block:

scope:
  $ref: "common/pii#database_scope"
  skip:                      # override just the skip patterns
    - type: glob
      pattern: "public.audit_*.*"

Importing the block for the other configuration type is rejected: a file config may not reference #database_scope, and a database config may not reference #file_scope. The error names the block to use instead. Such a configuration still saves, but is marked invalid and cannot start a run until the reference is corrected.

Resolution, validation, and the run snapshot

DataMasque resolves references when a discovery run starts: each run stores a snapshot of the fully expanded config, with every $ref inlined, so a run is reproducible even if a library later changes. It validates the expanded config against the importing configuration's own schema and rejects any block that does not fit the importing config.

At edit time a config that imports libraries is checked for structure only. These surface as an error only when you start a run:

• a referenced library is missing or has been deleted
• a referenced library is invalid
• a $ref points at a path that does not exist

Deleting a library in use

A library that is imported by active configs cannot be deleted without confirmation: the delete is rejected and lists the dependent configs. In the UI, deleting a library that is still imported opens a Cannot delete config library dialog that lists the dependent configurations; confirming with Force Delete deletes it anyway. Force-deleting a library that is in use marks the configs that import it invalid until they are pointed at another source; fixing or restoring the library re-validates them automatically.

Worked Examples

Each example below is a complete, self-contained configuration that can be copied and pasted into the YAML editor.

Example 1: Multilingual Keywords

Add nombre (Spanish) and vorname (German) as metadata rules for first name detection. This extends the default detection to recognise non-English column names.

labels:
  - name: first_name
    description: "First name of a person"
    categories: ["PII", "PHI", "PCI"]

metadata_rules:
  # Extend first_name detection with Spanish and German keywords
  - name: first_name_multilingual
    label: first_name
    column:
      type: keywords
      match: any
      words: [nombre, vorname, prenom, given_name, primeiro_nome]

idd_rules: []

This rule uses match: any so that a column named vorname, nombre, or prenom will all be detected as first_name. You can add this rule alongside the existing English-language rules to support multilingual databases.

Example 2: Country-Specific IDD Pattern (NZ IRD)

Detect New Zealand IRD (Inland Revenue Department) numbers using the nz_ird checksum algorithm. NZ IRD numbers are 8 or 9 digits with a weighted check digit.

labels:
  - name: ird_number
    description: "IRD number (New Zealand)"
    categories: ["PII"]

metadata_rules: []

idd_settings:
  enabled: true
  row_sample_size: 1000

idd_rules:
  - name: nz_ird_checksum
    label: ird_number
    matcher:
      type: checksum
      algorithm: nz_ird
      min_digits: 8
      max_digits: 9
      ignore_non_digit: true
    datatypes:
      - type: string
        min_length: 8
        max_length: 12
      - type: integer
        min_length: 8
        max_length: 9
    min_threshold: 60

This rule:

• Uses the nz_ird checksum algorithm to validate the check digit.
• Strips non-digit characters (hyphens, spaces) before validation.
• Restricts matching to string columns of 8–12 characters or integer columns of 8–9 digits.
• Requires at least 60% of sampled values to pass validation before flagging the column.

Example 3: Custom Label for Compliance Reporting

Define a custom label for internal audit tracking. This label categorises columns for compliance reporting.

labels:
  - name: gdpr_data_subject_id
    description: "GDPR data subject identifier for audit tracking"
    categories: ["GDPR", "Compliance"]

metadata_rules:
  - name: gdpr_subject_id
    label: gdpr_data_subject_id
    column:
      type: keywords
      match: any
      words: [data_subject_id, subject_identifier, gdpr_id]

idd_rules: []

Key points:

• Custom categories (GDPR, Compliance) appear in discovery reports and can be used to filter results.
• To track these columns for compliance without masking them, set skip_generation: true for the gdpr_data_subject_id label in your ruleset generation configuration. Discovery will assign the label to matching columns, but no masking rule will be generated for them.