Discovery YAML Specification

• Introduction
• Top-Level Structure
• Labels
  • Built-in Labels
  • Custom Labels
  • Label Fields
  • Built-in Mask Types
• Metadata Rules
  • Metadata Rule Fields
  • Field Matchers
  • Data Type Constraints
• IDD Settings
• IDD Rules
  • IDD Rule Fields
  • IDD Matcher Types
• Scope
  • Skip
  • Ignore
  • Non-Sensitive
  • Value-Matching Rules
• Files
• Worked Examples
  • Example 1: Multilingual Keywords
  • Example 2: Country-Specific IDD Pattern (NZ IRD)
  • Example 3: Custom Label for Compliance Reporting

Introduction

A discovery configuration controls how DataMasque discovers sensitive data in your databases, tabular files, and JSON documents. It defines the labels used to classify columns, the rules that match columns by name or data type (metadata rules), the rules that detect sensitive values by inspecting sampled data (In-Data Discovery, or IDD, rules), and scoping controls that include or exclude specific columns.

Discovery works as a two-stage pipeline: first, rules (metadata and IDD) scan the data and assign labels to each column they match; then the Ruleset Generator reads those labels and uses each label's mask field to produce a ready-to-use masking ruleset. This configuration controls the first stage — what labels exist and which rules apply them.

For an overview of database discovery, see Sensitive Data Discovery. For file discovery, see File Ruleset Generator.

DataMasque ships with a comprehensive built-in default configuration. You can create custom configurations to extend or replace the defaults — for example, to add detection rules for region-specific identifiers, multilingual column names, or organisation-specific compliance labels.

Creating and Editing Configurations

Discovery configurations are managed from the Discovery Configurations page in the DataMasque UI:

1. Navigate to Discovery Configurations in the sidebar.
2. Click Add Configuration to create a new blank configuration, or New from defaults to start with a copy of the current built-in defaults.
3. Give the configuration a name and edit the YAML in the built-in editor.
4. Click Save to validate and store the configuration.

Once saved, a configuration can be selected when starting a discovery run.

Top-Level Structure

A discovery configuration YAML file has the following top-level sections. The YAML below shows an example with representative values for each section:

labels:
  # Label definitions for classifying discovered columns
  - name: first_name
    description: "First name of a person"
    categories: ["PII", "PHI", "PCI"]
    mask: first_name_mask

metadata_rules:
  # Rules that match columns by name or type
  - name: first_name_keywords
    label: first_name
    column:
      type: keywords
      match: any
      words: [first_name, fname, given_name]

idd_settings:
  # Global settings for in-data discovery
  enabled: false
  row_sample_size: 1000

idd_rules:
  # Rules that detect sensitive data by inspecting values
  - name: email_pattern
    label: email
    matcher:
      type: pattern
      pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'

scope:
  # Subtractive scoping rules
  skip:
    - type: glob
      pattern: "public.audit_log.*"
  ignore:
    values:
      - name: Synthetic test data
        matcher: { type: pattern, pattern: '^TEST-' }
  non_sensitive:
    columns:
      - type: exact
        value: public.config.setting_name
    values:
      - name: Placeholder values
        matcher: { type: pattern, pattern: '^N/A$' }

document_discovery:
  # Per-collection document sampling for document databases (MongoDB); ignored for relational databases
  document_sample_size: 1000

files:
  # File-discovery run options (file configs only; invalid for database configs)
  include: ['*.csv']
  recurse: true

Section	Required	Default	Description
labels	Yes	—	Defines the set of labels that metadata rules and IDD rules can assign to discovered columns.
metadata_rules	Yes	—	Rules that match columns by their name, table, schema, or data type.
idd_settings	No	IDD disabled	Global settings for in-data discovery (sampling, thresholds).
idd_rules	Yes	—	Rules that detect sensitive data by inspecting sampled cell values.
scope	No	No exclusions	Subtractive scoping rules: columns to skip, values to ignore, and non-sensitive designations.
document_discovery	No	document_sample_size: 1000	Per-collection document sampling for document-database (e.g. MongoDB) schema discovery. No effect for relational databases; rejected in file discovery configs.
files	No	See Files	File-discovery run options (include/skip patterns, recurse, encoding, workers). Valid only for file discovery configs; rejected in database configs.

Labels

Labels classify discovered columns and optionally reference masking rules to apply when the label is detected. Every metadata rule and IDD rule must reference a label defined in the labels section.

Built-in Labels

The built-in default configuration includes labels for common sensitive data types such as first_name, email, social_security_number, and payment_card_number. To see the full list of built-in labels and their descriptions, download the default configuration from the GET /api/discovery/configs/defaults/ endpoint or view it on the Discovery Configurations page in the UI.

Custom Labels

You can define your own labels alongside or instead of the built-in ones. Custom labels behave identically to built-in labels and can be referenced by any metadata rule or IDD rule in the same configuration.

The following example defines a custom tax_file_number label that applies an imitate mask to any columns that match it:

labels:
  - name: tax_file_number
    description: "Australian Tax File Number"
    categories: ["PII"]
    mask: imitate_mask

Label Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique label name. Must be non-empty.
description	string	No	null	Human-readable description of this label.
categories	list[string]	No	null	Category tags for grouping labels (e.g. PII, PHI, PCI). Used in discovery reports for filtering and compliance.
mask	string	No	null	Reference to a built-in mask type (see Built-in Mask Types). Inline mask definitions are not supported.
skip_generation	boolean	No	false	When true, the label is used for discovery and reporting but no masking rule is auto-generated. Useful for audit-only labels.

When description or categories is null (omitted), the label has no description or no sensitive data categories. When mask is null, the label is still discovered and reported, but no masking rule is generated for it during ruleset generation — the same outcome as skip_generation: true.

Built-in Mask Types

The mask field on a label must reference one of the following built-in mask types. These determine how discovered columns are masked during ruleset generation.

Mask reference	Masking behaviour
first_name_mask	Random first name from a seed file.
last_name_mask	Random last name from a seed file.
full_name_mask	Random first name + last name.
realistic_email_mask	Realistic email in firstname.lastname@domain format.
imitate_mask	Imitates the character pattern of the original value.
date_of_birth_mask	Date with the original age retained.
social_security_number_mask	Random SSN-format number.
country_mask	Random country name from a seed file.
country_abbreviation_mask	Random two-letter country code (ISO 3166 alpha-2).
state_mask	Random state or province name from a seed file.
us_state_mask	Random US state name.
us_state_short_mask	Random US state abbreviation.
city_mask	Random city name from a seed file.
county_mask	Random county name from a seed file.
post_code_mask	Random postal code from a seed file.
street_type_mask	Random street type (Avenue, Road, etc.).
street_type_abbreviation_mask	Random street type abbreviation (Ave, Rd, etc.).
address_mask	Full address composed from seed file fields. Format varies by locality.
street_address_mask	Street address from a seed file.
suburb_mask	Suburb or neighbourhood name from a seed file.
gender_mask	Fixed value REDACTED.
age_mask	Random number between 1 and 99.
login_mask	Random alphanumeric string (8–16 characters).
mac_address_mask	Imitates the original MAC address format.
ip_address_mask	Random IPv4 address.
job_position_mask	Random job title from a seed file.
company_mask	Random company name from a seed file.
nric_mask	Random Singapore NRIC-format string.
luhn_checksum_mask	Imitates the original value with a valid Luhn checksum.
payment_card_number_mask	Random credit card number with a valid checksum.
brazil_cpf_mask	Random Brazilian CPF number.
expiry_date_mask	Random card expiry date in MM-YYYY format.

Masks that draw values from a seed file (e.g. first_name_mask, address_mask, city_mask) select the seed file based on the configured locality. For example, if the locality is set to AU, the generated ruleset will prefer Australian seed files where they exist.

Metadata Rules

Metadata rules match columns to labels based on column metadata — the column name, table name, schema name, or data type — without inspecting actual data values. They are evaluated before IDD rules and are the primary mechanism for detecting sensitive columns.

The following example shows a complete metadata rule that detects email columns by matching column names containing both "e" and "mail" as delimited words:

metadata_rules:
  - name: Email address
    label: email
    column:
      type: keywords
      match: all
      words: [e, mail]
    datatypes:
      - type: string
        min_length: 5

Metadata Rule Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique human-readable name for this rule.
label	string	Yes	—	Label to assign when this rule matches. Must reference a label defined in the labels section.
column	Field matcher	No*	null	Matching criteria for the column name. For database discovery, this matches the column name. For tabular file discovery, this matches the column header. For JSON discovery, this matches the last part of the JSON field path.
table	Field matcher	No*	null	Matching criteria for the table name. Only applies to database discovery.
schema_name	Field matcher	No*	null	Matching criteria for the schema name. Only applies to database discovery.
full_path	Field matcher	No*	null	Matching criteria for the full qualified path (schema.table.column).
datatypes	list[datatype]	No	null	Data type constraints. When null, the rule matches columns of any data type; when present, it only matches columns whose data type matches at least one entry.
allow_idd_override	boolean	No	true	Whether IDD rules can override this metadata match.

* At least one of column, table, schema_name, or full_path must be specified.

Tip: Most users should focus on the column field, which matches the database column name, tabular file column name, or JSON field name (the last part of the JSON path). The table and schema_name fields only apply to database discovery and are useful when the same column name appears in different tables with different sensitivity levels.

Field Matchers

The column, table, schema_name, and full_path fields each accept a field matcher that determines how names are matched. There are four matcher types:

keywords

Matches columns whose name contains specific keywords.

column:
  type: keywords
  match: any      # "any" or "all"
  words: [first_name, fname, given_name, nombre]

Field	Type	Required	Description
type	string	Yes	Must be keywords.
match	string	Yes	any — match if any word appears; all — match if all words appear.
words	list[string]	Yes	Words to match against the field name. At least one word is required.

When match is all, words are matched using delimiter-aware matching (respecting word boundaries such as underscores, spaces, and dashes). When match is any, words are matched as simple substrings, which is useful for abbreviations and aliases.

regex

Matches columns whose name matches a regular expression.

column:
  type: regex
  pattern: '^e_?mail'
  case_sensitive: false

Field	Type	Required	Description
type	string	Yes	Must be regex.
pattern	string	Yes	Regular expression pattern (see Common regular expression patterns).
case_sensitive	boolean	No	Whether matching is case-sensitive. Defaults to false (case-insensitive).

exact

Matches columns whose name exactly equals the given value (case-insensitive). Use this when you know the precise column name. A * in the value is treated as a wildcard matching zero or more characters, not a literal *.

column:
  type: exact
  value: email_address

Field	Type	Required	Description
type	string	Yes	Must be exact.
value	string	Yes	Exact column name to match.

glob

Matches columns whose name matches a glob-style pattern (case-insensitive), where * matches zero or more characters. Use this when you want to match a family of column names with a wildcard.

column:
  type: glob
  pattern: "*_name"

Field	Type	Required	Description
type	string	Yes	Must be glob.
pattern	string	Yes	Glob pattern to match. * matches any chars.

Data Type Constraints

Both metadata rules and IDD rules support a datatypes field to restrict matching to columns of specific data types. Each entry specifies a type and optional length constraints:

datatypes:
  - type: string
    min_length: 3
    max_length: 100
  - type: integer

Type	Supports Length Constraints	Description
string	Yes (min_length, max_length)	String/text columns.
integer	Yes (min_length, max_length)	Integer columns. Length refers to digit count.
number	Yes (min_length, max_length)	Numeric columns (float, decimal).
date	No	Date/datetime columns.

IDD Settings

The idd_settings section controls global settings for in-data discovery (IDD). IDD samples actual cell values to detect sensitive data that cannot be identified by column names alone.

Example configuration with IDD enabled:

idd_settings:
  enabled: true
  row_sample_size: 1000
  min_threshold: 50

Field	Type	Required	Default	Description
enabled	boolean	No	false	Whether in-data discovery is enabled. When false, only metadata rules are used.
row_sample_size	integer	No	1000	Number of rows to sample per column for IDD analysis. Must be at least 1.
min_threshold	integer	No	null	Default minimum match percentage (0–100) for IDD rules that don't specify their own threshold. When null, a single match triggers detection.
force	boolean	No	false	When true, every metadata-matched column is also IDD-sampled and data evidence can override the metadata label.

IDD Rules

IDD rules detect sensitive data by inspecting sampled cell values. Each rule specifies a matcher that tests individual values and an optional prematcher for pre-filtering.

The following example shows a complete IDD rule that detects email addresses by matching cell values against a regex pattern, requiring at least 60% of sampled rows to match:

idd_rules:
  - name: Email address (IDD)
    label: email
    matcher:
      type: pattern
      pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'
    datatypes:
      - type: string
        min_length: 5
    min_threshold: 60

IDD Rule Fields

Field	Type	Required	Default	Description
name	string	Yes	—	Unique name for this rule.
label	string	Yes	—	Label to assign when this rule matches. Must reference a label in the labels section.
matcher	IDD matcher	Yes	—	Primary matching criteria (see IDD matcher types below).
prematcher	IDD matcher	No	null	Optional pre-filter matcher. When set, only values that pass the prematcher are tested by the main matcher; when null, every value is tested.
datatypes	list[datatype]	No	null	Data type constraints. When null, the rule applies to columns of any data type; when present, it only applies to columns whose data type matches at least one entry.
min_threshold	integer	No	Inherited	Minimum match percentage (0–100) to trigger detection. See Threshold inheritance below.
test_words	boolean	No	false	If true, split each cell value on whitespace and test each word individually.
min_word_count	integer	No	null	Minimum word count per cell value. Values with fewer words are skipped. When null, no minimum is applied.
max_word_count	integer	No	null	Maximum word count per cell value. Values with more words are skipped. When null, no maximum is applied.

Threshold inheritance

min_threshold sets the minimum percentage of sampled values that must match before a column is flagged. When set on a rule, it overrides idd_settings.min_threshold for that rule; when null, the rule inherits idd_settings.min_threshold. When both the rule's min_threshold and idd_settings.min_threshold are null, any single matching row or file field counts as detection.

IDD Matcher Types

pattern

Matches cell values against a regular expression.

matcher:
  type: pattern
  pattern: '^[\w.+-]+@[\w-]+\.[\w.]+$'
  case_sensitive: false

Field	Type	Required	Default	Description
type	string	Yes	—	Must be pattern.
pattern	string	Yes	—	Regex pattern to match values.
must_not_contain	string	No	null	Regex pattern that values must NOT match. Values matching this are excluded.
case_sensitive	boolean	No	true	Whether the pattern match is case-sensitive.

checksum

Validates cell values against a checksum algorithm.

matcher:
  type: checksum
  algorithm: nz_ird
  min_digits: 8
  max_digits: 9
  ignore_non_digit: true

Field	Type	Required	Default	Description
type	string	Yes	—	Must be checksum.
algorithm	string	Yes	—	Checksum algorithm to use. See below.
min_digits	integer	Yes	—	Minimum number of digits for a valid value.
max_digits	integer	Yes	—	Maximum number of digits for a valid value.
ignore_regex	string	No	null	Regex pattern of characters to strip before checksum validation.
ignore_non_digit	boolean	No	true	Whether to strip all non-digit characters before validation.

Available checksum algorithms:

Algorithm	Description
luhn	Luhn algorithm (credit cards, IMEI, etc.)
brazilian_cpf	Brazilian CPF number validation
australian_business_number	Australian Business Number (ABN)
australian_company_number	Australian Company Number (ACN)
icp	ICP validation
weighted_dual_checksum	Weighted dual checksum
vin	Vehicle Identification Number
nz_ird	New Zealand IRD number

credit_card

Matches credit card numbers using Luhn validation combined with issuer-prefix verification.

matcher:
  type: credit_card
  min_digits: 13
  max_digits: 19
  ignore_non_digit: true

Field	Type	Required	Default	Description
type	string	Yes	—	Must be credit_card.
min_digits	integer	Yes	—	Minimum number of digits.
max_digits	integer	Yes	—	Maximum number of digits.
ignore_regex	string	No	null	Regex pattern of characters to strip before validation.
ignore_non_digit	boolean	No	true	Whether to strip all non-digit characters before validation.

ssn

Matches US Social Security Numbers.

matcher:
  type: ssn

Field	Type	Required	Description
type	string	Yes	Must be ssn.

No additional fields are required. The matcher validates the SSN format and area/group number rules.

seed_file

Matches cell values against values loaded from seed CSV files. You can reference both built-in seed files (shipped with DataMasque) and customer-uploaded seed files.

Built-in seed files are selected automatically based on the configured locality. For example, if the locality is set to AU, seed files containing Australian data are preferred where they exist.

To use a custom seed file:

1. Navigate to Files > Seeds in the DataMasque UI.
2. Upload your CSV file (must be a valid CSV with a header row).
3. Reference the filename and column in your IDD rule:

idd_rules:
  - name: custom_first_name_idd
    label: first_name
    matcher:
      type: seed_file
      files:
        - filename: custom_first_names.csv
          column: name
      case_sensitive: false
    min_threshold: 50

Field	Type	Required	Default	Description
type	string	Yes	—	Must be seed_file.
files	list[object]	Yes	—	Seed files and columns to match against. At least one entry required.
files[].filename	string	Yes	—	Filename of the uploaded seed CSV.
files[].column	string	Yes	—	Column name within the seed file to use for matching.
case_sensitive	boolean	No	false	Whether the value lookup is case-sensitive.

Scope

The scope section controls which columns and values are excluded from discovery or marked as non-sensitive. It is organised into three sub-sections by effect.

The following example shows all three sub-sections:

scope:
  skip:
    - type: glob
      pattern: "public.audit_*.*"

  ignore:
    values:
      - name: Synthetic test data
        matcher: { type: pattern, pattern: '^TEST-' }

  non_sensitive:
    columns:  # Use "locators" for file discovery configs.
      - type: exact
        value: public.config.setting_name
    values:
      - name: Placeholder values
        matcher: { type: pattern, pattern: '^N/A$' }

Skip

Columns matching any skip pattern are excluded from discovery entirely. They are not scanned, not IDD-sampled, and do not appear in results.

For database discovery, patterns match against the dot-separated schema.table.column path. All database scope patterns must be fully schema-qualified: exact requires a complete schema.table.column value, and regex/glob must include a schema prefix (for example ^schema\.table\. or schema.table.*).

For file discovery, patterns match against forward-slash-separated locator strings:

• CSV / tabular files: the column name (e.g. name, email)
• JSON files: the full path with forward slashes (e.g. users/0/email, root/nested/field)
• Parquet complex fields: the column name followed by the sub-field path (e.g. companies/company_foo/staff/*/name)

All pattern types (exact, glob, regex) work against these locator strings. Glob wildcards can match across path segments (e.g. users/*/email).

The following example skips columns using all three pattern types:

scope:
  skip:
    - type: exact
      value: public.migrations.id

    - type: regex
      pattern: '^public\.django_.*\.'
      case_sensitive: false

    - type: glob
      pattern: "public.audit_*.*"

Three pattern types are supported:

Field	Type	Required	Default	Description
type	string	Yes	—	exact, regex, or glob.
value	string	Yes (exact)	—	The exact path to match. Only used with type: exact.
pattern	string	Yes (regex/glob)	—	Pattern to match against the path. Used with regex or glob.
case_sensitive	boolean	No	false	Whether matching is case-sensitive. Applies only to regex and glob.

The case_sensitive flag does not apply to exact patterns: for file locators, exact always matches case-sensitively, and for database columns it follows the database's own identifier case rules.

Ignore

Values matching any ignore rule are scanned but dropped from discovery results. Because an ignore match hides the column entirely, these rules always require 100% of sampled values to match — min_threshold is fixed at 100 and cannot be overridden. The ignore section contains a values list of value-matching rules.

The following example drops any values starting with TEST-:

scope:
  ignore:
    values:
      - name: Synthetic test data
        matcher:
          type: pattern
          pattern: '^TEST-'

Each entry in values is a value-matching rule.

Non-Sensitive

The non_sensitive section marks columns or values as explicitly non-sensitive. They appear in discovery results with a non-sensitive designation.

It has two sub-fields:

• columns — column-level patterns (same pattern types as skip). Matched columns are marked non-sensitive regardless of metadata or IDD rules.
• values — value-level rules (same shape as value-matching rules). Matched values are reported as non-sensitive.

Note: File discovery configurations use locators instead of columns. Locator patterns match against the same forward-slash-separated strings described in Skip.

The following example marks a specific column and a value pattern as non-sensitive:

scope:
  non_sensitive:
    columns:  # Use "locators" for file discovery configs.
      - type: exact
        value: public.config.setting_name
      - type: glob
        pattern: "public.system_*.*"
    values:
      - name: Placeholder values
        matcher:
          type: pattern
          pattern: '^N/A$'

Value-Matching Rules

Value rules used inside scope.ignore.values and scope.non_sensitive.values have the same shape as IDD rules but without a label field.

Field	Type	Required	Default	Description
name	string	Yes	—	Unique name for this rule.
matcher	IDD matcher	Yes	—	Primary matching criteria (same IDD matcher types as IDD rules).
prematcher	IDD matcher	No	null	Optional pre-filter matcher.
datatypes	list[datatype]	No	null	Data type constraints.
min_threshold	integer	No	Inherited	Minimum match percentage (0–100). Fixed at 100 for ignore rules; configurable for non-sensitive rules.
test_words	boolean	No	false	If true, split each cell value on whitespace and test each word individually.
min_word_count	integer	No	null	Minimum word count per cell value. Values with fewer words are skipped. When null, no minimum is applied.
max_word_count	integer	No	null	Maximum word count per cell value. Values with more words are skipped. When null, no maximum is applied.

Document Discovery Settings

The document_discovery section controls how the top-level schema of a document database (for example MongoDB) is inferred. Document databases have no fixed schema, so DataMasque samples documents from each collection and unions their top-level field names to build the column list.

This section applies only to document databases. It is accepted but has no effect for relational databases, and a file discovery configuration rejects it.

document_sample_size sets how many documents are sampled per collection. Sampling more than 5% of a collection forces a slower server-side sort, so DataMasque honours the value but logs a warning when it is exceeded.

Example raising the sample size for large collections:

document_discovery:
  document_sample_size: 5000

Field	Type	Required	Default	Description
document_sample_size	integer	No	1000	Number of documents to sample per collection when inferring its top-level schema. Must be at least 1.

Files

The files section contains file-discovery run options. It is only available in file discovery configurations. Including a files section in a database discovery configuration will produce a validation error.

All fields are optional. Unset fields fall back to the built-in defaults: scan all files, no recursing into subdirectories, UTF-8 encoding, one worker.

Example scanning CSV and JSON files recursively, skipping any archive directory:

files:
  include:
    - '*.csv'
    - '*.json'
  skip:
    - '**/archive/**'
  recurse: true
  encoding: utf-8
  workers: 4

Field	Type	Required	Default	Description
include	list[string]	No	All files	Glob patterns selecting which files to scan.
skip	list[string]	No	None	Glob patterns excluding files from the scan.
recurse	boolean	No	false	Whether to descend into subdirectories when looking for files to discover.
encoding	string	No	utf-8	Text encoding used to read files.
workers	integer	No	1	Number of parallel file-discovery workers (≥ 1).

Worked Examples

Each example below is a complete, self-contained configuration that can be copied and pasted into the YAML editor.

Example 1: Multilingual Keywords

Add nombre (Spanish) and vorname (German) as metadata rules for first name detection. This extends the default detection to recognise non-English column names.

labels:
  - name: first_name
    description: "First name of a person"
    categories: ["PII", "PHI", "PCI"]
    mask: first_name_mask

metadata_rules:
  # Extend first_name detection with Spanish and German keywords
  - name: first_name_multilingual
    label: first_name
    column:
      type: keywords
      match: any
      words: [nombre, vorname, prenom, given_name, primeiro_nome]

idd_rules: []

This rule uses match: any so that a column named vorname, nombre, or prenom will all be detected as first_name. You can add this rule alongside the existing English-language rules to support multilingual databases.

Example 2: Country-Specific IDD Pattern (NZ IRD)

Detect New Zealand IRD (Inland Revenue Department) numbers using the nz_ird checksum algorithm. NZ IRD numbers are 8 or 9 digits with a weighted check digit.

labels:
  - name: ird_number
    description: "IRD number (New Zealand)"
    categories: ["PII"]
    mask: imitate_mask

metadata_rules: []

idd_settings:
  enabled: true
  row_sample_size: 1000

idd_rules:
  - name: nz_ird_checksum
    label: ird_number
    matcher:
      type: checksum
      algorithm: nz_ird
      min_digits: 8
      max_digits: 9
      ignore_non_digit: true
    datatypes:
      - type: string
        min_length: 8
        max_length: 12
      - type: integer
        min_length: 8
        max_length: 9
    min_threshold: 60

This rule:

• Uses the nz_ird checksum algorithm to validate the check digit.
• Strips non-digit characters (hyphens, spaces) before validation.
• Restricts matching to string columns of 8–12 characters or integer columns of 8–9 digits.
• Requires at least 60% of sampled values to pass validation before flagging the column.

Example 3: Custom Label for Compliance Reporting

Define a custom label for internal audit tagging. This label categorises columns for compliance reporting without generating masking rules.

labels:
  - name: gdpr_data_subject_id
    description: "GDPR data subject identifier for audit tracking"
    categories: ["GDPR", "Compliance"]
    skip_generation: true

metadata_rules:
  - name: gdpr_subject_id
    label: gdpr_data_subject_id
    column:
      type: keywords
      match: any
      words: [data_subject_id, subject_identifier, gdpr_id]

idd_rules: []

Key points:

• skip_generation: true means discovery will tag matching columns with this label, but no masking rule will be auto-generated. This is useful when the column needs to be tracked for compliance but should not be masked.
• Custom categories (GDPR, Compliance) appear in discovery reports and can be used to filter results.