DataMasque Portal

Safe Data Preview

Overview

A Safe Data Preview is a statistical summary of data without revealing any row-level identifying information.

It is enabled by default as part of In-Data Discovery runs, and is intended to help users configure their DataMasque masking rulesets.

Configuration

Safe Data Preview is enabled by default whenever In-Data Discovery runs. To configure or disable it, add a safe_data_preview block to the idd_settings of your discovery configuration.

idd_settings:
  enabled: true
  row_sample_size: 1000
  safe_data_preview:
    enabled: true
    min_group_size: 5
    string_level: patterns      # lengths | patterns | first_chars
    numeric_level: histograms   # summaries | histograms
    temporal_level: histograms  # summaries | histograms

In the above configuration:

  • enabled enables or disables Safe Data Preview. false disables Safe Data Preview but speeds up In-Data Discovery (default: true)
  • min_group_size is the smallest count a group needs to be presented. Counts below it are suppressed (minimum: 5).
  • string_level, numeric_level and temporal_level set how much detail each kind reveals, from least to most disclosive (see Kind-specific statistics).

Note: A higher row_sample_size will give a more accurate Safe Data Preview.

Statistics

Common statistics

Regardless of kind (numeric, string, etc.), each sample's Safe Data Preview returns:

Statistic Description
Row count Values in the sample.
Null count Sampled values that were null.
Distinct count Distinct sampled values (where computable).

All three are exact and not bounded by min_group_size.

Kind-specific statistics

Safe Data Preview categorises each discovered field into one of five kinds. Each kind's disclosure setting controls how much it reveals. A higher setting includes previews of lower settings.

Kind Setting What it adds Bounded by min_group_size Example
string first_chars top 5 most common first quarters of words ✓ Jo•••••• S••••
patterns top 5 most common character-shapes ✓ AA-99
proportions ✗ 40% letters, 40% digits, 20% other
lengths min, max, mean, median ✗ min 6, max 8
top 5 most common lengths ✗ most common: 8, 7
numeric histograms value histogram ✓ [0, 200): 120
summaries quartiles and percentiles ✗ median 50
temporal histograms date / time histogram ✓ 2024-01: 300
summaries quartiles and percentiles ✗ median 2024-02-15
boolean n/a true/false counts ✓ 76% true / 24% false
unsupported n/a n/a ✗ not available

Statistics bounded by min_group_size only show values if they can be grouped into at least min_group_size rows (default: 5). For example, J••• S•••• will only show if there are 5 J••• S•••• after redaction: e.g. John Smith, John Smith, John Sooth, Joan Smith, and Jonn Smitt.

The following shows how a safe data preview for a numeric-kind column: age will be shown in the ruleset generator.

Numeric preview popover

Unsupported reasons

If a column or field matches one of the following conditions it will be classified as an "unsupported" kind:

Reason Example
No rows sampled When a column does not have any values
All sampled values are null -
All sampled values are identical -
A value count is below the minimum group size boolean whose true or false count < min_group_size
Values couldn't be interpreted as a single consistent type JSON with mixed types
Binary data isn't previewed blob or other binary column
This data type has no Safe Data Preview Spatial and complex column types
Table has too many columns for Safe Data Preview a table with more than 150 columns
This column could not be previewed -

Privacy guarantees

Safe Data Preview never exposes row-level identifying information:

  • Aggregation: Apart from non-identifying statistics (percentiles, quartiles, lengths and proportions), each statistic aggregates multiple rows.
  • Small group suppression: Any group smaller than min_group_size is suppressed, so e.g. J••• (2) never appears.
  • P5 and P95 instead of min and max: Numeric summaries report the 5th and 95th percentile instead of the true minimum and maximum, and numeric and temporal histograms anchor their ranges at those percentiles, which prevents revealing identifying outliers such as an age of 93 or an oldest date of birth.

Limitations

  • Representativeness: Tabular files, and every database except PostgreSQL, SQL Server, Db2 LUW and Oracle 12c+, take a contiguous block as its sample (i.e. row_sample_size from a particular offset in the case of databases), which may not be representative of the whole column.
  • Distinct count: included only when the source can compute it and are omitted for unhashable values such as JSON lists.
  • Grouped files: files sharing a discovery result share one Safe Data Preview, each column or JSON field sampled from one file in the group.
  • Wide tables: a table with more than 150 columns is not previewed, to keep the memory used while sampling bounded; its columns are reported as unsupported.
  • Database Column types: Safe Data Preview only reads database columns whose type is on an allow-list. File fields are not filtered by type.
  • Speed: Safe Data Preview fetches more columns than In-Data Discovery alone so will be slower than In-Data Discovery by itself. To prioritise discovery speed, set enabled: false.

Next steps

Open the Ruleset Generator to see a Safe Data Preview beside each discovered column.

For database discovery, the Safe Data Preview is also included in the schema discovery CSV report.