Date Parsing in DataMasque

DataMasque has two ways to parse dates. They can be parsed with date format preservation, or by specifying a date format.

• Date Format Preservation
  • Parameters
    • Date Order
    • On Unknown Text
    • Fallback Date Format
  • Usage
    • mask_table and mask_tabular_file
    • mask_file
    • json and xml
  • How Settings Cascade
    • Set Once for Multiple Entities
    • Override an Inherited Setting for One Entity
    • Disable the unstructured_text Default
  • Times and Timezones
  • Interaction with typecast
    • How to Coerce Unstructured Dates to a Structured Format
    • typecast Without a Date Mask
  • Things to Know
  • Compatibility
• Specifying a Date Format
  • Date Format Directives
    • Parsing Years with/without Centuries (%Y vs. %y)
• Related Topics

Date Format Preservation

Date format preservation allows preserving the format of unstructured dates while masking the actual date underneath.

For example, if one document contained the same date written four different ways: 14/03/1978, 14-Mar-1978, March 14, 1978, and 19780314, DataMasque is able to understand these are the same date with different formats, and mask them, while preserving their respective formats.

DataMasque also handles a shortened part (Jul), a missing part (no day or year), and weekdays, and keeps whatever mix of lower, upper, or title case the words use.

Note that if date format preservation is enabled, a typecast following a date mask is not required.

Note: Currently only the from_random_date mask supports date format preservation.

date_format_preservation is enabled by default for compatible date submasks under the unstructured_text mask and disabled everywhere else. This means you most likely do NOT need to enable date_format_preservation unless you have an unstructured text column as described in Example: Unstructured Text.

Example: Unstructured Text

Unstructured text can contain structured (2005-08-19) or unstructured (19th of Aug, 2005) dates. For the clinical notes example below, an ai_detect matcher is used to detect DATE entities which are then masked with date_format_preservation. Under an unstructured_text mask, date format preservation is enabled by default, so the ruleset below does not contain an explicit date_format_preservation block with enabled: true.

version: "1.0"
tasks:
  - type: mask_table
    table: medical_records
    key: id
    rules:
      - column: clinical_notes
        masks:
          - type: unstructured_text
            matchers:
              ai_detect:
                - label: DATE
            masks:
              - label: DATE
                masks:
                  - type: from_random_date
                    min: "1980-01-01"
                    max: "2000-01-01"

Example output:

Parameters

All available settings are shown in the block below. All the shown settings are the defaults.

date_format_preservation:
  enabled: false
  date_order: month_first
  on_unknown_text:
    action: remove
    exempt: []
  fallback_date_format: '%Y-%m-%d'

date_format_preservation blocks can be added at the column level or at the label level below an unstructured_text mask.

Warning: date_format_preservation reads dates out of text, so use it on text columns only. With date_format_preservation enabled, from_random_date fails the run when it receives a date or datetime from a DATE or TIMESTAMP column or from a previous mask.

• enabled (optional): Turn date format preservation on or off. Defaults to true for an unstructured_text mask label and false everywhere else. See How Settings Cascade.
• date_order (optional): Which number is the day when a date reads two ways, such as 03/04/25. One of day_first or month_first. Defaults to month_first.
• on_unknown_text (optional): What to do with a word beside the date that the parser does not know. action can be one of retain, remove, or treat_as_invalid, defaults to remove. exempt defaults to []. See On Unknown Text.
• fallback_date_format (optional): The format to use for a null value or if parsing has failed. Accepts Python date format codes. Defaults to %Y-%m-%d.

Date Order

For ambiguous dates such as 03/04/2025, setting date_order: day_first makes the parser interpret 03 as the day and 04 as the month. For dates such as 15/05/2017 and January the 1st, 1998, there is no ambiguity so date_order has no effect.

- column: unstructured_dates
  hash_columns:
    - constant_column
  date_format_preservation:
    enabled: true
    date_order: day_first
  masks:
    - type: from_random_date
      min: "1980-01-01"
      max: "2000-01-01"

On Unknown Text

Date preservation works by identifying components inside text. For example, in "the 21st of jan": "21st" is a day component, and "jan" is an abbreviated month. Since these are recognised as parts of a date, they are maskable components and are safe. This leaves the parts "the " and " of ". DataMasque has a set of safe strings it considers safe under all circumstances, which includes these parts. Specifically, the following are treated as safe:

• whitespace
• punctuation (commas, apostrophes etc) and symbols (@, # etc)
• DataMasque's internal allow-list
• anything you add via exempt - see below

DataMasque's internal allow-list contains exactly:

["at", "day", "early", "in", "late", "mid", "of", "on", "the", "time", "year"]

You can extend this allow-list by adding items to the exempt list. For example to allow "GMT" you can add it with: exempt: [gmt]. The exempt list is not case-sensitive.

To take a concrete example, a date with a misspelling, Aprril 15th, contains Aprril with two rs. DataMasque does not recognise this text so it is treated as unknown text. The action field then specifies what happens when unknown text like this is encountered.

Suppose the mask is from_random_date which generates a replacement of 2009-12-02 with the default fallback_date_format (see below).

WARNING: if non-date inputs are provided to a compatible mask, such as "John Smith was born on the 21st of Feb", retain causes sensitive data in those inputs to pass through unmasked. In this case "John Smith" remains in the output string. The same preservation of sensitive data occurs if "John" and/or "Smith" are on the exempt list and you set action: remove or action: treat_as_invalid.

Fallback Date Format

The fallback_date_format is used to specify the date format used under any of the following conditions:

1. The parser has failed to parse the date.
2. Unknown text is encountered and action: treat_as_invalid is set.
3. The value is null.

Note: with a date mask, a masking run never fails if the parser fails. See typecast Without a Date Mask.

- column: unstructured_dates
  date_format_preservation:
    enabled: true
    fallback_date_format: "%m.%d.%Y"
  hash_columns:
    - constant_column     # used here only so `from_random_date` generates the same date for every row
  masks:
    - type: from_random_date
      min: "1980-01-01"
      max: "2000-01-01"

Usage

Expand the rulesets below for examples.

mask_table and mask_tabular_file

version: "1.0"
tasks:
  - type: mask_table
    table: patients
    key: id
    rules:
      - column: notes
        date_format_preservation:
          enabled: true
        masks:
          - type: unstructured_text
            matchers:
              ai_detect:
                - label: DATE
            masks:
              - label: DATE
                date_format_preservation:
                  date_order: day_first
                masks:
                  - type: from_random_date
                    min: "1980-01-01"
                    max: "2000-01-01"

mask_file

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - masks:
          - type: unstructured_text        # preservation is enabled by default here
            matchers:
              ai_detect:
                - label: DATE
            masks:
              - label: DATE
                date_format_preservation:
                  date_order: day_first
                masks:
                  - type: from_random_date
                    min: "1980-01-01"
                    max: "2000-01-01"

json and xml

version: "1.0"
tasks:
  - type: mask_table
    table: patients
    key: id
    rules:
      - column: json_data
        date_format_preservation:
          enabled: true
        masks:
          - type: json
            transforms:
              - path:
                  - dob
                masks:
                  - type: from_random_date
                    min: "1980-01-01"
                    max: "2000-01-01"

      - column: xml_data
        date_format_preservation:
          enabled: true
        masks:
          - type: xml
            transforms:
              - path: "//Patient"
                node_transforms:
                  - type: text
                    masks:
                      - type: from_random_date
                        min: "1980-01-01"
                        max: "2000-01-01"

How Settings Cascade

A date_format_preservation block can sit in two places: on a column rule, and under a label inside unstructured_text. The label is more specific, so if both state the same field, the label wins. A mask_file rule has no column, so a label is the only place its settings can go.

Settings merge field by field, not by block. A label that states only date_order still takes fallback_date_format from the column. An unspecified field takes its default.

enabled is the one field whose default depends on where it is read. A mask under unstructured_text preserves the date format unless explicitly disabled. Everywhere else, preservation is disabled unless it's turned on.

For the examples below, suppose the mask generates 2005-10-07 for the DATE_OF_BIRTH entity and 2026-03-14 for the DATE entity.

Set Once for Multiple Entities

- column: clinical_notes
  date_format_preservation:
    enabled: true
    on_unknown_text:
      action: treat_as_invalid
  masks:
    - type: unstructured_text
      matchers:
        ai_detect:
          - label: DATE
          - label: DATE_OF_BIRTH
      masks:
        - label: DATE
          masks:
            - type: from_random_date
              ...
        - label: DATE_OF_BIRTH
          masks:
            - type: from_random_date
              ...

Override an Inherited Setting for One Entity

- column: clinical_notes
  date_format_preservation:
    enabled: true
    fallback_date_format: "%Y.%m.%d"
  masks:
    - type: unstructured_text
      matchers:
        ai_detect:
          - label: DATE
          - label: DATE_OF_BIRTH
      masks:
        - label: DATE
          masks:
            - type: from_random_date
              ...
        - label: DATE_OF_BIRTH
          date_format_preservation:
            fallback_date_format: "%d_%m_%Y"
          masks:
            - type: from_random_date
              ...

Disable the unstructured_text Default

- column: clinical_notes
  date_format_preservation:
    enabled: false
  masks:
    - type: unstructured_text
      matchers:
        ai_detect:
          - label: DATE
      masks:
        - label: DATE
          masks:
            - type: from_random_date
              ...
            - type: typecast
              typecast_as: string

Times and Timezones

The general principle for date_format_preservation is dates retain their formats, but if the input also contains a time/timezone then these are passed through unmasked. If times must be masked, do not use date_format_preservation.

However, this is conditional on there being recognisable date components to begin with. If no date components can be identified, the parser considers this a parse failure and falls back to the Fallback Date Format. See the last row of the table below for an example.

For example with a generated date 2007-07-05:

ACDT, ACST, ACT, ACWST, ADT, AEDT, AEST, AET, AFT, AKDT, AKST, ALMT, AMST, AMT, ANAT, AQTT, ART,
AST, AWST, AZOST, AZOT, AZT, BIOT, BIT, BNT, BOT, BRST, BRT, BST, BTT, CAT, CCT, CDT, CEST, CET,
CHADT, CHAST, CHOST, CHOT, CHST, CHUT, CIST, CKT, CLST, CLT, COST, COT, CST, CT, CVT, CWST, CXT,
DAVT, DDUT, DFT, EASST, EAST, EAT, ECT, EDT, EEST, EET, EGST, EGT, EST, ET, FET, FJT, FKST, FKT,
FNT, GALT, GAMT, GET, GFT, GILT, GIT, GMT, GST, GYT, HAEC, HDT, HKT, HMT, HOVST, HOVT, HST, ICT,
IDLW, IDT, IOT, IRDT, IRKT, IRST, IST, JST, KALT, KGT, KOST, KRAT, KST, LHST, LINT, MAGT, MART,
MAWT, MDT, MEST, MET, MHT, MIST, MIT, MMT, MSK, MST, MT, MUT, MVT, MYT, NCT, NDT, NFT, NOVT, NPT,
NST, NT, NUT, NZDST, NZDT, NZST, OMST, ORAT, PDT, PET, PETT, PGT, PHOT, PHST, PHT, PKT, PMDT, PMST,
PONT, PST, PT, PWT, PYST, PYT, RET, ROTT, SAKT, SAMT, SAST, SBT, SCT, SDT, SGT, SLST, SRET, SRT,
SST, SYOT, TAHT, TFT, THA, TJT, TKT, TLT, TMT, TOT, TRT, TST, TVT, ULAST, ULAT, UT, UTC, UYST, UYT,
UZT, VET, VLAT, VOLT, VOST, VUT, WAKT, WAST, WAT, WET, WGST, WGT, WIB, WIT, WITA, WST, YAKT, YEKT, Z

Interaction with typecast

With date format preservation enabled, compatible masked dates are automatically coerced to a string when required. As such, a typecast mask is NOT required after a compatible mask.

However, typecast can be used to coerce an unstructured masked date into a structured format if you desire.

Note: With date format preservation on, a typecast with a date_format overrides the preserved shape. If date_format is excluded, a typecast with typecast_as: string has no effect.

How to Coerce Unstructured Dates to a Structured Format

version: "1.0"
tasks:
  - type: mask_table
    table: patients
    key: id
    rules:
      - column: notes
        masks:
          - type: unstructured_text
            matchers:
              ai_detect:
                - label: DATE
            masks:
              - label: DATE
                date_format_preservation:
                  enabled: true
                masks:
                  - type: from_random_date
                    ...
                  - type: typecast
                    typecast_as: string
                    date_format: "%d.%m.%Y"

Note: In an unstructured_text label, if typecasting to date or datetime, make sure you typecast back to a string at the end. Otherwise, the run fails.

typecast Without a Date Mask

If date_format_preservation is enabled, the input text will be parsed using the date_format_preservation parser.

If the input is missing fields, a typecast to string with a date_format will fail the run. This is because for typecast to convert a parsed value to a structured date, all fields must be present.

Warning: It is not recommended to use typecast without a date mask with date_format_preservation enabled.

- column: unstructured_dates
  date_format_preservation:
    enabled: true
  masks:
    - type: typecast
      typecast_as: string
      date_format: "%Y-%m-%d"

Things to Know

• If a weekday exists in the input, the weekday changes to the true weekday of the masked date.
• A two-digit year uses a fixed pivot at 69. A year below 69 becomes 20xx, and a year of 69 or more becomes 19xx. So '50 becomes 2050, and '75 becomes 1975. This matches how DataMasque parses years without a century.
• Sept becomes Sep, Tues becomes Tue, and Thurs becomes Thu.

Compatibility

If date_format_preservation is enabled:

• chain: masks run in order, each reading the output of the one before. A date mask keeps the shape of the text it receives.
• concat: each mask inside concat reads the input concat receives, so a date mask inside a concat preserves the format, regardless of the siblings.
• typecast: can be used to convert a date to a string, date, or datetime. Note that typecasting to date or datetime does not preserve the format. See Interaction with typecast.

The following are not compatible:

• imitate_date mask.
• from_random_date mask with epoch parameters. This applies to the whole column, so a json or xml mask cannot use epoch parameters in one transform and date format preservation in another.
• Wildcard column rules (column: "*"), when masking with Amazon DynamoDB and MongoDB.
• Number and boolean values in a json document.

Specifying a Date Format

Often, rulesets need to specify a date format to convert text into a date. For example, using Date/Time masks, executing type casts or performing Conditional Masking.

Date Format Directives

Date parsing in DataMasque follows standard %-based directives, as documented in strftime() and strptime() Format Codes.

An important point to note is how years with and without centuries are handled.

Parsing Years with/without Centuries (%Y vs. %y)

Two directives are available to parse years:

• %Y to parse years with the century (e.g., 2020, 1999, 1867, etc.).
• %y to parse years without the century (e.g., 20, 99, 67, etc.).

When parsing years without a century using %y, the following interpretation applies:

• Years from 00 to 68 are interpreted as the years 2000 to 2068.
• Years from 69 to 99 are interpreted as the years 1969 to 1999.

While %y provides a convenient way to parse two-digit years, it introduces ambiguity in interpretation, especially near the transition between centuries.

It is recommended to use %Y whenever possible to avoid ambiguity, and ensure consistent interpretation across platforms and environments.

Related Topics

• Unstructured Text Masking
• from_random_date mask
• typecast mask
• imitate_date mask