DataMasque Portal

Deterministic masking with files

File masking tasks (mask_file) support deterministic generation of masked values which are based on input values. You can provide a hash_sources list in a rule, similar to the hash_columns for database masking. However, with hash_sources you must specify additional parameters to fetch the hash.

Parameters

  • json_path (optional): A path to the value in the JSON data to use for the hash.
  • xpath (optional): An Xpath to the value stored in the XML data to use for the hash.
  • file_path (optional): A boolean value, when set to true the hash will be performed on the file's path (relative to the base directory). Defaults to false.
  • case_transform (optional): upper/lower - Apply a case transform on the input value, for consistent hash values irrespective of case.
  • trim (optional): If the value extracted from the json_path, or xpath contains leading or trailing white space, it can be trimmed by setting this option to true. Defaults to false.
  • regex (optional): A regular expression pattern to extract and concatenate values from the hash source before hashing. See About regex for details.
  • match_index (optional): A zero-based index specifying which regex match to use for hashing. If omitted, all matches are concatenated. Requires regex. See About regex.

Notes Each of the json_path, xpath and file_path are shown as optional but exactly one of these needs to be specified. For more information on json_path or xpath please refer to the JSON documentation/XML documentation. hash_sources for multirecord files works the same as for object file, except the hash_sources are applied to each record in the file instead of the entire file.

When hash_sources are specified as part of a json transform or xml node_transform:

  • The json_path/xpath should be a relative path from the current node.
  • If hash_sources are specified as part of the task and within the json/xml mask:
  • The hash_sources will be overwritten with the hash_sources specified in the json/xml mask.
  • file_path is no longer a valid parameter.

Example with json_path

This example will mask a JSON file which contains the following JSON data:

{
  "users": {
    "name": "Richard",
    "addresses": [
      {
        "type": "postal",
        "city": "Nelson"
      },
      {
        "type": "physical",
        "city": "Napier"
      }
    ]
  }
}

We want to mask the city of the users, hashed with the name of the users. To do this we need to specify a json_path to the value we want to use for the hash ([users,name]) in the hash_sources, as shown in the ruleset below. The masked value we would want to replace the city with, would be best if it was from one of the seed files DataMasque provides.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - hash_sources:
          - json_path: ['users', 'name']
        masks:
          - type: json
            transforms:
              - path: ['users','addresses', '*','city']
                masks:
                  - type: from_file
                    seed_file: DataMasque_mixed_countries_addresses.csv
                    seed_column: city

As shown in the results below, when the name has the same values for both users the resulting masked values for the city is also the same.

Note Each row of the results table below represents a separate file.

Show result

Before After
JSON data 
{"users":
  {"name": "Evelyn","addresses": [
    {"type": "postal","city": "Boronia"},
    {"type": "physical","city": "Cramphorne"}
  ]}}
 
{"users":
  {"name": "Sarah","addresses": [
    {"type": "postal","city": "Muckleford"},
    {"type": "physical","city": "Bandy Creek"}
  ]}}
 
{"users":
  {"name": "Bob","addresses": [
    {"type": "postal","city": "Doncaster"},
    {"type": "physical","city": "Shenton"}
  ]}}
 
{"users":
  {"name": "Sarah","addresses": [
    {"type": "postal","city": "Woronora"},
    {"type": "physical","city": "Lee Point"}
  ]}}
 
{"users":
  {"name": "Evelyn","addresses": [
    {"type": "postal","city": "Nelson"},
    {"type": "physical","city": "Napier"}
  ]}}
JSON data 
{"users":
  {"name":"Evelyn","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}
 
{"users":
  {"name":"Sarah","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"Bob","addresses":[
    {"type":"postal","city":"Bolivia"},
    {"type":"physical","city":"Bolivia"}
  ]}}
 
{"users":
  {"name":"Sarah","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"Evelyn","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}

If case_transform is specified, then the casing of the hash input values will not affect the consistency of the final masked result.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - hash_sources:
          - json_path: ['users', 'name']
            case_transform: lower
        masks:
        - type: json
          transforms:
          - path: ['users','addresses', "*",'city']
            masks:
            - type: from_file
              seed_file: DataMasque_mixed_countries_addresses.csv
              seed_column: city

Note Each row of the results table below represents a separate file.

Show result

Before After
JSON data 
{"users":
  {"name": "EVELYN","addresses": [
    {"type": "postal","city": "Boronia"},
    {"type": "physical","city": "Cramphorne"}
  ]}}
 
{"users":
  {"name": "SARAH","addresses": [
    {"type": "postal","city": "Muckleford"},
    {"type": "physical","city": "Bandy Creek"}
  ]}}
 
{"users":
  {"name": "Bob","addresses": [
    {"type": "postal","city": "Doncaster"},
    {"type": "physical","city": "Shenton"}
  ]}}
 
{"users":
  {"name": "Sarah","addresses": [
    {"type": "postal","city": "Woronora"},
    {"type": "physical","city": "Lee Point"}
  ]}}
 
{"users":
  {"name": "evelyn","addresses": [
    {"type": "postal","city": "Nelson"},
    {"type": "physical","city": "Napier"}
  ]}}
JSON data 
{"users":
  {"name":"EVELYN","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}
 
{"users":
  {"name":"SARAH","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"Bob","addresses":[
    {"type":"postal","city":"Bolivia"},
    {"type":"physical","city":"Bolivia"}
  ]}}
 
{"users":
  {"name":"Sarah","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"evelyn","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}

If trim is specified as true, then any whitespace around the hash input values will not affect the consistency of the final masked result. In the example below, trim has been applied to the hash_source, and values in the hash_source containing whitespace have not affected the final masked result.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - hash_sources:
          - json_path: ['users', 'name']
            trim: true
        masks:
        - type: json
          transforms:
          - path: ['users','addresses', "*",'city']
            masks:
            - type: from_file
              seed_file: DataMasque_mixed_countries_addresses.csv
              seed_column: city

Show result

Before After
JSON data 
{"users":
  {"name": "Evelyn  ","addresses": [
    {"type": "postal","city": "Boronia"},
    {"type": "physical","city": "Cramphorne"}
  ]}}
 
{"users":
  {"name": " Sarah ","addresses": [
    {"type": "postal","city": "Muckleford"},
    {"type": "physical","city": "Bandy Creek"}
  ]}}
 
{"users":
  {"name": "Bob","addresses": [
    {"type": "postal","city": "Doncaster"},
    {"type": "physical","city": "Shenton"}
  ]}}
 
{"users":
  {"name": "Sarah  ","addresses": [
    {"type": "postal","city": "Woronora"},
    {"type": "physical","city": "Lee Point"}
  ]}}
 
{"users":
  {"name": "  Evelyn","addresses": [
    {"type": "postal","city": "Nelson"},
    {"type": "physical","city": "Napier"}
  ]}}
JSON data 
{"users":
  {"name":"Evelyn  ","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}
 
{"users":
  {"name":" Sarah ","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"Bob","addresses":[
    {"type":"postal","city":"Bolivia"},
    {"type":"physical","city":"Bolivia"}
  ]}}
 
{"users":
  {"name":"Sarah  ","addresses":[
    {"type":"postal","city":"Canning Vale"},
    {"type":"physical","city":"Canning Vale"}
  ]}}
 
{"users":
  {"name":"  Evelyn","addresses":[
    {"type":"postal","city":"Stamford"},
    {"type":"physical","city":"Stamford"}
  ]}}

Example with xpath

This example will mask two different XML files but generate consistent masked data by using user id as a hash. The first contains a user's info.

<User id="8472">
    <Name>Evelyn</Name>
    <Address>
        <City>Nelson</City>
    </Address>
</User>

The second contains a list of user's log messages:

<Logs userId="8472">
    <Message to="Evelyn" date="2022-08-09" type="welcome"/>
    <Message to="Evelyn" date="2022-08-10" type="forgot_password"/>
</Logs>

In both cases, we want the user's name to be consistent, and this can be done by specifying the user's ID as the hash source for each file. We'll create a ruleset with two mask_file tasks, and use includes to map the right task to the correct file.

The first ruleset will mask the User/Name text value, and the second will mask the to attribute of each Log/Message element. The first ruleset will use the id attribute on User as the hash source, while the second will use the userId attribute on Logs.

Here's the ruleset to accomplish this:

version: "1.0"
tasks:
  - type: mask_file
    include:
      - glob: user_info_*.xml
    rules:
      - hash_sources:
        - xpath: 'User/@id'
        masks:
          - type: xml
            fallback_masks:
              - type: from_fixed
                value: '<User />'
            transforms:
              - path: 'User/Name'
                node_transforms:
                - type: text
                  masks:
                  - type: from_file
                    seed_file: DataMasque_firstNames_mixed.csv
  - type: mask_file
    include:
      - glob: user_logs_*.xml
    rules:
      - hash_sources:
        - xpath: 'Logs/@userId'
        masks:
          - type: xml
            fallback_masks:
              - type: from_fixed
                value: '<Logs />'
            transforms:
              - path: 'User/Name'
                node_transforms:
                - type: attribute
                  attributes: 'to'
                  masks:
                  - type: from_file
                    seed_file: DataMasque_firstNames_mixed.csv

The two output files would use consistent names. The first file:

<User id="8472">
    <Name>Sarah</Name>
    <Address>
        <City>Nelson</City>
    </Address>
</User>

And the second:

<Logs userId="8472">
    <Message to="Sarah" date="2022-08-09" type="welcome"/>
    <Message to="Sarah" date="2022-08-10" type="forgot_password"/>
</Logs>

Example with file_path

This example will just show an example ruleset to mask files based on the file path: This applies to any mask_file task.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      - hash_sources:
          - file_path: true
        masks:
          - type: json
            transforms:
              - path: ['users','addresses', '*','city']
                masks:
                  - type: from_file
                    seed_file: DataMasque_mixed_countries_addresses.csv
                    seed_column: city

With this ruleset the hashing will be performed on the file path, so each file with a different path with have differing masked values. This is useful if you want repeatable masking based on the name of a file.

Example with relative json_path

Referring to the document from the JSON path example above, DataMasque can generate a hash from a value that is relative to the current node being masked.

In the following example, the node being masked will be one of the ['users', 'addresses', '*','city'] text nodes.

The string .. is used to navigate up through the document hierarchy (analogous to using cd .. to move up a directory when browsing a files in a terminal).

First we show the ruleset, then explain the hash source's json_path.

version: "1.0"
tasks:
  - type: mask_file
    rules:
      masks:
        - type: json
          transforms:
            - path: ['users','addresses', '*','city']
              hash_sources:
                - json_path: ['..', '..', '..', 'name']
              masks:
                - type: from_file
                  seed_file: DataMasque_mixed_countries_addresses.csv
                  seed_column: city

Each transform in transforms is applied to each city found at path. There is only one transform in this example.

To fetch the name of the user for that particular city, to use in hashing, we must ascend up three nodes by using the .. operator three times:

  1. First ascending from the city text node to the address object.
  2. Then to the addresses array containing the list of addresses.
  3. Finally, to the user object that contains the addresses array.

The current node is now the user object. The final element in the json_path (name) can be used to fetch the name of that user.

In this way the city used in the from_file mask is generated based on the hash of the name of that user.

Example with relative xpath

Since the xml mask uses standard XPaths for node traversal, normal relative XPath syntax may be applied, as shown in this next example.

version: "1.0"
tasks:
  - type: mask_file
    include:
      - glob: user_info_*.xml
    rules:
      masks:
        - type: xml
          fallback_masks:
            - type: from_fixed
              value: '<User />'
          transforms:
            - path: '/User/Name'
              node_transforms:
                - type: text
                  hash_sources:
                    - xpath: '../@id'
                  masks:
                    - type: from_file
                      seed_file: DataMasque_firstNames_mixed.csv
  - type: mask_file
    include:
      - glob: user_logs_*.xml
    rules:
      masks:
        - type: xml
          fallback_masks:
            - type: from_fixed
              value: '<Logs />'
          transforms:
            - path: '/User/Name'
              node_transforms:
                - type: attribute
                  attributes: 'to'
                  hash_sources:
                    - xpath: '../@userId'
                  masks:
                    - type: from_file
                      seed_file: DataMasque_firstNames_mixed.csv

The above example uses two hash_sources with xpath attributes. The first uses ../@id to fetch the id attribute of the parent XML node (<User>), and the second similarly uses ../@userId to fetch the userId attribute.