Table References

• What is a table reference?
• Example: one map, two systems
• When to use a table reference
• Creating a table reference
• The source field
• Format and CSV options
• Names, deletion and lifecycle
• Size, storage and operations
• Using a table reference in a ruleset

A table reference names an existing table or file of known key-to-value rows — a map — so that rulesets can read it while they mask. A ruleset uses a table reference in two ways:

• as the source of a masking seed, which is what makes cross-system consistent masking possible: the same real-world entity masks to the same value in a database, a tabular file, and a JSON document, even when each system stores and keys that entity differently;
• as the source of the values an in or not_in condition tests against, so that which rows a rule masks is decided by a list you maintain outside the ruleset.

What is a table reference?

A table reference points at an existing table or file — through one of your connections — and gives it a name. It does not copy or store the data; at run time DataMasque reads the referenced table or file and uses it as a lookup while masking.

A table reference is made up of:

• a name, unique among your table references, that rulesets use to refer to it;
• a connection, which tells DataMasque where the map lives and how to reach it;
• a source, the table or file within that connection (see The source field);
• options, which describe the file format for file-backed maps (see Format and CSV options).

Example: one map, two systems

Here is a table reference doing its job before we get into the detail. A small map named customer_identity ties each customer's identifiers together — the CRM database's customer_id, the billing system's billing_ref, and the account_ref that identifies the customer's account to both:

customer_id	billing_ref	account_ref
1	BIL-4417	ACC-1001
2	BIL-9021	ACC-1002

A ruleset seeds a name mask from the map's account_ref, matching on whatever key the local system happens to have — here the database's customer_id:

version: "1.0"
tasks:
  - type: mask_table
    table: customers
    key: customer_id
    rules:
      - column: first_name
        hash_columns:
          - table_reference: customer_identity
            source_key: [customer_id]
            target_key: [customer_id]
            value: [account_ref]
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed

The source_key refers to a column in the table being masked. target_key and value refer to columns in the table reference. The rule is "Find the row where target_key in the table reference matches the value of source_key in the row being masked, and hash on the corresponding value of value from the table reference".

DataMasque hashes the account_ref it finds to choose the replacement name; the account reference itself is never written to the masked column. In a separate ruleset, you can then mask another database that keys on the billing_ref, changing the target_key to billing_ref in the hash_columns:

version: "1.0"
tasks:
  - type: mask_table
    table: '"BILLING_CUSTOMERS"'
    key: '"BILLING_REF"'
    rules:
      - column: first_name
        hash_columns:
          - table_reference: customer_identity
            source_key: ['"BILLING_REF"']
            target_key: [billing_ref]
            value: [account_ref]
        masks:
          - type: from_file
            seed_file: DataMasque_firstNames_mixed.csv
            seed_column: firstname-mixed

The masking seeds from the same map and resolves a customer to the same account_ref, and so masks that customer to the same name in both databases.

When to use a table reference

Reach for a table reference when the same entity appears in more than one place and must mask to the same value everywhere, but the systems do not share a key you can hash on directly. For example:

• a customer whose masked name must match across a customers database table and a monthly CSV extract that identifies the customer by a different column;
• a shared mapping table (an "identity map" maintained outside DataMasque) that dictates how a set of account numbers must be masked, which several rulesets must all obey.

If every system already keys the entity the same way, ordinary deterministic masking with hash_columns/hash_sources is enough. A table reference is what lets two systems that key an entity differently still agree, by translating each system's key into a shared seed value drawn from the map.

Creating a table reference

Note: Managing table references requires an administrator or mask builder account. The API endpoints reject API tokens — use a user token (see API).

Manage table references from their own Table References list page. In the Manage section of the navigation, click Table References. The list shows each table reference with its name, connection, source and last-modified date.

From here you can click New table reference to add one, edit an existing one, or Delete one (under the three-dots menu).

The add/edit form asks for these fields:

• Name — the unique name rulesets will use to refer to this table reference.
• Connection — the connection the map is read from.
• Path (for a file connection) or Table (for a database connection) — the file or table to read (see The source field). This is the source field in the API.
• File format — a CSV/Parquet choice, shown for file connections only. Choosing CSV also reveals the CSV options: Delimiter, Quote character, Encoding, and a Convert a matching value to NULL checkbox that enables the Null string field (see Format and CSV options).

The source field

The meaning of the Source field depends on the kind of connection the table reference is attached to:

• For a file connection, the source is a path to the file within the connection's file system (for example maps/customer_identity.csv). Any path is accepted; the file's suffix carries no meaning and does not determine how DataMasque reads the file — the format option decides that.
• For a database connection, the source is a schema-qualified schema.table reference (for example reference.customer_identity). DataMasque rejects an unqualified table name; the source must name both the schema and the table. The connection must be a relational database. Table references are not supported on DynamoDB, MongoDB, and DocumentDB.

A database source follows the same rules as a ruleset's table value. Write them unquoted where the database's own casing rules find them:

reference.customer_identity

Enclose either part in double quotation marks where it is case-sensitive, or where it contains a period, whitespace, or a double quotation mark or backslash of its own. Each part is quoted separately, and a literal double quotation mark or backslash is written after a backslash:

"Reference"."Customer Identity"
"Case\"Sensitive\\Schema"."Case.Sensitive Table"

Unlike a ruleset, where the whole value has to be wrapped in single quotation marks to survive YAML, the Source field takes the identifier on its own — no surrounding single quotation marks.

Format and CSV options

Format and CSV options apply to file connections only. DataMasque ignores them for database-backed maps, and ignores all of the CSV-specific options when the format is parquet.

Option	Applies to	Default	Description
format	file	csv	The file format of the map. One of csv or parquet.
delimiter	CSV files	,	The field delimiter.
encoding	CSV files	utf-8	The character encoding used to decode the file.
quotechar	CSV files	"	The character used to quote fields that contain the delimiter.
null_string	CSV files	(none)	A CSV cell whose value equals this string is read as null; set it to an empty string to read empty cells as null.

Two things to keep in mind:

• The format is always an explicit choice — DataMasque never infers it from the file's suffix. DataMasque reads a file named customers.csv as CSV only because format is csv, not because of its extension. Set format deliberately.
• DataMasque always reads a CSV map with a header row. It treats the first row of the file as column names, because the ruleset addresses map columns by name (target_key and value). Headerless CSV files are not supported.

Names, deletion and lifecycle

• Names are unique: two table references cannot share a name.
• Deleting a table reference frees its name, so you can create a new table reference that reuses the name.
• Deleting a connection cascades to its table references. When you delete a connection, DataMasque deletes every table reference that reads through it, too.
• A run that names a deleted table reference, or one that reads through a deleted connection, fails. The run log names the table reference that could not be resolved.

Size, storage and operations

DataMasque reads a table reference fresh at run time and builds it into a lookup on the agent before masking starts. A few limits and operational points follow from that.

Row limit

A single map may contain at most ten million rows. DataMasque rejects larger maps with an error.

A map must also contain at least one row. DataMasque fails the run when the source is empty. A CSV file that holds only a header row counts as empty.

Key size limit

A lookup matches the target_key columns of the map against the source_key columns of the row being masked. The values of those columns, taken together, may be at most 65,535 bytes for any one row, so key a table reference on identifier-shaped columns rather than on a column of large text.

Table references per run

A masking run may load at most 64 table references. DataMasque refuses a run that asks for more before it reads any of them.

Disk storage during a masking run

So that lookups stay fast while a run masks, DataMasque downloads each map it needs to disk in the agent container before masking starts, and removes it when the run ends. The maps a run downloads live on a volume of their own, and the storage they may occupy is capped by a budget — 20 GiB by default — set by the MASQUE_TABLE_REFERENCE_STORAGE environment variable (the Helm chart and ECS task template declare it; it accepts Kubernetes-style sizes such as 40Gi). Before building a map, DataMasque checks it against both the budget and the volume's free space, and fails the run if the next map will not fit, rather than masking part-way and then running out.

No key or value from the map's source is written to the volume in the clear. The keys and the values are encrypted, and the hashes the map searches on are keyed, under keys that exist only for the run and are held only in the agent's memory, so nothing on the volume can be matched back to a value someone guesses at.

Encrypting a value adds 16 bytes to it where it is stored, so an index that seeds — one built on value columns — occupies about a quarter more of the volume than the same map occupied in an earlier release. A deployment that sized MASQUE_TABLE_REFERENCE_STORAGE tightly against a measured workload may need around 26% more headroom for those indexes than it needed before.

The agent holds the run's keys in memory it locks for as long as the run lasts, so the copy that lasts cannot be written to swap. Each masking call works on a copy of the key it needs for as long as that call takes; it is the run-long copy that is locked. Locking asks for a non-zero locked-memory limit (RLIMIT_MEMLOCK) — one 4 KiB page per agent process that reads a map. Where the deployment does not allow it, the run carries on with an unlocked page and the agent logs a warning naming that limit: the maps are still encrypted, and it is only the protection from swap that is lost. A hardened deployment that sets the limit to zero can raise it to restore that protection.

If a run is hard-killed — an out-of-memory kill, for example — it has no chance to remove its own map files, which then sit on the volume until they are reclaimed. The keys those files were written under died with the run, so what is left behind is unreadable ciphertext. Restarting the agent container clears these leftover files. So if a map fails to load for lack of space and an earlier run had crashed, restarting the agent container is the first thing to try.

Database permissions for a database-backed map

DataMasque reads a database-backed map with SELECT queries only. It never writes to the map's table — no INSERT, UPDATE, DELETE, or DDL of any kind — so the connection's database user needs SELECT on the map's schema.table and no other privilege.

Indexing a database-backed map

DataMasque reads the whole map once — for a large table, in ordered pages — projecting only the columns the ruleset uses and ordering by them so the paging is stable. For a large mapping table it is worth having a covering index on those columns (the target_key and value columns a ruleset reads), so the database can return the pages without re-sorting the entire table for each one.

Using a table reference in a ruleset

With the map defined, a ruleset can seed masking from it, or test a value against one of its columns.

To seed masking from a map, which form you reach for depends on the task:

• mask_table and mask_tabular_file use the table_reference hash source in hash_columns;
• mask_file uses the table_reference lookup in the json mask for JSON and NDJSON documents.

Both share the same source_key, target_key, value, and on_missing keys, and each linked page walks through a worked example of one entity masking identically across systems.

To decide which rows a rule masks, an in or not_in condition can read the values it tests against from a column of a map, rather than listing them in the ruleset.