References Across Projects
Rulesets, ruleset generation configs, discovery configs and libraries refer to other resources by name:
libraries in imports and $ref,
table references in table_reference,
and seed files in seed_file, file_name (for from_blob), filename (for discovery configurations),
sql_file and sql_select_file.
This page describes how DataMasque works out which resource a name refers to.
Bare and qualified names
A name can be written in two ways:
- A bare name, such as
customers, refers to a resource in the same project. - A qualified name, such as
Sales/customers, refers to a resource in the project named before the/.
version: "1.0"
imports:
- common_masks # Library "common_masks" in this ruleset's project
- Shared/phone_masks # Library "phone_masks" in the project "Shared"
tasks:
- type: mask_table
table: customers
key: id
rules:
- column: first_name
masks:
- type: from_file
seed_file: Shared/first_names.csv # Seed file "first_names.csv" in the project "Shared"
seed_column: first_name
A seed file name can contain only one /, between the project and the file name.
In a ruleset, a value that matches no seed file is read as a path under the files directory, with a warning in the run log
(see Move files copied into subfolders).
The Files page and the seed file panel in the ruleset editor show each seed file by its qualified name,
such as Sales/first_names.csv, and copy that name for you to paste into a ruleset.
They show built-in seed files by their bare name.
Built-in seed files
Built-in seed file names start with DataMasque_.
Since built-in seed files are usable by any project and belong to no project, they always use a bare name:
- type: from_file
seed_file: DataMasque_firstNames_mixed.csv
seed_column: firstname-mixed
Resolving names during a run
A run's project is that of its ruleset (for masking runs) or its discovery config (for discovery runs). A discovery run with no discovery config takes the project of its connection. All references are resolved within the context of that project. As such, a seed file or table reference name inside a library is looked up in the importing ruleset or discovery config's project, not in the library's project. For example, given these seed files:
# Seed file `first_names.csv` in project A
first_name
Aaron
Amelia
Angela
Andrew
# Seed file `first_names.csv` in project B
first_name
Bob
Betty
Belinda
Brian
and this library and ruleset:
# my_library in project A
masks:
file_mask:
type: from_file
seed_file: first_names.csv
seed_column: first_name
# ruleset in project B
imports:
- A/my_library
tasks:
- type: mask_table
table: customers
key: id
rules:
- column: first_name
masks:
- $ref: A/my_library#masks/file_mask
This ruleset will mask the first_name column to the names beginning with B,
because the seed file is looked up in project B (where the ruleset is), not project A (where the library is).
If instead you wanted to use the seed file in project A,
change the seed_file line in the library to A/first_names.csv.
DataMasque looks up a bare seed file, table reference, or library name in the project of the run.
Bare names are also used to refer to built-in seed files, as mentioned above.
A bare library name in imports is looked up in the project of the ruleset or discovery config that imports it.
Matching names
In a qualified name, the project part is case-insensitive, but the resource part is case-sensitive.
So sales/customers refers to a library called customers in the project Sales,
but not to a library called Customers (capital C).
In-flight masking
In-flight masking does not use projects. In an in-flight masking ruleset plan, you must:
- Name a built-in seed file using its bare name, for example
DataMasque_firstNames_mixed.csv. - Name an uploaded seed file by its fully qualified name, such as
Sales/first_names.csv.
A ruleset plan that names an uploaded seed file by its bare name is rejected with an error such as:
Error creating ruleset plan: Seed file "first_names.csv" not found. Does it exist in the built-in seeds? If it is an uploaded seed file, name it with its project, for example "<project>/first_names.csv".
Updating references when resources or projects change
DataMasque does not automatically update references when resources move between projects, when resources are renamed or deleted, or when projects are renamed or deleted. The following sections explain what happens with each different type of resource. In most cases, you need to update all affected references. When you move resources, the Move to project dialog shows a notice explaining each case.
References to seed files and table references in YAML documents
DataMasque does not check that a seed file or table reference exists when a ruleset is saved.
It refuses a path of three or more parts under seeds/ or auth/.
If no resource matches a name, the run fails when it starts,
with an error naming the seed file or table reference and where DataMasque looked for it.
For example:
SQL script "cleanup.sql" not found. Does it exist in the project "Sales" or the built-in seeds?
Discovery configs check seed file names when they are saved, but these references are not checked again when the seed file is renamed, moved or deleted.
Update the affected references as follows:
- When a seed file or table reference moves to another project,
rulesets and discovery configurations in its old project can no longer refer to it by its bare name.
Change these references, and any that name the old project, to name the new project,
such as
Finance/first_names.csv. - When a ruleset or discovery configuration moves to another project,
its bare seed file and table reference names are looked up in the destination project,
including names inside the libraries it imports.
To keep using a seed file or table reference that stays in the old project,
add the old project to its name, such as
Sales/first_names.csv. - When a seed file or table reference is renamed or deleted, change or remove each reference to it.
When a table reference or its connection moves, the table reference keeps the link to its connection, whether or not both are in the same project.
In-flight masking ruleset plans must always name uploaded seed files with a qualified reference, so update any plan that uses a moved seed file.
Libraries
When a library (ruleset library or discovery config library) is moved, renamed or deleted, each ruleset, ruleset generation config and discovery configuration that imports it becomes invalid, with a reason such as:
Imported library "Sales/phone_masks" was moved to "Shared/phone_masks".
Imported library "Sales/phone_masks" was renamed to "Sales/phone_formats".
Imported library "Sales/phone_masks" was deleted.
A ruleset or configuration that imports a library by its bare name also becomes invalid when it moves to another project and the library does not.
To fix it, update the import to name the library's current project, such as Finance/phone_masks,
and save the ruleset or config.
Alternatively, move the importing resource too, so that both are in the same project again.
Note the following:
- If the library comes back, for example if you move it back or create a library with the same name, the rulesets and configs that import it become valid again without being saved.
- A bare import can end up resolving to a different library.
For example, you could have a ruleset in the
Salesproject that imports a library calledmy_library, then deletemy_libraryfromSalesand movemy_libraryfrom another project intoSales. DataMasque marks the ruleset invalid, because it cannot tell if the new library is the one you meant. Check that the import names the library you intended. Then qualify the import with the ruleset's project, such asSales/my_library, or make a small edit, such as adding a comment, and save the ruleset.
Connections and credential files
When connections and credential files move, no changes are necessary. A connection keeps using its credential file (such as a MySQL SSL ZIP, Oracle wallet, or Snowflake key) regardless of whether the connection and credential file now reside in the same project or in different projects.
Renaming a project
Renaming a project does not update references to its old name. A qualified import of a library in the renamed project becomes invalid, and a qualified seed file or table reference name makes the run fail when it starts. Update the references to match the new project name. Be aware that if another project is later given the old name, any references not yet updated will pick up libraries, seed files and table references from that project.
Deleting a project
When a project is deleted, its resources are not deleted, but rather moved into the destination project. The situation with regard to references, ruleset validation and so on is then the same as if the resources had been moved by hand. Follow the relevant steps from the above sections.