Databricks

DataMasque can mask Databricks data in more than one way. Which approach you use depends on the Databricks product and, for lakehouse data, the table's storage format. This page helps you choose; the detailed setup and limitations live on the linked pages.

• Choosing how to mask Databricks data
  • Databricks SQL Warehouse (Delta tables)
  • Databricks Lakebase (PostgreSQL OLTP)
  • masque-bricks (non-Delta tables)
• Masking non-Delta tables with masque-bricks
  • Overview
  • Prerequisites
  • Configuring S3 access for Databricks
  • Running the workflow with masque-bricks

Choosing how to mask Databricks data

Your data	Use	How it works
Delta tables in Unity Catalog (managed or external)	Databricks SQL Warehouse connection	Masked in place via the Databricks SQL connector. No external storage. Recommended.
Lakebase (PostgreSQL-compatible OLTP)	Databricks Lakebase connection	Standard database connection, masked in place.
Non-Delta tables (Parquet, CSV, JSON, ORC, …)	masque-bricks	Export to S3 → file-mask → reload. Supplemental tool for tables that cannot be masked in place.

Databricks SQL Warehouse (Delta tables)

A SQL analytics endpoint backed by Unity Catalog. DataMasque connects to it directly via the Databricks SQL connector and masks Delta tables in place, within the target's own Unity Catalog catalog and schema, with no external staging bucket. Both managed and external Delta tables are supported. This is the default path for masking lakehouse data.

In-place masking uses MERGE INTO, which Unity Catalog supports only on Delta tables. Non-Delta tables are skipped during discovery and rejected before a run starts; use masque-bricks for those.

See Database Connections → Databricks SQL Warehouse connections for setup, the masking process, limitations, and post-masking data-persistence details.

Databricks Lakebase (PostgreSQL OLTP)

A PostgreSQL-compatible OLTP database. DataMasque connects to it as a standard database connection and masks in place.

See Database Connections → Databricks Lakebase connections for setup and limitations.

masque-bricks (non-Delta tables)

A supplemental command-line tool for masking Databricks tables that the in-place SQL Warehouse connection cannot handle. See Masking non-Delta tables with masque-bricks for when to use it and how it works.

---

Masking non-Delta tables with masque-bricks

masque-bricks is a supplemental tool for masking Databricks tables that cannot be masked in place by the SQL Warehouse connection, most commonly non-Delta tables stored as columnar files (Parquet, CSV, JSON, ORC, and so on) in cloud object storage. It exports the table to S3, masks the files with DataMasque file masking, and loads the masked data back.

For Delta tables, use the in-place SQL Warehouse connection instead: it requires no S3 staging bucket and keeps the data within Unity Catalog governance.

Overview

The pipeline runs in four steps, all of which are automated by masque-bricks:

1. Export: Databricks SQL writes the source table to S3 as Parquet files.
2. Read: DataMasque file masking reads the raw Parquet files from S3.
3. Write: DataMasque writes the masked Parquet files back to S3.
4. Import: Databricks SQL loads the masked Parquet files into the target table.

Prerequisites

• Databricks workspace with SQL Warehouse access.
• DataMasque instance with file masking enabled.
• An S3 bucket accessible to both Databricks and DataMasque.
• IAM roles configured so Databricks can read and write to the S3 bucket. See File connections for the DataMasque-side S3 access configuration.

Configuring S3 access for Databricks

Databricks needs permission to read and write to your S3 bucket. This typically involves creating an IAM role for Databricks and registering an external location in Databricks Unity Catalog that points to the bucket.

Refer to the Databricks documentation for the up-to-date setup steps:

• Configure S3 access with instance profiles
• Create an external location in Unity Catalog
• Manage external locations

Example IAM policy

The IAM role attached to your Databricks workspace needs the following permissions on the S3 bucket used as the staging area:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

For DataMasque's S3 access requirements, see File connections.

Running the workflow with masque-bricks

Install and configure the tool by following the instructions in the masque-bricks repository, then run:

# Full pipeline: export → mask → import
masque-bricks run \
  --table source_table \
  --schema default \
  --target-table masked_table \
  --ruleset-file rulesets/pii.yaml

For full configuration options and advanced usage, refer to the masque-bricks repository.