DataMasque Portal

Amazon DynamoDB connections

DataMasque masks Amazon DynamoDB tables by exporting them to Amazon S3, masking the exported files, and importing the masked data into new tables.

Masking process

Amazon DynamoDB databases are masked by exporting the table to an S3 bucket, splitting the exported files into batch files of manageable sizes, masking the data, and then importing into a new Amazon DynamoDB table with a new name. The existing table is retained. This method is used because table scans and bulk updates of Amazon DynamoDB are not performant.

The following diagram shows the high level data masking workflow for Amazon DynamoDB:

DataMasque reference deployment

The data in the S3 bucket is removed at the end of a successful masking run.

Important! Since this staging bucket will have sensitive data stored in it temporarily, access to the bucket should be confined to trusted users/IAM roles only.

Due to the speed of Amazon DynamoDB Data Export and Amazon DynamoDB Data Import, masking in Amazon DynamoDB can take longer than standard SQL database masks. Most of this time is taken up by the Amazon DynamoDB import and export processes, which are internal to AWS and so DataMasque has no control over their performance. Masking a table with a single item can take up to ten minutes, while masking 1,000,000 items takes around 20 minutes, so the time does not scale linearly with the number of items.

Warning!: On failed or cancelled masking runs, the staging S3 bucket should be checked and left over export data may need to be manually cleaned up. This is because DataMasque might not be able to clean up exported data as it depends on when and how the masking run failed. For example, if an export is in process when a run is cancelled then AWS may continue to execute it and export data.

If the import fails, the run log gives the failure code and message from AWS, and DataMasque deletes the empty table the import created. It deletes the table only when Amazon DynamoDB confirms it is the one that import created; otherwise the run log has a warning naming the table, which you can check and delete in the AWS Management Console.

For more information please refer to the AWS Documentation for Amazon DynamoDB

Required privileges

Masking an Amazon DynamoDB table has five steps:

  1. The table to be masked is exported to an S3 bucket.
  2. DataMasque reads the exported data and masks it.
  3. The masked data is uploaded back to the same S3 bucket.
  4. An Amazon DynamoDB import is run to load the masked data into a new table.
  5. The masked and unmasked data is removed from the S3 bucket.

This process is much faster than masking data in place. Because of this approach, your DataMasque instance requires access to both the Amazon DynamoDB table being masked, and the staging S3 bucket to store the data in transit.

Access control

It is important to restrict access on the S3 bucket you configure to use on your Amazon DynamoDB connection.

Disable object lock

Object Lock must be disabled on your S3 bucket to allow DataMasque to clean up staging data when masking is completed. If Object Lock is not disabled, staging data will be left in your S3 bucket.

Optional configurations

The following configurations are not required, but recommended:

  • Disable Bucket Versioning.

For AWS EC2 instances: Follow the steps below to configure the IAM Role which can be attached to the EC2 running DataMasque to allow for masking of Amazon DynamoDB database tables and accessing your S3 bucket:

To mask a DynamoDB table in a different AWS account from where DataMasque is, please refer to the Cross Account Functionality documentation.

  1. Open the Amazon IAM console, create an IAM policy. The following example IAM policy grants the required privileges for Amazon DynamoDB and S3, where <bucket-name> and <table-name> should be replaced with the S3 staging bucket and table being masked respectively. Sid attributes have been added to explain the purpose of each statement. Further explanation of the Resource attributes are given after the example policy.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEventLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ExportedDataAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketPolicyStatus",
                "s3:GetBucketObjectLockConfiguration",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name>",
                "arn:aws:s3:::<bucket-name>/*"
            ]
        },
        {
            "Sid": "SourceTableExport",
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeTable",
                "dynamodb:DescribeContinuousBackups",
                "dynamodb:ExportTableToPointInTime",
                "dynamodb:Scan"
            ],
            "Resource": [
                "arn:aws:dynamodb:<region>:<account-id>:table/<table-name>"
            ]
        },
        {
            "Sid": "SchemaDiscoveryTableListing",
            "Effect": "Allow",
            "Action": [
                "dynamodb:ListTables"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ExportProcessStatus",
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeExport"
            ],
            "Resource": [
                "arn:aws:dynamodb:<region>:<account-id>:table/<table-name>/*"
            ]
        },
        {
            "Sid": "TargetTableImportAndCleanup",
            "Effect": "Allow",
            "Action": [
                "dynamodb:DeleteTable",
                "dynamodb:DescribeTable",
                "dynamodb:ImportTable",
                "dynamodb:UpdateTable"
            ],
            "Resource": [
                "arn:aws:dynamodb:<region>:<account-id>:table/<table-name>*"
            ]
        },
        {
            "Sid": "ImportProcessStatus",
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeImport"
            ],
            "Resource": [
                "arn:aws:dynamodb:<region>:<account-id>:table/<table-name>*/*"
            ]
        }
    ]
}

Note: Since the policy above makes use of wildcards (*) you may receive warnings when adding the policy through the AWS console. Since not all the wildcards can be removed (as explained below) these warning can be ignored.

Since the name of the S3 staging bucket is defined as part of the DataMasque connection, it is static, so it can be hard-coded as part of the policy. The Amazon DynamoDB table names are defined as part of ruleset, so more flexibility is needed when specifying the IAM policy.

The SourceTableExport statement controls access to the source table to get information about it. DataMasque requires point-in-time-recovery (PITR)/continuous backups to be enabled for a table in order to be able to export it, therefore it requires permission be able to check that the table has these enabled.

The Resource arguments for the SourceTableExport statement specify the source tables that DataMasque has access to. Specific tables can be named, or * can be used to allow access to all tables in your account. Multiple Resource entries can be added to grant access to multiple named tables. Here are some Resource examples:

  • "arn:aws:dynamodb:*:*:table/*": allow access to all tables in your account.
  • "arn:aws:dynamodb:*:*:table/users": allow access an example users table.
  • "arn:aws:dynamodb:us-east-1:123456789012:table/users": allow access an example users table, in the us-east-1 region, for the account 123456789012.

Two of the permissions are needed only for schema discovery, and can be left out if it is not used:

  • dynamodb:Scan, in the SourceTableExport statement, lets DataMasque read a sample of items from each table. The Resource entries of that statement must cover every table that schema discovery is to report.
  • dynamodb:ListTables, in the SchemaDiscoveryTableListing statement, lets DataMasque find the tables to discover. This action does not accept a table as its resource, so it is in a statement of its own with "Resource": "*".

Note: An existing role must have both permissions added before schema discovery is run on an Amazon DynamoDB connection. A table that the role cannot read is skipped, with an error in the run log that names the table and the permission that was refused, and the run continues with the other tables. A run fails when the role cannot list the tables, or when no table can be read.

The ExportProcessStatus statement contains permissions to allow DataMasque to query the status of the source table's export process. Under Resource, the source table name can be set, followed by /*. A trailing wildcard is required as the ID of the export process is unknown until started. A * can be specified instead of a table name to allow access to export statuses for all tables in your account. Multiple Resource entries can be added to grant access to multiple named tables. Here are some Resource examples:

  • "arn:aws:dynamodb:*:*:table/*": allow access to all export processes in your account.
  • "arn:aws:dynamodb:*:*:table/users/*": allow access to export processes for just an example users table.
  • "arn:aws:dynamodb:us-east-1:123456789012:table/users/*": allow access to export processes for just an example users table, in a specific region and account.

The permissions under TargetTableImportAndCleanup allow importing masked data from S3 to the target table. The Resource attributes make use of the wildcard as sometimes the name of the target table cannot be predicted. The name of the target table is generated from the source table, with a suffix added. By default, the suffix is random, therefore the example shows how to specify the source table name followed by a * (i.e. any suffix).

If specifying the suffix as part of the DataMasque connection settings (the Specify Suffix setting), then the suffix can be specified in the policy too.

Multiple Resource entries can be added to grant access to more named tables. Here are some Resource examples:

  • "arn:aws:dynamodb:*:*:table/*": allow access to import into any table.
  • "arn:aws:dynamodb:*:*:table/users*": allow access to import into an example table users with any suffix.
  • "arn:aws:dynamodb:*:*:table/users-masked": allow access to import into an example table users-masked. Must be used in conjunction with the Specify Suffix setting. In this case, the suffix -masked was specified.
  • "arn:aws:dynamodb:us-east-1:123456789012:table/users-masked": allow access to import into an example table users-masked, in a specific region and account. Again, used with the Specify Suffix setting.

Note that the dynamodb:DeleteTable permission is only required if the Replace existing tables option is turned on in the Amazon DynamoDB connection's Advanced Settings.

The ImportProcessStatus statement contains permissions to allow DataMasque to query the status of the target table's import process. Under Resource, the source table name can be supplied, followed by a *, to allow any suffix. It is then followed by /* to allow access to any import process status for that table. As with the export process, the import process ID is not known until the process is started, so the use of wildcard is necessary.

As with the access to the table itself, if the suffix has been specified as part of the connection settings, it can be used in the policy.

Multiple Resource entries can be added to grant access to multiple named tables. Here are some Resource examples:

  • "arn:aws:dynamodb:*:*:table/*": allow access to all import processes in your account.
  • "arn:aws:dynamodb:*:*:table/users*/*": allow access to import processes for a users table with any suffix.
  • "arn:aws:dynamodb:*:*:table/users-masked/*": allow access to import processes for an example table users-masked. Must be used in conjunction with the Specify Suffix setting. In this case, the suffix -masked was specified.
  • "arn:aws:dynamodb:us-east-1:123456789012:table/users-masked/*": allow access to import processes for an example table users-masked, in the specified region and account. Again, used with the Specify Suffix setting.

  1. Create an IAM role:

    • Select AWS service as the trusted entity, EC2 as the service.
    • Attach the policy created from the previous step to the role.

  2. Open the Amazon EC2 and associate the IAM role to your EC2.

Important:* EC2 instance are unable to determine the AWS region in which they reside. The region is provided if setting up a ~/.aws/credentials file, but not in the case of attaching a policy. In this case, the Amazon DynamoDB table name in the YAML ruleset must be prefixed by the region and a forward slash. For example, the table users in region us-east-1 should be specified as us-east-1/users:

version: "1.0"
tasks:
  - type: mask_table
    table: '"us-east-1/users"'

Advanced SSE Key Permissions

All Amazon DynamoDB tables are server-side encrypted, however the way the encryption keys are managed is configurable. Please refer to Amazon DynamoDB Encryption key management for background on the types of key management available.

There are three scenarios when using DataMasque, where extra permissions are required on the IAM policy to access KMS keys:

  1. The option Stored in your account, and owned and managed by you is selected when configuring how the target Amazon DynamoDB table should be imported.
  2. The option Use configuration from the original table is selected when configuring how the target Amazon DynamoDB table should be imported, and the source table uses Stored in your account, and owned and managed by you for key management.
  3. The option Use configuration from the original table is selected when configuring how the target Amazon DynamoDB table should be imported, and the source table uses AWS managed key for key management.

In these cases, an extra IAM statement should be added to the policy to allow access to the particular keys(s) that the table(s) use:

{
    "Sid": "KMSAccess",
    "Effect": "Allow",
    "Action": [
        "kms:CreateGrant",
        "kms:DescribeKey",
        "kms:Decrypt"
    ],
    "Resource": [
      "<key-arn>"
    ]
}

Note: If directly choosing AWS managed key as the key management type for the new table, no extra permissions are required. The extra permissions are only needed if using the source table's configuration and the source table uses AWS managed key as DataMasque must perform extra checks to verify the actual key ARN of the source table.

<key-arn> should be replaced with the ARN of the key that is to be used. ARNs for keys can be found in the AWS KMS Console. ARNs for keys you have created are found by choosing Customer-managed keys from the sidebar, then clicking the alias or ID of the key you want to use. The ARN is displayed under General configuration. To find the AWS managed key for Amazon DynamoDB, choose AWS managed keys from the sidebar then click the aws/dynamodb alias. The ARN is displayed under General configuration.

Enabling Point-in-time Recovery Backup

Point-in-time Recovery Backup must be enabled in order for DataMasque to perform masking on Amazon DynamoDB. To enable this, follow the instructions in the official AWS Hands-on lab for Amazon DynamoDB.

Config carried over to masked tables

When masking an Amazon DynamoDB table, the new recreated table will have the following properties set to the exact values that the source table was configured with without reconstructing them.

  • KeySchema
  • AttributeDefinitions

The following properties will be set to mirror the source table.

The following properties WILL NOT be carried over from the source table.

Notes:

  • BillingMode defaults to PROVISIONED for masked tables as BillingMode is not returned when calling describe_table on PROVISIONED tables. When BillingMode is set to PAY_PER_REQUEST, this will be carried over to the masked table.
  • ProvisionedThroughput will be fixed for imported tables, auto-scaling will not be enabled.
  • LocalSecondaryIndexes will be carried over from the source table and recreated on the masked table as GlobalSecondaryIndexes. This is due the fact that LocalSecondaryIndexes can only be added when creating the table and cannot be created when importing the table from S3. These GlobalSecondaryIndexes will be created with the minimum read/write capacity (1/1) as this will affect the cost of Amazon DynamoDB.
    Please refer to the AWS Documentation relating to read/write capacity.

Connection parameters

Amazon DynamoDB specific parameters

For Amazon DynamoDB databases, the following additional fields apply:
S3 bucket name Name of the S3 bucket to temporarily store Amazon DynamoDB exports and imports.
Encryption key management The method of key management used for server-side encryption of the created masked table.
KMS key ID (optional) The ID of the key to use for SSE, if using an account-managed key.

If Cross account access is enabled the following parameter is also required:
IAM Role ARN The IAM Role ARN with access to the bucket.

For more information please refer to the cross-account documentation.

Amazon DynamoDB data is exported into a directory called, AWSDynamoDB in the root of the S3 bucket specified. This directory will be created as part of the export process.

For details on the Encryption key management and KMS key ID fields, see the section Amazon DynamoDB Encrypted Connections.

Specifying Amazon DynamoDB tables

The Amazon DynamoDB tables to mask are not specified as part of the connection settings, instead they are specified in the ruleset YAML. Refer to the table masks documentation for information on specifying table names.

Amazon DynamoDB Advanced Settings

Masking of Amazon DynamoDB creates a new table with a name based on the source table. The following fields control how the table is named, and removal of existing tables:
Masked Table Name: Suffix Masked tables are created as the source table with a random suffix. Enter a suffix to use instead of a random one.
Masked Table Name: Append datetime in ISO format Append the date and time of the masking run to the new table name (as well as a suffix, if specified).
Replace existing tables If checked, DataMasque will replace tables with the same name; otherwise, the masking run will fail if tables already exist with the generated name.

Example Generated Table Names

In this example, masking will be applied to an Amazon DynamoDB table named users. This table shows example new table names based on different combinations of the advanced settings.

Suffix Append datetime in ISO format Result New table name
unspecified Not checked Random suffix added users-603d015a1
-masked Not checked Specified suffix added users-masked
unspecified Checked Datetime added users-202303071922
-masked Checked Suffix and datetime added users-masked-202303071922

Note that the suffix is added verbatim: the leading dash is part of the suffix as a separator. This may be omitted if no separator is desired. A dash is always added before the ISO date.

Replace existing tables setting

Sometimes the name of the Amazon DynamoDB masked table may already exist. It is not very likely to happen when using a random suffix or appending the current date and time, but when specifying a suffix and masking a table more than once, the new table will already exist (unless it was removed between masking runs).

If Replace existing tables is Checked then an existing table with the same name will be deleted. A new table will be created with masked data.

If Replace existing tables is Not checked, and an existing table with the generated name already exists, then DataMasque will stop the masking run and log an error.

This table summarises the behaviour:

Replace Existing Tables Table With Generated Name Exists Outcome
Not Checked No Table created
Not Checked Yes Masking run fails
Checked No Table created
Checked Yes Existing table deleted and new table created

Note that since a suffix is always added during masking, the source table will never be deleted as the generated table name will always differ from the source table name.

Encrypted connections

All communication to Amazon DynamoDB is encrypted. Various options are available for server-side encryption (SSE) of data at rest.

Encryption settings can be specified at the connection level, which will apply to all tables when a ruleset is executed against that connection. However, the settings can be overridden on a per-table basis, with configuration specified in Run Options before starting a run. This will be looked at in more detail in the next section. First, we will explain what the options mean.

Encryption key management

Amazon DynamoDB has different options for how encryption keys are handled, and you can choose which you would like to use for SSE.

  • Owned by Amazon DynamoDB

    • AWS owned key, specifically owned and managed by Amazon DynamoDB. You are not charged an additional fee for using this key. This is the default setting in the AWS console and for DataMasque.

  • AWS managed key

    • The key is stored in your account and is managed by AWS Key Management Service (AWS KMS). AWS KMS charges apply.

  • Stored in your account, and owned and managed by you

    • The key is stored in your AWS account and is managed by AWS Key Management Service (AWS KMS). AWS KMS charges apply. If choosing this option, a KMS key ID must be specified, and your IAM privileges must grant access to the specified key.

  • Use configuration from the original table

    • Selecting this will use the same key as the source table. AWS do not recommend this. If the original table specifies AWS managed key or Stored in your account, and owned and managed by you, then AWS KMS charges apply. Additionally, your IAM profile must grant access to describe the alias/aws/dynamodb KMS key alias (in order for DataMasque to check whether a given key is managed by AWS or your account). If the original table specifies Stored in your account, and owned and managed by you then your IAM profile must grant access to the KMS key of the original table.

Please also read the Amazon DynamoDB encryption at rest documentation for more explanation on key types in AWS/DynamoDB.

KMS key ID

When selecting the encryption key management option Stored in your account, and owned and managed by you, the ID of the key must be specified. Typically, this is in an ARN format, for example arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab.

Your IAM privileges must grant access to the key. See Advanced SSE Key Permissions for more information.

Specifying SSE Options

SSE options can be configured in the Connection form and these will act as the defaults for all tables in the connection. These settings can be overridden per table, for a run, by specifying per-table settings in the Run Options panel on the Database Masking Dashboard.

Refer to the run options documentation for more information on setting options for a specific masking run.

After selecting an Amazon DynamoDB connection in the Connections list, the Override connection's Server Side Encryption settings checkbox will be available to select. To configure the settings per table, click the gear icon next to this setting, which will show the Encryption key management form.

Encryption key management form

The Table name should match the table name as specified in your ruleset. If the table name in the ruleset includes an AWS region, then it should also be included here. The options for Encryption key management and KMS key ID match those in the two prior sections.

Schema discovery

Schema discovery is supported on Amazon DynamoDB connections. An Amazon DynamoDB table declares only its key attributes, so DataMasque samples items from each table and reports the attributes it finds in them.

Schema discovery needs the dynamodb:Scan and dynamodb:ListTables permissions, which are described under Access control.

How tables are sampled

  • Tables are listed in one AWS region only: the Default region name of the connection. Tables in other regions are not discovered.
  • When the connection has no Default region name, schema discovery uses the AWS region of the environment that the DataMasque instance runs in, for example the region of its EC2 instance. The run log names the region that is used. If no region can be determined from the environment, the run fails before any table is read. Set a Default region name on the connection and run schema discovery again.
  • A table that the IAM role cannot read is skipped. The run log has an error that names the table and the permission that was refused, and the run continues with the other tables. A table whose reads Amazon DynamoDB still throttles after they have been retried is skipped in the same way, with an error that says the read was throttled. A skipped table is not in the discovery report, so the run log is the only record of why it is missing. A run fails when no table can be read.
  • A table whose name is longer than 128 characters is left out of the report. The run log has a warning that names each table left out.
  • Each table is sampled, not read in full. The sample is taken from several places spread across the table, not only from its start. The sample is not random: while the table's items, the sample size and the item count Amazon DynamoDB reports are unchanged, every run samples the same items. Adding, removing or updating items can change the sample. So can a change in the item count, which Amazon DynamoDB refreshes about every six hours, because it decides how many places the table is sampled from.
  • document_sample_size in the discovery configuration sets how many items are sampled per table. The default is 1000.
  • There is also a size limit of 64 MiB of data read per table. When a table reaches it before the requested number of items has been read, sampling of that table stops and the run log has a warning that the table was sampled only in part.
  • On a table with provisioned capacity, reads are paced to half of the table's provisioned read capacity, so discovery of a table with little read capacity is slow. The run log names each table that is paced. A table with on-demand capacity is not paced.

Schema discovery reads the tables directly. It does not export them to the S3 bucket and does not need point-in-time recovery to be enabled.

What is reported

  • Only top-level attributes are reported. A map or a list is reported as one attribute.
  • The partition key and sort key of a table are marked as its primary key.
  • An attribute found in fewer than 1% of the sampled items is left out of the report. The run log lists each attribute left out. The key attributes of a secondary index are always reported, however few of the sampled items hold them.
  • A table with no items reports only the attributes it declares: its partition key and sort key, and the key attributes of its secondary indexes. Each is reported with its declared type: str for a String, decimal for a Number and binary for a Binary. The run log says that the table is empty.
  • A partition key, a sort key or a key attribute of a secondary index is never left out of the report. A scope.skip pattern does not remove one, and neither does an ignored keyword, such as *_id, whether it is entered for one run or is a global ignored keyword. A key attribute that an ignored keyword matches is reported with no match. The run log has a warning for each table that names each such attribute a pattern or a keyword matched.
  • An attribute that exists only in items that were not sampled is not reported. A generated ruleset still redacts it when the table has a task in that ruleset.
  • All numbers are reported as decimal. See Data types in schema discovery for the full list of types.
  • An attribute that holds values of more than one type is reported with a combined type, for example decimal, str.
  • The row count of a table is not reported.
  • All tables are reported under the schema name DYNAMO_DB, as Amazon DynamoDB has no schemas. The name DYNAMO_DB is shown in the discovery report and in the run log. It is not written in a tables pattern or in a ruleset. Amazon DynamoDB connections have no schema selection in the Ruleset Generator.

In-Data Discovery is not yet supported on Amazon DynamoDB connections. In the Ruleset Generator, the In-Data Discovery switch is disabled for an Amazon DynamoDB connection. A run that uses a saved discovery configuration, the built-in defaults or a run_schema_discovery task can still have In-Data Discovery enabled. Such a run continues without In-Data Discovery: attributes are matched on their names only, no data preview is produced, and the run log has this warning:

In-Data Discovery is not yet supported on Amazon DynamoDB connections, so attributes are matched on their names only.

The run then finishes with warnings. The warning is logged once for the run, not once for each table. The options saved with the discovery result record In-Data Discovery as not enabled, whether or not the run has a discovery configuration. To run without the warning, set enabled to false in the idd_settings section of the discovery configuration.

Narrowing a discovery run

A run with no tables patterns samples every table that the IAM role can list. When an account holds many tables, list tables patterns in a discovery configuration so that only the tables you name are sampled. Amazon DynamoDB has no schemas, so a pattern is matched against the table name alone:

tables:
  - type: glob
    pattern: "customer_*"
  - type: glob
    pattern: "orders"

A table name that contains a dot is matched as a whole, so orders.v2 selects the table named orders.v2.

Generating a ruleset

The Ruleset Generator can build a ruleset from the results of schema discovery. A generated ruleset never masks a table's partition key or sort key, gives each key attribute of a secondary index a rule of its own, and has a task for every table that at least one attribute is selected from. Each task ends with a rule that redacts every attribute no other rule covers, so attributes the discovery sample did not see are not left unmasked. That rule masks Null values too: an attribute that is Null in an item is replaced with the fixed text redacted. An attribute named * gets no rule of its own, because a rule for the column * is read as that rule; it redacts the attribute instead. A table that nothing is selected from has no task. A table that schema discovery did not report is not in the generated ruleset at all: one that was skipped because the IAM role could not read it, one that a tables pattern left out, or one that did not exist when schema discovery ran. Check the run log of the schema discovery run for skipped tables. The data type of each masked value follows the mask that produced it. See Data types of masked values. See Amazon DynamoDB Tables for the details.

Limitations

When using Amazon DynamoDB, In-Data Discovery is unsupported.

Joining tables using join syntax is unsupported.

The following tasks are unavailable:

  • run_sql
  • mask_unique_key
  • truncate_table
  • build_temp_table
  • run_data_discovery

The alternative methods below can be considered in absence of the unavailable features:

The from_unique mask should be used instead of mask_unique_key.

The where option in mask_table in unsupported. Conditional masking with if should be used instead.

Supported data types

The following Amazon DynamoDB data types are supported.

Data type Description
string Unicode string with UTF-8 binary encoding. Can be a length of zero, or up to 400KB of data.
number Numerical value with 38 digits of precision.
boolean Can be true or false.
null Represents an attribute with an unknown or undefined state.
list Stores and ordered collection of values. Should be masked with a json mask.
map Stores and unordered collection of name-value pairs. Should be masked with a json mask.
binary (partial support) Store any binary data up to 400KB. While DataMasque does not have mask types for manipulating binary data, values from binary columns can be copied to other columns or replaced with non-binary values (e.g. null).

Data types of masked values

The data type a masked value is written in follows the mask that produced it. A mask that produces a number writes a Number over a Number attribute. A mask that produces a boolean writes a Boolean over a Boolean attribute. A mask that produces text writes a String, whatever the text looks like. One case also looks at the value: a number written over a Boolean attribute is written as a Boolean when it is 0 or 1. A masked value of another data type than the attribute does not fail the run.

Attribute data type Masked value Written as
Number A number Amazon DynamoDB can store, from a mask that produces numbers, such as from_random_number. Number
Number A number Amazon DynamoDB cannot store, from any mask. String
Number Text from any other mask, such as redacted, 82, 007 or 0101010101. String
Boolean true or false, or the number 0 or 1, such as the result of from_random_boolean. Boolean
Boolean Any other number, or text. The data type of the masked value
String, Binary, Null, List, Map, Set Any value. The data type of the masked value

The list below gives the masks that produce each data type, and the edge cases.

  • The masks that produce a number whatever value they mask are numeric_bucket, typecast with typecast_as set to integer, float or decimal, from_fixed with a number that is not in quotes, such as value: 82, from_random_boolean, and the date and time masks in epoch mode, such as from_random_datetime or retain_year with epoch_date and epoch_unit set. Each writes its number as a Number over a Number attribute.

  • imitate, from_unique_imitate and imitate_nz_ird produce a number when the value they mask is a number, so over a Number attribute they write a Number. imitate_unique also writes a Number over a Number attribute, except when its masked number is one Amazon DynamoDB cannot store, as described below. A mask that copies a value, such as from_column or do_nothing, produces a number when the value it copies is one.

  • from_random_number is the only mask that produces a number as text. Its text over a Number attribute is written as a Number, as described below.

  • The masks that produce a boolean are typecast with typecast_as set to boolean, and from_fixed with true or false. from_random_boolean produces the number 1 or 0, which is written over a Boolean attribute as a Boolean.

  • Every other mask produces text, which is written as a String even when it is made of digits. Among them are from_unique, from_format_string, and from_fixed with a quoted value such as '82' or '0101010101': each writes a String in every item it masks, a Number attribute included.

  • The last mask of a rule decides. In a rule with several masks, or in a chain mask, the data type is that of the result of the last mask. A typecast as the last mask therefore sets the data type either way: typecast_as: integer or typecast_as: decimal after a mask that produces digits writes a Number, and typecast_as: string after from_random_number writes a String.

  • A masked number Amazon DynamoDB cannot store is written as a String, whatever mask produced it, and the run does not fail. A key or index key attribute is the exception: it keeps the data type the table declares, so the import of a key value Amazon DynamoDB cannot store fails. It is a number with more than 38 significant digits, one outside the range a Number holds, or an infinite or not-a-number float, which a typecast to float of the text inf or nan produces. from_random_number can produce one when its max is very large, such as above 1e38, or when its decimal_places gives the number more than 38 digits in all. imitate_unique can produce one from a Number written with an exponent, such as 1E+40, because it masks the integer of the value, which has 41 digits. A random number just below zero that is rounded to -0 or -0.00 is written as the Number 0 or 0.00.

  • Each item is written in the data type the rule that last masked it decides. When several rules mask one attribute, such as rules in the branches of an if, an item masked by a rule that ends in from_random_number gets a Number, and an item masked by a rule that ends in a mask that produces text gets a String. An item that no rule masks, because a condition or a skip left it out, keeps its value and its data type. The rule for the column * follows the same rule through its own masks.

  • An attribute can therefore hold more than one data type after masking when the ruleset produces different kinds of value for it:

    • Rules in different branches of an if that end in masks of different kinds, for example from_random_number in one branch and from_fixed with the quoted text 'n/a' in the other. Each item gets the kind its branch produces, because that is what the ruleset asks for.
    • A from_file mask whose value_on_missing is a number that is not in quotes. An item the mask gives value_on_missing, because its filter matches no row of the seed file or the seed value is the null_string, gets that number as a Number, and every other item gets text from the seed file as a String.
    • A from_random_number or imitate_unique mask that can produce a number Amazon DynamoDB cannot store, as described above.
    • A from_random_number mask over an attribute that is Null or a String in some items before masking: only its text over a Number is written as a Number, so those items get a String while the items that contained a Number get a Number.

    To write one data type in every item, end each rule that masks the attribute with a typecast mask.

  • A null masked value is written as Null on an attribute of any data type.

  • A masked boolean on a Number attribute is written as a Boolean.

  • This applies to an attribute that is masked as a whole. A value masked inside a List or a Map with a json mask is written in the data type of the masked value.

  • A String or Number key attribute of the table or of a secondary index is always written in the data type its key schema declares. The run fails when the masked value of a Number key is not a number.

  • A Binary key attribute is not converted: its masked value is written in its own data type, and Amazon DynamoDB accepts only a Binary value for it. Leave a Binary key attribute unmasked, or copy a Binary value into it with a mask such as from_column. A generated ruleset gives a Binary key attribute of a secondary index a do_nothing rule, with a comment that says the attribute is not masked.

Data types in schema discovery

Schema discovery reports each attribute with one of the following data types.

Amazon DynamoDB data type Reported as
String str
Number decimal
Boolean boolean
Null none
Binary binary
List list
Map dict
String Set set
Number Set set
Binary Set set

A whole number and a fractional number are both reported as decimal, as Amazon DynamoDB has one Number type for both. An attribute that is Null in some items and has one other type in the rest is reported with that other type.