DataMasque Portal

DataMasque Installation on AWS Elastic Container Service (ECS) Fargate

DataMasque can be deployed on Amazon ECS Fargate using Elastic File System (EFS) as the persistent storage layer. This page details how to deploy DataMasque using publicly-available Terraform templates.

Contents

Deployment Overview

DataMasque is deployed on ECS Fargate using Infrastructure as Code with Terraform. The Terraform templates are available in DataMasque's public GitHub repository:

https://github.com/datamasque/DataMasque-AWS-ECS-Deployment

This Terraform-based deployment process automates the creation of the following resources:

  • ECS Fargate cluster
  • ECS Services and Task Definitions
  • EFS filesystem and access point
  • Required IAM roles for ECS tasks
  • Amazon RDS PostgreSQL instance
  • Security groups for ECS and RDS
  • AWS Cloud Map service discovery namespace and service entries

Note: In contrast to other platform deployments, the internal database used by the DataMasque application must be deployed as an external PostgreSQL database. For this purpose, the Terraform plan deploys an Amazon RDS database within the same VPC as the ECS cluster.

Prerequisites

Before deploying DataMasque on ECS:

  • Ensure you have the required software tools installed.
  • Create repositories in your private ECR (if not using the DataMasque public ECR).
  • Create and verify a certificate for the Application Load Balancer in AWS Certificate Manager.
  • Determine what version of DataMasque you're deploying.
  • If using a private ECR, download the release package.

Required Software

Before starting the deployment, ensure the following software is installed:

Container Image Repository

DataMasque container images can be deployed from either our public ECR repositories or your own private ECR. Choose the option that best fits your environment.

Public ECR

Container images for the DataMasque application running on ECS are available in DataMasque's public ECR in the us-east-1 region.

  • public.ecr.aws/u4t6n7u6/datamasque/in-flight-server
  • public.ecr.aws/u4t6n7u6/datamasque/agent-queue
  • public.ecr.aws/u4t6n7u6/datamasque/agent
  • public.ecr.aws/u4t6n7u6/datamasque/admin-server
  • public.ecr.aws/u4t6n7u6/datamasque/admin-frontend

Private ECR

To install from a private ECR, create the following ECR repositories:

  • <ECR host>/<prefix>/admin-db
  • <ECR host>/<prefix>/admin-frontend
  • <ECR host>/<prefix>/admin-server
  • <ECR host>/<prefix>/agent
  • <ECR host>/<prefix>/agent-queue
  • <ECR host>/<prefix>/in-flight-server

The <ECR host> will be similar to 123456789012.dkr.ecr.us-east-1.amazonaws.com, containing your AWS account ID and region.

The <prefix> can be anything you like; in this documentation, the prefix datamasque is used as an example.

With these example values, the first repository would be 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/admin-db, and so on.

Pushing Images to Private ECR

Extract the DataMasque Docker package and cd into the installation directory:

$ tar -xvf datamasque-docker-v<version>.pkg
$ cd datamasque/<version>/

Authenticate with your ECR:

$ docker login -u AWS -p $(aws ecr get-login-password) <ecr_host>

Push the images to ECR using the ecr-image-push.sh command. It must be called with your ECR host as the first and only argument. For example:

$ ./ecr-image-push.sh 123456789012.dkr.ecr.us-east-1.amazonaws.com

Finally, verify that the images are uploaded to your ECR using the AWS Console.

Deployment Steps

Step 1: Clone the Terraform Repository

Use git to obtain a local checkout of DataMasque's Terraform repository.

git clone https://github.com/datamasque/DataMasque-AWS-ECS-Deployment.git
cd DataMasque-AWS-ECS-Deployment

Step 2: Update Terraform Backend Configuration

Update the parameters specific to Terraform State in the cluster-config/backend.tf file based on your environment. For more information, refer to the Terraform documentation: https://developer.hashicorp.com/terraform/language/backend/s3

Below is the example backend.tf. Replace the parameter values based on your environment.

terraform {
  backend "s3" {
    key            = "datamasque-ecs/tfstate"
    bucket         = "bucket-name"
    use_lockfile   = true
    acl            = "bucket-owner-full-control"
    region         = "ap-southeast-2"
  }
}

Step 3: Update Configuration Files

Update the YAML configuration files to match your environment.

Environment Configuration File

The config/<environment_name>.yml file defines deployment parameters specific to your environment. The filename must exactly match the Terraform workspace name, plus .yml extension. For example, if the workspace is called production, create a file named production.yml in the config directory.

The example shown below is for the public ECR:

ecs:
  clusters:
    datamasque-ecs:  # Name of the ECS Fargate cluster. NB: It is not possible to change this after deployment.
      dnsNamespace: datamasque  # AWS Cloud Map namespace. NB: It is not possible to change this after deployment.
      rds:
        multiAz: false
      albCertificate: xxxx-xxxx-xxxx-xxxx-xxxx  # UUID of the certificate in AWS Certificate Manager
      ecr:
        ecrRepoName: datamasque
        ecrImageTag: 2-28-0-final-xxxxx  # DataMasque image tag
        ecrRepo: public
      masqueVersion: "2.28.0"  # DataMasque version being deployed
      loggingLevel: "INFO"  # DataMasque logging level
      agentContainer:
        cpu: 2048  # CPU allocation for DataMasque agent container (in units, where 1024 = one vCPU)
        memory: 4096  # Memory allocation for DataMasque agent container (in MB)
        desiredCount: 1  # Number of agent containers to deploy
      inflightContainer:
        cpu: 512  # CPU allocation for DataMasque inflight container (in units)
        memory: 2048  # Memory allocation for DataMasque inflight container (in MB)
        desiredCount: 1  # Number of inflight containers to deploy

The ecrImageTag can be determined by examining the contents of the ECR, or reading it from the DataMasque installation package:

$ tar -tvf datamasque-docker-v<version>.pkg | grep admin-server\: | awk -F':' '{print $NF}'

Note: It is not possible to change the name of the cluster or DNS namespace after deployment.

Note: The above specifications (2 vCPUs and 4GB RAM for the DataMasque agent, and 0.5 vCPUs and 2GB RAM for in-flight masking) are the minimum supported values. You can increase these resource allowances if required.

Note: If the platform is licensed through AWS License Manager, deploying more than one agent and/or more than one in-flight masking container requires additional licenses. Refer to the Licensing page for more information.

For a private ECR, edit the ecr block as follows:

ecr:
  ecrRepoName: <prefix>  # Your ECR repo prefix from above
  ecrImageTag: 2-28-0-final-xxxxx  # DataMasque image tag
  ecrRepo: private

The ECR host and region are taken from your currently selected AWS profile.

Network Configuration File

The config/common_configs.yml file contains environment-specific VPC, subnet, and network settings. Ensure the environment name matches the Terraform workspace name and the name of the deployment parameters file.

production:
  vpcid: "vpc-xxxx"  # VPC ID where DataMasque is being deployed
  db_subnetgroup: "datamasque-db-subnet-group"  # Name of the db subnet group in which to deploy RDS. Must already exist.
  inbound_cidr_range: 10.0.0.0/16  # CIDR range to allow connections to DataMasque application
  subnets:  # Subnet IDs in the VPC. At least one subnet is required.
    subneta: "subnet-xxxx"
    subnetb: "subnet-yyyy"
    subnetc: "subnet-zzzz"

For subnets, at least one subnet is required; normally there are three, one for each availability zone in your AWS region. The subnet IDs can be viewed in your VPC configuration in AWS Console; they have the form subnet-xxxx where xxxx is a 17-character hexadecimal string.

Step 4: (Optional) Edit IAM Role permissions

The Terraform code provisions an ECS Task IAM Role that is assumed by the DataMasque application containers to interact with various AWS services. The default role includes broad permissions that will work out-of-the-box for ease of deployment, but we strongly recommend limiting the permissions to only the required resources in a production environment.

The task IAM role definition can be found in the cluster-config/iam.tf file in your checked-out Terraform repository. You can edit this as necessary to limit the resource access available to DataMasque, using the principle of least privilege.

The default task role includes the following permissions:

  • Basic operation and troubleshooting
    • Mount EFS volumes for persistent storage
    • Enable SSH access via Session Manager
  • DataMasque licensing
    • Check out and check in licenses
    • List and describe ECS tasks
  • File masking, and database masking that uses S3 (Snowflake, Dynamo, Redshift)
    • Read/write access to all S3 buckets
  • AWS Secrets Manager integration
    • List and retrieve all secrets in the region
  • AWS Step Functions integration
    • Read and execute all step functions

You can restrict the permissions as necessary, and even remove the S3, Secrets Manager and/or Step Functions permissions if you don't intend to use these features. However, the EFS, ECS tasks, and License Manager permissions are mandatory for using DataMasque. Enabling SSH access via Session Manager is also strongly recommended for ease of troubleshooting issues with your DataMasque installation.

Step 5: Deploy DataMasque

Select the correct AWS profile and Terraform workspace, then execute terraform plan. The Terraform workspace name must match the naming of the environment configuration file and content of the network configuration file, as per the above instructions.

export AWS_PROFILE=<AWS profile name>
cd cluster-config
terraform init
terraform workspace select -or-create production
terraform plan

If the above commands succeed, terraform plan should show that there are 50 resources to be created. Verify the output looks correct, then run terraform apply to create the ECS cluster. The apply operation takes about 5-10 minutes.

Step 6: Verify Deployment

In the ECS console, click on the cluster. Check all services show as green - there should be five services, all showing 1/1 Tasks running. The cluster can take a few minutes to reach this state after the terraform apply completes.

Once all tasks have reached a running state, you should be able to access the DataMasque UI, as described in the next section.

Accessing DataMasque

Using the Application Load Balancer

Identify the Load Balancer associated with the frontend-service task:

  1. In AWS ECS Console, select your cluster.
  2. Under Services, click on the <cluster name>-frontend-service entry.
  3. In the Status section, under Load Balancer health, click on the load balancer.
  4. The hostname is in the Details section, under DNS name.

The hostname has the format internal-<cluster name>-dm-alb-xxxxxx.<region>.elb.amazonaws.com, where xxxxxx is a series of digits.

Ensure the load balancer's security group permits any access you might need. By default, the security group allows traffic from the subnet configured in the inbound_cidr_range in the network configuration file (common_configs.yml). If you need additional access, modify the ALB security group's inbound rules. Click on the Security tab, then select the security group. You can modify the inbound rules as required.

Important: The Terraform deployment creates three security groups: one for the load balancer, one for the cluster, and one for the RDS database. You only need to grant access on the load balancer's security group. Do not modify the other two security groups.

Access DataMasque using this hostname: https://<ALB hostname>/. The port is the default HTTPS port of 443, so does not need to be specified. Be sure to use HTTPS, not HTTP.

Using DNS

In your DNS hosted zone, you can assign a DNS entry pointing to the Load Balancer address. This will allow you to use a more convenient hostname to access DataMasque.

Important: If you plan to use a custom hostname, configure and test the DNS record before performing the first-time installation, or add the planned hostname to the allowed hostnames at the bottom left of the installation page. If you set this up later, you will need to use the original hostname to log in to DataMasque, then edit the allowed hostnames on the Settings page.

Updating the Configuration

To update the deployment configuration, for example to add additional agent containers, allocate more CPU and memory resources, or update the ECS Task IAM Role permissions, simply perform the above steps again - edit the configuration, then run terraform plan and terraform apply.

Warning! Do NOT use terraform destroy. This will irreversibly delete your entire DataMasque installation, including any connections, rulesets, and other data. You only need to use terraform plan and terraform apply to update the deployment.

Note: It is not possible to change the name of the cluster or DNS namespace after deployment.

Note: To help avoid state conflicts and stuck locks, ensure terraform commands for this deployment are only run by one user at a time.

Upgrading

To upgrade DataMasque:

  • If using a private ECR, push the images using the ecr-image-push.sh script as documented above.
  • Update the ecrImageTag and masqueVersion fields in the environment configuration file.
  • Run terraform plan and terraform apply.

Each container will be destroyed and replaced with the new version. However, the RDS database will not be replaced, so all existing connections and rulesets are retained. It is normal for the containers to restart, possibly several times, during upgrade.

Switching from a private ECR to DataMasque's public ECR, or vice versa, is possible. Provided the images are all the same version, it doesn't matter which ECR they come from.

Upgrading from v2.x to v3.x

License Changes

  • File-based license: Contact DataMasque support for a new v3.x license.
  • Cloud contract: If you have the option to switch to AWS Contract License in the My Account page, your license should continue working without requiring a new license file.

Note: New licenses are provided at the same term, free of charge.

Database

No database migration is required for ECS deployments. The RDS database is managed independently and is compatible with DataMasque v3.x.

IAM Permissions

There are three distinct sets of IAM permissions involved when deploying DataMasque on ECS:

  • Deployment permissions: Associated with the AWS profile you are using to run terraform commands. They allow Terraform to create, modify, and delete AWS resources as necessary.
  • ECS Task Execution Role permissions: Defined the Terraform template and should not be changed. Permissions for this role allow the cluster to operate at a high level: start and stop containers, access EFS volumes, read secret values created by Terraform, and write logs to Amazon Cloudwatch.
  • ECS Task Role permissions: Defined in the Terraform template. For extra security, you can customize these permissions if needed. These permissions apply to the DataMasque containers running in ECS, allowing them to access S3 buckets for file masking, check out licenses from AWS License Manager, and so on.

IAM Permissions Required for Deployment

The following IAM permissions are required for deploying DataMasque on ECS using Terraform. Create an IAM policy with these permissions and attach it to the user or role running Terraform:

Note: Replace YOUR-TERRAFORM-STATE-BUCKET with your actual S3 bucket name used for Terraform state management. This is the same bucket name that you configured in backend.tf. Note: Replace YOUR-ECS-CLUSTER-NAME with the name of the ECS cluster, as defined in the config/<terraform workspace name>.yml file (datamasque-ecs in the example configuration above).

IAM Permissions required for Terraform deployment

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TerraformStateManagement",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR-TERRAFORM-STATE-BUCKET/*",
                "arn:aws:s3:::YOUR-TERRAFORM-STATE-BUCKET"
            ]
        },
        {
            "Sid": "ECSManagement",
            "Effect": "Allow",
            "Action": [
                "ecs:CreateCluster",
                "ecs:DeleteCluster",
                "ecs:DescribeClusters",
                "ecs:RegisterTaskDefinition",
                "ecs:DeregisterTaskDefinition",
                "ecs:DescribeTaskDefinition",
                "ecs:CreateService",
                "ecs:UpdateService",
                "ecs:DeleteService",
                "ecs:DescribeServices",
                "ecs:ListServices",
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:TagResource"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ECRAccess",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ECRRepositoryManagement",
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository",
                "ecr:DeleteRepository",
                "ecr:DescribeRepositories",
                "ecr:PutImageTagMutability",
                "ecr:SetRepositoryPolicy"
            ],
            "Resource": "arn:aws:ecr:*:*:repository/datamasque/*"
        },
        {
            "Sid": "IAMRoleManagement",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:AttachRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PutRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:ListRolePolicies",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfilesForRole",
                "iam:TagRole",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::*:role/YOUR-ECS-CLUSTER-NAME-*"
            ]
        },
        {
            "Sid": "IAMPolicyManagement",
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:DeletePolicy",
                "iam:ListPolicyVersions",
                "iam:GetPolicy",
                "iam:GetPolicyVersion"
            ],
            "Resource": [
                "arn:aws:iam::*:policy/datamasque-*"
            ]
        },
        {
            "Sid": "IAMManagedPolicyAccess",
            "Effect": "Allow",
            "Action": [
                "iam:ListEntitiesForPolicy"
            ],
            "Resource": "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
        },
        {
            "Sid": "SecretsManagerManagement",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:DeleteSecret",
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:PutSecretValue",
                "secretsmanager:UpdateSecret",
                "secretsmanager:TagResource",
                "secretsmanager:GetResourcePolicy"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:YOUR-ECS-CLUSTER-NAME-*"
        },
        {
            "Sid": "EFSManagement",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:CreateFileSystem",
                "elasticfilesystem:DeleteFileSystem",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:CreateAccessPoint",
                "elasticfilesystem:DeleteAccessPoint",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:CreateMountTarget",
                "elasticfilesystem:DeleteMountTarget",
                "elasticfilesystem:DescribeMountTargets",
                "elasticfilesystem:DescribeMountTargetSecurityGroups",
                "elasticfilesystem:ModifyMountTargetSecurityGroups",
                "elasticfilesystem:PutFileSystemPolicy",
                "elasticfilesystem:DeleteFileSystemPolicy",
                "elasticfilesystem:TagResource",
                "elasticfilesystem:DescribeLifecycleConfiguration"
            ],
            "Resource": "*"
        },
        {
            "Sid": "RDSManagement",
            "Effect": "Allow",
            "Action": [
                "rds:CreateDBInstance",
                "rds:DeleteDBInstance",
                "rds:DescribeDBInstances",
                "rds:ModifyDBInstance",
                "rds:CreateDBSubnetGroup",
                "rds:DeleteDBSubnetGroup",
                "rds:DescribeDBSubnetGroups",
                "rds:AddTagsToResource",
                "rds:ListTagsForResource",
                "rds:RemoveTagsFromResource"
            ],
            "Resource": "*"
        },
        {
            "Sid": "NetworkingRead",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSecurityGroupRules",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "SecurityGroupManagement",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:CreateTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ServiceDiscovery",
            "Effect": "Allow",
            "Action": [
                "servicediscovery:CreatePrivateDnsNamespace",
                "servicediscovery:CreateService",
                "servicediscovery:DeleteService",
                "servicediscovery:DeleteNamespace",
                "servicediscovery:GetNamespace",
                "servicediscovery:GetService",
                "servicediscovery:GetOperation",
                "servicediscovery:ListNamespaces",
                "servicediscovery:ListServices",
                "servicediscovery:RegisterInstance",
                "servicediscovery:DeregisterInstance",
                "servicediscovery:TagResource",
                "servicediscovery:ListTagsForResource"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Route53ForServiceDiscovery",
            "Effect": "Allow",
            "Action": [
                "route53:CreateHostedZone",
                "route53:DeleteHostedZone",
                "route53:GetHostedZone",
                "route53:ListHostedZones",
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets",
                "route53:GetChange"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ELBManagement",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerAttributes",
                "elasticloadbalancing:DeleteListener"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudWatchLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:DeleteLogGroup",
                "logs:CreateLogStream",
                "logs:DeleteLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutRetentionPolicy",
                "logs:TagResource",
                "logs:ListTagsForResource"
            ],
            "Resource": "*"
        }
    ]
}

ECS Task IAM Role

As mentioned in the deployment steps above, the Terraform template contains a ECS Task IAM Role with fairly broad permissions. This is suitable for getting started with DataMasque on ECS, but for a production deployment, you should restrict the permissions to only the necessary resources.

For production environments, apply the principle of least privilege to grant only the necessary permissions to AWS resources. An example policy document is shown below. Replace the resource ARNs with the specific resources required for your DataMasque deployment. You can remove sections relating to Secrets Manager, Step Functions, and/or S3 access if you do not intend to use these features.

Note: The resource names in this policy (e.g., my-bucket, my-secret-prefix, my-ecs-cluster) are examples only. Replace them with your actual resource names.

Task Role Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3ObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*"
            ]
        },
        {
            "Sid": "S3BucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:GetBucketPolicyStatus",
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket"
            ]
        },
        {
            "Sid": "SecretsManagerListAll",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "SecretsManagerGetSecret",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:my-secret-prefix/*"
        },
        {
            "Sid": "LicenseManager",
            "Effect": "Allow",
            "Action": [
                "license-manager:CheckoutLicense",
                "license-manager:CheckInLicense"
            ],
            "Resource": "*"
        },
        {
            "Sid": "StepFunctionsListAll",
            "Effect": "Allow",
            "Action": [
                "states:ListStateMachines"
            ],
            "Resource": "*"
        },
        {
            "Sid": "StepFunctionsExecute",
            "Effect": "Allow",
            "Action": [
                "states:ListExecutions",
                "states:StartExecution"
            ],
            "Resource": "arn:aws:states:*:*:stateMachine:my-state-machine-*"
        },
        {
            "Sid": "ECSAccess",
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeTasks",
                "ecs:ListTasks"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ecs:cluster": "arn:aws:ecs:*:*:cluster/my-ecs-cluster"
                }
            }
        },
        {
            "Sid": "EFSAccess",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:ClientMount",
                "elasticfilesystem:ClientWrite"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "elasticfilesystem:AccessPointArn": "arn:aws:elasticfilesystem:*:*:access-point/fsap-*"
                }
            }
        },
        {
            "Sid": "SessionManager",
            "Effect": "Allow",
            "Action": [
                "ssm:UpdateInstanceInformation",
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
        }
    ]
}

Troubleshooting ECS Installation

The following provides some guidance for troubleshooting ECS installation failures. For general DataMasque troubleshooting, refer to Troubleshooting.

Terraform deployment failures

Terraform errors are normally caused by AWS permission issues. Ensure the AWS user or role associated with the selected profile has sufficient permissions to create the ECS tasks and all associated resources.

Resolution

You can retry the terraform plan and terraform apply commands after a failure and Terraform will pick up where it left off. For initial deployments only, in the event that retrying a terraform apply fails, you can run terraform destroy to delete all resources, then retry the terraform apply.

Warning! Do NOT use terraform destroy if you have encountered errors modifying an existing installation. This will irreversibly delete your entire DataMasque installation, including any connections, rulesets, and other data. If you cannot resolve the issue, contact DataMasque support for assistance.

Note: Only delete resources directly in AWS Console as a last resort, when even multiple terraform destroy commands fail to clean everything up without error.

Stuck Terraform locking

When a Terraform operation is interrupted due to network issues, credential expiration, or other failures, Terraform may leave the state locked. This prevents further operations until the state is unlocked.

Subsequent Terraform commands may fail with an error like:

Error: Error acquiring the state lock
Lock Info:
  ID: 6db9f459-f291-262b-0759-28c2aa1f1e7a
  Path: your-state-path
  Operation: OperationTypeApply
  Who: username@hostname
  Version: 1.x.x
  Created: 2025-07-30 05:19:34.35955352 +0000 UTC

Resolution

First, confirm the state is actually locked by running a read-only command such as terraform plan:

terraform plan

You should receive a similar error to the above.

Next, use the lock ID from the error message to force unlock:

terraform force-unlock <LOCK_ID>

For example: terraform force-unlock 6db9f459-f291-262b-0759-28c2aa1f1e7a

Test that Terraform can now access the state:

terraform plan -lock=false

The -lock=false flag prevents acquiring a new lock during the test.

If the terraform plan succeeds, you can retry the terraform apply.

Containers cannot pull images

This manifests as ECS Console errors such as:

  • CannotPullContainerError: Error response from daemon: pull access denied
  • CannotPullContainerError: manifest for <image:tag> not found
  • CannotPullContainerError: Get https://<ecr-url>/v2/: net/http: request timeout

You may also see that containers are in STOPPED state for the following reasons:

  • repository-url not found
  • manifest unknown
  • no basic auth credentials
  • timeout

Resolution

  • Check the ecrImageTag, ecrRepo and ecrRepoRegion values in the environment configuration file.
  • If using a private ECR, check connectivity and access permissions for the ECR.

One or more containers cyclically restart

  • Check the deployment was successful - did terraform apply issue any error messages?
  • Check the logs in the task. Note that it is normal for the frontend service to fail to start if the admin-server container and/or the in-flight-server container is not running, so look at those containers' logs for the root cause.
  • Ensure the images are up-to-date, and have consistent versions.
  • Check the status of the Postgres RDS.
  • Ensure there are no security group configurations that are firewalling traffic to or from the RDS, and ensure the subnet configuration in the config/common_configs.yml file is correct.

Next Steps

It is recommended you back up your Terraform configuration files, for example in a version control system.

Then:

  • Perform the Post-Installation Setup.
  • Configure licensing through AWS License Manager, if your DataMasque deployment is licensed this way. You can do this by visiting the My Account page and selecting the AWS Contract License option. For more information, refer to the Licensing page.