DataMasque Portal

AI Engine on Kubernetes

The Local AI Engine can run in a Kubernetes cluster on a GPU node, installed with Helm. This guide covers Amazon Elastic Kubernetes Service (EKS) and Red Hat OpenShift Service on AWS (ROSA). For what the AI Engine does and how to choose its model, see the AI Engine guide.

Note: The DataMasque AI Engine is currently a preview feature. The AI Engine on Kubernetes is tested on EKS with a g6.xlarge node (NVIDIA L4, 24 GB).

For a preview build please contact support@datamasque.com.

Overview

Layouts

The AI Engine can be used in three layouts:

  1. No AI Engine (the default). The DataMasque Helm chart installs no AI Engine and the ai_detect matcher is unavailable.
  2. Same cluster. The DataMasque Helm chart installs the AI Engine next to DataMasque when aiEngine.enabled is true. See Layout 2.
  3. Other cluster. The AI Engine runs in its own cluster, installed from the datamasque-ai-engine-only.tgz chart, and you enter its URL in DataMasque. See Layout 3.

The DataMasque Helm chart download from the DataMasque Portal holds both charts. Extract it so that the datamasque-helmchart directory is in the current directory, as for an install without the AI Engine:

  • datamasque-helmchart is the DataMasque chart. It installs DataMasque, and the AI Engine with it when aiEngine.enabled is true (layouts 1 and 2).
  • datamasque-helmchart/datamasque-ai-engine-only.tgz is the AI Engine chart on its own, packaged. It installs only the AI Engine (layout 3).

Requirements

  • One NVIDIA GPU with 24 GB of video memory for every AI Engine pod. The GPU must have compute capability sm_86, sm_89, sm_100 or sm_120. The NVIDIA T4 is not supported.
  • A GPU node with a disk of at least 100 GiB.
  • The NVIDIA driver and the NVIDIA device plugin (or the NVIDIA GPU Operator on ROSA) on that node.
  • A storage class that supports ReadWriteMany, such as the Elastic File System (EFS) class used by DataMasque on EKS and on ROSA.
  • An Elastic Container Registry (ECR) that the nodes can pull from.
  • The AI Engine package of version 1.26.11.2 or later, which holds ecr-image-push.sh, and the AI Engine models. Earlier AI Engine builds run as root and cannot read their files as the user of the chart. To get them, contact support@datamasque.com.

Preparing the Images

The cluster pulls the AI Engine image and a model image from your ECR. These steps run on a machine with aws, Docker and curl, in the extracted AI Engine package.

  1. Download the models into a new folder. The folder receives one base and one small model.
$ mkdir -p "$HOME/ai-engine-models"
$ MODEL_DOWNLOAD_URL=https://internal-prod-dm-models-public-s3.s3.ap-southeast-2.amazonaws.com/models \
  MODEL_PATH="$HOME/ai-engine-models" ./download_model.sh latest
  1. Log Docker in to your ECR. Add --region and --profile when the registry is not in your default region or profile:
$ aws ecr get-login-password --region us-east-1 --profile my-profile \
  | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

See the AWS private registry authentication guide.

  1. Push the images. The script takes the registry host without /datamasque, and the model folder. It creates the ECR repositories datamasque/local-ai-engine and datamasque/local-ai-engine-model if they do not exist. It takes the region from the registry host and uses the aws default profile, so export AWS_PROFILE first when it is not the default:
$ export AWS_PROFILE=my-profile
$ ./ecr-image-push.sh 123456789012.dkr.ecr.us-east-1.amazonaws.com "$HOME/ai-engine-models"

The script pushes both images and prints the image and model values for the chart, for example:

image:
  repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
  tag: 1.26.11.2
model:
  image:
    repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
    tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101
  baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
  smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc

Keep these values for installing the AI Engine. If a push fails, run the script again. The chart pulls only from this registry and fetches nothing from the internet.

Note: The worker nodes need read access to ECR. The AmazonEC2ContainerRegistryReadOnly policy must be attached to the node role (the worker role on ROSA).

GPU Nodes on EKS

These steps add a GPU node group to an existing EKS cluster, such as the one created in Cluster Configuration. The node group uses the Amazon Linux 2023 NVIDIA AMI, which has the NVIDIA driver and container toolkit, and a taint so that only the AI Engine runs on the GPU node. The label nvidia.com/gpu.present is the one the NVIDIA device plugin chart's default affinity selects, so the plugin runs on the GPU node only; on a CPU node it would crash with Failed to initialize NVML.

  1. Save the following as gpu-nodegroup.yaml, replacing the cluster name and region with those of your cluster:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: datamasque-cluster  # replace with your cluster name
  region: us-east-1  # replace with your region

managedNodeGroups:
  - name: ng-gpu
    amiFamily: AmazonLinux2023
    instanceType: g6.xlarge  # NVIDIA L4, 24 GB
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    volumeSize: 100  # GiB, the minimum
    volumeType: gp3
    labels:
      nvidia.com/gpu.present: "true"
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule

For a cluster that eksctl did not create, add a vpc block with the cluster's id, subnets and securityGroup, as described in the eksctl documentation. Without it, eksctl stops with VPC configuration required for creating nodegroups on clusters not owned by eksctl. When the cluster's subnets are private, also add privateNetworking: true to the node group. Without it, eksctl stops with all public subnets from <vpc id>, that the cluster was originally created on, have been deleted.

  1. Create the node group. Add --profile when you use a named AWS profile:
$ eksctl create nodegroup --config-file=gpu-nodegroup.yaml
  1. Check whether the node already offers GPUs:
$ kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

If the GPU column shows 1 for the GPU node, eksctl has already installed the NVIDIA device plugin; skip step 4. If it shows <none>, continue.

The plugin that eksctl installs, the DaemonSet nvidia-device-plugin-daemonset in kube-system, also runs on the CPU nodes. There its pods restart again and again (CrashLoopBackOff) with Failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND. This does not affect the GPU node. To run the plugin on the GPU node only, create the node group with --install-nvidia-plugin=false as in Dark Site on EKS, and install the plugin in step 4.

  1. Install the NVIDIA device plugin with Helm. Save the following as device-plugin-values.yaml. It keeps the chart's default affinity, which selects the node label from step 1, and adds a toleration for the GPU taint:
tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
$ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
$ helm repo update
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
    --namespace nvidia-device-plugin --create-namespace \
    --version 0.20.1 \
    -f device-plugin-values.yaml
  1. Check that the node offers the GPU. The GPU column shows 1 when the plugin is running:
$ kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Dark Site on EKS

A cluster without internet access cannot reach https://nvidia.github.io/k8s-device-plugin, and the images of the NVIDIA device plugin come from nvcr.io. The plugin that eksctl installs with the node group also pulls from nvcr.io. Instead, put the chart and its image in your ECR, then install the plugin from them. The version below is the one that step 4 installs.

  1. In step 2, create the node group with --install-nvidia-plugin=false, so that eksctl does not install its own plugin:
$ eksctl create nodegroup --config-file=gpu-nodegroup.yaml --install-nvidia-plugin=false
  1. On a machine with internet access, download the chart:
$ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
$ helm pull nvdp/nvidia-device-plugin --version 0.20.1
  1. On the same machine, copy the plugin image to your ECR. Replace the registry host with yours, and log Docker in to the ECR as in Preparing the Images:
$ aws ecr create-repository --repository-name k8s-device-plugin --region us-east-1
$ docker pull nvcr.io/nvidia/k8s-device-plugin:v0.20.1
$ docker tag nvcr.io/nvidia/k8s-device-plugin:v0.20.1 123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin:v0.20.1
$ docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin:v0.20.1
  1. Take nvidia-device-plugin-0.20.1.tgz to a machine that reaches the cluster, and in step 4 install from the file with the same device-plugin-values.yaml and the repository of your ECR:
$ helm upgrade -i nvdp ./nvidia-device-plugin-0.20.1.tgz \
    --namespace nvidia-device-plugin --create-namespace \
    -f device-plugin-values.yaml \
    --set image.repository=123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin

GPU Nodes on ROSA

Note: These steps are checked against the Red Hat and NVIDIA references and have not been run on a live ROSA cluster.

The AI Engine needs a machine pool of GPU nodes, the Node Feature Discovery (NFD) Operator, and the NVIDIA GPU Operator.

  1. Create a machine pool with a GPU instance type and a taint. Replace my-cluster with your cluster name. A machine pool with taints can only be created when the cluster has another machine pool without taints. The default disk size of a machine pool is 300 GiB, above the 100 GiB the AI Engine needs.
$ rosa create machinepool --cluster=my-cluster --name=gpu \
    --instance-type=g6.xlarge --replicas=1 \
    --taints=nvidia.com/gpu=true:NoSchedule

See Managing compute nodes using machine pools and the rosa CLI reference.

  1. Install the Node Feature Discovery Operator and create a NodeFeatureDiscovery instance, following Installing the NFD Operator. Check that the GPU node has the label feature.node.kubernetes.io/pci-10de.present=true:
$ oc get nodes -l feature.node.kubernetes.io/pci-10de.present=true
  1. Install the NVIDIA GPU Operator and create its ClusterPolicy, following Installing the NVIDIA GPU Operator on OpenShift. The ClusterPolicy must have real settings: a ClusterPolicy with an empty spec fails to deploy. When the policy reports State: ready, check that the node offers the GPU:
$ oc get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

The AI Engine runs under the default restricted-v2 security context constraint (SCC) of OpenShift. Do not add the anyuid policy for it, and do not create a custom SCC. The anyuid step in ROSA Deployment applies to DataMasque only.

Dark Site on ROSA

A cluster without internet access needs the operator images in a mirror registry. Mirror the NFD Operator and the NVIDIA GPU Operator catalogs with oc-mirror, following Mirroring images for a disconnected installation and NVIDIA's disconnected environment guide. The AI Engine images come from your own ECR, as in Preparing the Images.

Storage

State Storage

The AI Engine keeps the model that was chosen in Settings on a state volume. Every pod of the AI Engine shares it, so the volume must support ReadWriteMany.

  • In layout 2, the DataMasque chart creates the EFS class efs-sc-ai-engine on efsFileSystemId for the state volume (aiEngine.state.persistence.createStorageClass, default true). No state value is needed. To use a ReadWriteMany class of your own, set aiEngine.state.persistence.createStorageClass to false and aiEngine.state.persistence.storageClassName to that class. The class must not be the DataMasque class persistence.efs.StorageClassName, or helm install fails with:
  aiEngine.state.persistence.storageClassName must not be persistence.efs.StorageClassName: the state claim could bind the static DataMasque volume. Keep the default efs-sc-ai-engine class or name another ReadWriteMany class

With createStorageClass set to true, a storageClassName that names an existing class of yours fails with the following message, where <name> is that class:

  aiEngine.state.persistence.storageClassName names the existing storage class <name>, which the chart would replace: set aiEngine.state.persistence.createStorageClass to false to use it

On ROSA (platform set to openshift), the chart creates no class: set aiEngine.state.persistence.storageClassName to a ReadWriteMany class of your cluster.

  • For the standalone chart, set state.persistence.storageClassName to a ReadWriteMany class, or state.persistence.existingClaim to an existing ReadWriteMany claim. With neither set, helm install fails with:
  state.persistence.storageClassName (a ReadWriteMany class) or state.persistence.existingClaim is required

The chart deletes the state claim it creates when you uninstall the release. It never deletes an existingClaim. The efs-sc-ai-engine class keeps a volume when its claim is deleted (reclaimPolicy: Retain), so after an uninstall the volume stays Released with its EFS access point. Delete them with kubectl delete pv <name> and aws efs delete-access-point when you no longer need them.

Model Storage

By default, each pod copies the model files from the model image into an emptyDir volume every time it starts. To keep the files between restarts, set model.persistence.enabled to true. The chart then creates a claim, and a restart copies only the files that are missing or have a different size:

model:
  persistence:
    enabled: true
    storageClassName: ebs-sc
    accessMode: ReadWriteOnce
    size: 10Gi

ebs-sc is the EBS gp3 class that the DataMasque chart creates by default (persistence.ebs.StorageClassName) in layout 2. In layout 3, use an EBS class of your cluster.

Set model.persistence.existingClaim to use a claim you created. The chart never deletes it.

A ReadWriteOnce model claim allows one replica. With replicaCount above 1, helm install fails with:

model.persistence.accessMode ReadWriteOnce allows one replica; use ReadWriteMany for replicaCount > 1

Use a ReadWriteMany class and accessMode: ReadWriteMany for more than one replica. When the models are on EFS, the file system needs Elastic throughput. Check the EFS throughput mode and set it with:

$ aws efs update-file-system --file-system-id fs-00000000000000000 --throughput-mode elastic

Installing the AI Engine

The values below use the output of ecr-image-push.sh from Preparing the Images. Each value is required unless it is marked with a default or as optional.

Layout 2: Same Cluster as DataMasque

Add the following to the values.yaml of the DataMasque chart, as described in DataMasque Deployment on EKS or on ROSA. The AI Engine values go under the aiEngine key:

aiEngine:
  enabled: true
  image:
    repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
    tag: 1.26.11.2
  model:
    image:
      repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
      tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101  # optional: defaults to the two file names below without `.tar.enc`, joined with `_`
    baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
    smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc
  modelSize: base  # default: base. The model served until Settings chooses another: base (Precision) or small (Express)
  replicaCount: 1  # default: 1. One GPU per pod

If you changed namespace in the DataMasque chart, set aiEngine.namespaceOverride to the same value. Otherwise the render fails with the following message, here for namespace set to dm and aiEngine.namespaceOverride left at datamasque:

aiEngine.namespaceOverride (datamasque) must equal namespace (dm): set aiEngine.namespaceOverride=dm

Then install or upgrade DataMasque as usual:

$ helm upgrade --install datamasque ./datamasque-helmchart -n datamasque -f values.yaml

On ROSA, run the helm and kubectl commands in this guide from the DataMasque project, as in ROSA Deployment, with oc for kubectl and without -n datamasque.

The AI Engine pod requests one nvidia.com/gpu and tolerates the taint nvidia.com/gpu, so it runs on the GPU node. To place it on a node with a different taint or label, set aiEngine.podPlacement.tolerations, aiEngine.podPlacement.nodeSelector or aiEngine.podPlacement.affinity.

Check that the pod is ready:

$ kubectl get pods -n datamasque -l app.kubernetes.io/name=datamasque-ai-engine

Layout 3: Different Cluster from DataMasque

In the cluster with the GPU node, save the following as ai-engine-values.yaml. The state.persistence.storageClassName must be a ReadWriteMany class of that cluster. A load balancer makes the AI Engine reachable from the DataMasque cluster:

image:
  repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
  tag: 1.26.11.2
model:
  image:
    repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
    tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101  # optional: defaults to the two file names below without `.tar.enc`, joined with `_`
  baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
  smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc
state:
  persistence:
    storageClassName: efs-sc  # a ReadWriteMany class
service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal

The three annotations are for the AWS Load Balancer Controller, which creates an internal Network Load Balancer. A cluster without that controller uses the in-tree controller, which reads these annotations instead:

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Install the chart from the datamasque-ai-engine-only.tgz package of the DataMasque Helm chart download:

$ helm install engine ./datamasque-helmchart/datamasque-ai-engine-only.tgz -f ai-engine-values.yaml \
    --namespace datamasque-ai-engine --create-namespace
$ kubectl get service engine-ai-engine -n datamasque-ai-engine

The EXTERNAL-IP column shows the load balancer address once it exists. The AI Engine has no login, so the chart refuses a LoadBalancer Service without the internal annotations above. helm install then fails with:

service.type LoadBalancer needs an internal load balancer: set service.annotations service.beta.kubernetes.io/aws-load-balancer-type: external, service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip and service.beta.kubernetes.io/aws-load-balancer-scheme: internal (AWS Load Balancer Controller), or service.beta.kubernetes.io/aws-load-balancer-internal: "true" (in-tree controller)

Allow only the DataMasque cluster to reach port 9021. The DataMasque cluster must have a network route to the load balancer, for example through VPC peering.

The AI Engine URL

  • In layout 2, the chart gives DataMasque the URL of the AI Engine as its default. For the release datamasque in the namespace datamasque, the URL is http://datamasque-ai-engine.datamasque.svc.cluster.local:9021.
  • In layout 3, enter http://<load balancer address>:9021 as the AI Engine URL on the Settings page.

A URL saved in Settings, or through the API, always wins over the chart's default, including an empty URL. Choosing a model in Settings saves the URL with it, so the chart's default stops applying after that, even when the saved URL is the same as the chart's. A helm upgrade never changes a saved URL. To use the chart's URL again after saving another or an empty one, enter the URL in Settings.

Running More than One AI Engine Pod

Set replicaCount above 1 in layout 2 to run more pods, each with its own GPU. The pods share the state volume, so a model chosen in Settings reaches every pod. Each pod notices the change within seconds, stops taking new requests, and restarts itself with the new model once nothing is queued or running on it. On a model switch, a pod waits for its clients to fetch their asynchronous /detect results, and drops a result that no client has asked for in 30 seconds. The pods can restart at different times, and Settings shows the AI Engine as loading until a pod is ready. A pod that restarts for another reason also serves the model that was chosen.

DataMasque's agent submits a job with POST /detect/async and polls GET /detect/async/{id}. Each pod holds its jobs in memory, so a poll must reach the pod that took the job. In layout 2, the ClusterIP Service keeps each client on one AI Engine pod. In layout 3, a load balancer does not, so helm install fails for a replicaCount above 1 with:

service.type LoadBalancer allows one replica: a load balancer does not keep a client on one pod, so an async job's poll can reach another pod and get 404. Use replicaCount 1, or a ClusterIP Service for more

If the state volume is replaced with a different EFS file system or access point, the choice made in Settings is lost and the AI Engine serves the model from modelSize until Settings chooses another.

Startup Times and Memory

The AI Engine pod is ready when the model is loaded. The pod waits up to 15 minutes for that (startupProbe.periodSeconds of 10 seconds times startupProbe.failureThreshold of 90), then restarts.

The times on an EKS g6.xlarge node:

Step emptyDir models Model claim on gp3
Pull the images About 3.5 minutes on a new node, none once cached The same as emptyDir
Copy the models About 30 seconds at every start About 20 seconds at the first start, then none
Load the model About 2 minutes for the base (Precision) model About 2 minutes for the base (Precision) model

The image pull is once per node: the engine image is about 5.3 GB and the model image about 4.6 GB. With a model claim, a restart finds the files up to date and skips the copy. The memory request of the pod is resources.requests.memory, which is 10Gi. The pod used at most 8.4 GiB of memory on EKS. The chart sets no memory or CPU limit, because the taint keeps the GPU node for the AI Engine.

Upgrading the AI Engine

The AI Engine image tag (image.tag, or aiEngine.image.tag in layout 2) is separate from the DataMasque tag. Upgrading DataMasque does not upgrade the AI Engine. An AI Engine build expires on the first day of the month after the month of its version. The expiry_date field of its /health response shows the date. Renew the image before then:

  1. Get the new AI Engine package, and run ecr-image-push.sh from it as in Preparing the Images.
  2. Set the new image.tag (aiEngine.image.tag in layout 2), and a new model.image.tag and model.baseFile or model.smallFile if the script printed different ones.
  3. Upgrade the release:
$ helm upgrade datamasque ./datamasque-helmchart -n datamasque -f values.yaml

For layout 3, run helm upgrade engine ./datamasque-helmchart/datamasque-ai-engine-only.tgz -f ai-engine-values.yaml --namespace datamasque-ai-engine.

With replicaCount 1, the pod is replaced without a second pod starting first, so the AI Engine is unavailable while the new pod loads, and helm upgrade --wait returns before the new pod is ready: check the pod with kubectl get pods as below. With more replicas, the pods are replaced one at a time and the others keep serving new runs. A masking run that starts while no engine pod is ready waits up to 10 minutes for it, as described in Choosing the model. A run that is already masking with ai_detect when the engine pod it uses is replaced fails, because that pod held the run's jobs. With replicaCount 1 the run fails with Failed to connect to AI Engine, and with more replicas with AI Engine async poll (request_id=<id>) returned HTTP error - Request <id> not found. Upgrade when no such run is in progress, or start the run again once the pod is ready.

If the new pod does not become ready, go back to the previous release:

$ helm history datamasque -n datamasque
$ helm rollback datamasque -n datamasque

For layout 3, use helm history engine and helm rollback engine with --namespace datamasque-ai-engine. A rollback in layout 2 restores the whole DataMasque release. Check that the AI Engine is ready again with kubectl get pods.

Troubleshooting

Symptom Cause and fix
Pod Pending with Insufficient nvidia.com/gpu No node offers a free GPU. Check that the GPU node group exists and that the device plugin runs, with the GPU column check in GPU Nodes on EKS. Each pod needs its own GPU, so replicaCount cannot exceed the number of GPU nodes.
Pod Pending with untolerated taint The GPU node has a taint other than nvidia.com/gpu. Set aiEngine.podPlacement.tolerations (layout 2) or podPlacement.tolerations (layout 3) to match it.
State claim Pending, event reports Volume capabilities not supported The state storage class does not support ReadWriteMany, as with gp3. Use an EFS class, see State Storage.
ImagePullBackOff The node cannot pull the image. If the node role lacks the AmazonEC2ContainerRegistryReadOnly policy, attach it. Otherwise check image.repository, image.tag and model.image.tag (aiEngine.image.repository and so on in layout 2) with kubectl describe pod. A tag that was never pushed gives the same state: roll back as in Upgrading the AI Engine. DataMasque pods are not affected.
Init:Error The copy-model container found no model files. Read kubectl logs <pod> -c copy-model: no *.tar.enc in /models means the model image holds no model. Push it again with ecr-image-push.sh from a folder that download_model.sh filled. <file> is not in /models means model.baseFile or model.smallFile (aiEngine.model. in layout 2) names a file the model image does not hold: set them to the file names in the image, or push the image again.
Settings shows No answer from the AI Engine, or a run fails with did not serve its model within 10 minutes DataMasque cannot reach the URL, or the pod is not ready. Check that the pod is ready and that the URL in Settings is the one in The AI Engine URL.
Settings shows that a model file is damaged or incomplete; delete it and download it again The model image holds a damaged model file. On Kubernetes, run download_model.sh into a new folder and push the model image again with ecr-image-push.sh, as in Preparing the Images. The new image has the same tag, so set model.image.pullPolicy (aiEngine.model.image.pullPolicy in layout 2) to Always and upgrade the release; the new pod pulls the image and copies the new file.
Pod CrashLoopBackOff soon after it starts, image.tag (aiEngine.image.tag in layout 2) below 1.26.11.2 AI Engine builds before 1.26.11.2 run as root and cannot read their files as the chart's user 1000 (the log shows PermissionError: [Errno 13] Permission denied: '/.cache'). Use 1.26.11.2 or later.
Settings shows Build has expired The AI Engine build is past its expiry. Renew the image as in Upgrading the AI Engine.