AI Engine on Kubernetes
The Local AI Engine can run in a Kubernetes cluster on a GPU node, installed with Helm. This guide covers Amazon Elastic Kubernetes Service (EKS) and Red Hat OpenShift Service on AWS (ROSA). For what the AI Engine does and how to choose its model, see the AI Engine guide.
Note: The DataMasque AI Engine is currently a preview feature. The AI Engine on Kubernetes is tested on EKS with a
g6.xlargenode (NVIDIA L4, 24 GB).For a preview build please contact support@datamasque.com.
- Overview
- Preparing the Images
- GPU Nodes on EKS
- GPU Nodes on ROSA
- Storage
- Installing the AI Engine
- Running More than One AI Engine Pod
- Startup Times and Memory
- Upgrading the AI Engine
- Troubleshooting
Overview
Layouts
The AI Engine can be used in three layouts:
- No AI Engine (the default). The DataMasque Helm chart installs no AI Engine and the
ai_detectmatcher is unavailable. - Same cluster. The DataMasque Helm chart installs the AI Engine next to DataMasque when
aiEngine.enabledistrue. See Layout 2. - Other cluster. The AI Engine runs in its own cluster, installed from the
datamasque-ai-engine-only.tgzchart, and you enter its URL in DataMasque. See Layout 3.
The DataMasque Helm chart download from the DataMasque Portal holds both charts.
Extract it so that the datamasque-helmchart directory is in the current directory, as for an install without the AI Engine:
datamasque-helmchartis the DataMasque chart. It installs DataMasque, and the AI Engine with it whenaiEngine.enabledistrue(layouts 1 and 2).datamasque-helmchart/datamasque-ai-engine-only.tgzis the AI Engine chart on its own, packaged. It installs only the AI Engine (layout 3).
Requirements
- One NVIDIA GPU with 24 GB of video memory for every AI Engine pod.
The GPU must have compute capability
sm_86,sm_89,sm_100orsm_120. The NVIDIA T4 is not supported. - A GPU node with a disk of at least 100 GiB.
- The NVIDIA driver and the NVIDIA device plugin (or the NVIDIA GPU Operator on ROSA) on that node.
- A storage class that supports
ReadWriteMany, such as the Elastic File System (EFS) class used by DataMasque on EKS and on ROSA. - An Elastic Container Registry (ECR) that the nodes can pull from.
- The AI Engine package of version 1.26.11.2 or later, which holds
ecr-image-push.sh, and the AI Engine models. Earlier AI Engine builds run as root and cannot read their files as the user of the chart. To get them, contact support@datamasque.com.
Preparing the Images
The cluster pulls the AI Engine image and a model image from your ECR.
These steps run on a machine with aws, Docker and curl, in the extracted AI Engine package.
- Download the models into a new folder. The folder receives one base and one small model.
$ mkdir -p "$HOME/ai-engine-models"
$ MODEL_DOWNLOAD_URL=https://internal-prod-dm-models-public-s3.s3.ap-southeast-2.amazonaws.com/models \
MODEL_PATH="$HOME/ai-engine-models" ./download_model.sh latest
- Log Docker in to your ECR. Add
--regionand--profilewhen the registry is not in your default region or profile:
$ aws ecr get-login-password --region us-east-1 --profile my-profile \
| docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
See the AWS private registry authentication guide.
- Push the images. The script takes the registry host without
/datamasque, and the model folder. It creates the ECR repositoriesdatamasque/local-ai-engineanddatamasque/local-ai-engine-modelif they do not exist. It takes the region from the registry host and uses theawsdefault profile, so exportAWS_PROFILEfirst when it is not the default:
$ export AWS_PROFILE=my-profile
$ ./ecr-image-push.sh 123456789012.dkr.ecr.us-east-1.amazonaws.com "$HOME/ai-engine-models"
The script pushes both images and prints the image and model values for the chart, for example:
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
tag: 1.26.11.2
model:
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101
baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc
Keep these values for installing the AI Engine. If a push fails, run the script again. The chart pulls only from this registry and fetches nothing from the internet.
Note: The worker nodes need read access to ECR. The
AmazonEC2ContainerRegistryReadOnlypolicy must be attached to the node role (the worker role on ROSA).
GPU Nodes on EKS
These steps add a GPU node group to an existing EKS cluster, such as the one created in
Cluster Configuration.
The node group uses the Amazon Linux 2023 NVIDIA AMI, which has the NVIDIA driver and container toolkit,
and a taint so that only the AI Engine runs on the GPU node.
The label nvidia.com/gpu.present is the one the NVIDIA device plugin chart's default affinity selects,
so the plugin runs on the GPU node only; on a CPU node it would crash with Failed to initialize NVML.
- Save the following as
gpu-nodegroup.yaml, replacing the cluster name and region with those of your cluster:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: datamasque-cluster # replace with your cluster name
region: us-east-1 # replace with your region
managedNodeGroups:
- name: ng-gpu
amiFamily: AmazonLinux2023
instanceType: g6.xlarge # NVIDIA L4, 24 GB
minSize: 1
maxSize: 1
desiredCapacity: 1
volumeSize: 100 # GiB, the minimum
volumeType: gp3
labels:
nvidia.com/gpu.present: "true"
taints:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
For a cluster that eksctl did not create, add a vpc block with the cluster's id, subnets and securityGroup,
as described in the eksctl documentation.
Without it, eksctl stops with VPC configuration required for creating nodegroups on clusters not owned by eksctl.
When the cluster's subnets are private, also add privateNetworking: true to the node group.
Without it, eksctl stops with all public subnets from <vpc id>, that the cluster was originally created on, have been deleted.
- Create the node group. Add
--profilewhen you use a named AWS profile:
$ eksctl create nodegroup --config-file=gpu-nodegroup.yaml
- Check whether the node already offers GPUs:
$ kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
If the GPU column shows 1 for the GPU node, eksctl has already installed the NVIDIA device plugin; skip step 4.
If it shows <none>, continue.
The plugin that eksctl installs, the DaemonSet nvidia-device-plugin-daemonset in kube-system, also runs on the CPU nodes.
There its pods restart again and again (CrashLoopBackOff) with Failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND.
This does not affect the GPU node.
To run the plugin on the GPU node only, create the node group with --install-nvidia-plugin=false
as in Dark Site on EKS, and install the plugin in step 4.
- Install the NVIDIA device plugin with Helm.
Save the following as
device-plugin-values.yaml. It keeps the chart's default affinity, which selects the node label from step 1, and adds a toleration for the GPU taint:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
$ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
$ helm repo update
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--namespace nvidia-device-plugin --create-namespace \
--version 0.20.1 \
-f device-plugin-values.yaml
- Check that the node offers the GPU. The
GPUcolumn shows1when the plugin is running:
$ kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
Dark Site on EKS
A cluster without internet access cannot reach https://nvidia.github.io/k8s-device-plugin, and the images of the
NVIDIA device plugin come from nvcr.io.
The plugin that eksctl installs with the node group also pulls from nvcr.io.
Instead, put the chart and its image in your ECR, then install the plugin from them.
The version below is the one that step 4 installs.
- In step 2, create the node group with
--install-nvidia-plugin=false, so thateksctldoes not install its own plugin:
$ eksctl create nodegroup --config-file=gpu-nodegroup.yaml --install-nvidia-plugin=false
- On a machine with internet access, download the chart:
$ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
$ helm pull nvdp/nvidia-device-plugin --version 0.20.1
- On the same machine, copy the plugin image to your ECR. Replace the registry host with yours, and log Docker in to the ECR as in Preparing the Images:
$ aws ecr create-repository --repository-name k8s-device-plugin --region us-east-1
$ docker pull nvcr.io/nvidia/k8s-device-plugin:v0.20.1
$ docker tag nvcr.io/nvidia/k8s-device-plugin:v0.20.1 123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin:v0.20.1
$ docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin:v0.20.1
- Take
nvidia-device-plugin-0.20.1.tgzto a machine that reaches the cluster, and in step 4 install from the file with the samedevice-plugin-values.yamland the repository of your ECR:
$ helm upgrade -i nvdp ./nvidia-device-plugin-0.20.1.tgz \
--namespace nvidia-device-plugin --create-namespace \
-f device-plugin-values.yaml \
--set image.repository=123456789012.dkr.ecr.us-east-1.amazonaws.com/k8s-device-plugin
GPU Nodes on ROSA
Note: These steps are checked against the Red Hat and NVIDIA references and have not been run on a live ROSA cluster.
The AI Engine needs a machine pool of GPU nodes, the Node Feature Discovery (NFD) Operator, and the NVIDIA GPU Operator.
- Create a machine pool with a GPU instance type and a taint.
Replace
my-clusterwith your cluster name. A machine pool with taints can only be created when the cluster has another machine pool without taints. The default disk size of a machine pool is 300 GiB, above the 100 GiB the AI Engine needs.
$ rosa create machinepool --cluster=my-cluster --name=gpu \
--instance-type=g6.xlarge --replicas=1 \
--taints=nvidia.com/gpu=true:NoSchedule
See Managing compute nodes using machine pools
and the rosa CLI reference.
- Install the Node Feature Discovery Operator and create a
NodeFeatureDiscoveryinstance, following Installing the NFD Operator. Check that the GPU node has the labelfeature.node.kubernetes.io/pci-10de.present=true:
$ oc get nodes -l feature.node.kubernetes.io/pci-10de.present=true
- Install the NVIDIA GPU Operator and create its
ClusterPolicy, following Installing the NVIDIA GPU Operator on OpenShift. TheClusterPolicymust have real settings: aClusterPolicywith an emptyspecfails to deploy. When the policy reportsState: ready, check that the node offers the GPU:
$ oc get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
The AI Engine runs under the default restricted-v2 security context constraint (SCC) of OpenShift.
Do not add the anyuid policy for it, and do not create a custom SCC.
The anyuid step in ROSA Deployment applies to DataMasque only.
Dark Site on ROSA
A cluster without internet access needs the operator images in a mirror registry.
Mirror the NFD Operator and the NVIDIA GPU Operator catalogs with oc-mirror, following
Mirroring images for a disconnected installation
and NVIDIA's disconnected environment guide.
The AI Engine images come from your own ECR, as in Preparing the Images.
Storage
State Storage
The AI Engine keeps the model that was chosen in Settings on a state volume.
Every pod of the AI Engine shares it, so the volume must support ReadWriteMany.
- In layout 2, the DataMasque chart creates the EFS class
efs-sc-ai-engineonefsFileSystemIdfor the state volume (aiEngine.state.persistence.createStorageClass, defaulttrue). No state value is needed. To use aReadWriteManyclass of your own, setaiEngine.state.persistence.createStorageClasstofalseandaiEngine.state.persistence.storageClassNameto that class. The class must not be the DataMasque classpersistence.efs.StorageClassName, orhelm installfails with:
aiEngine.state.persistence.storageClassName must not be persistence.efs.StorageClassName: the state claim could bind the static DataMasque volume. Keep the default efs-sc-ai-engine class or name another ReadWriteMany class
With createStorageClass set to true, a storageClassName that names an existing class of yours fails with the following message,
where <name> is that class:
aiEngine.state.persistence.storageClassName names the existing storage class <name>, which the chart would replace: set aiEngine.state.persistence.createStorageClass to false to use it
On ROSA (platform set to openshift), the chart creates no class:
set aiEngine.state.persistence.storageClassName to a ReadWriteMany class of your cluster.
- For the standalone chart, set
state.persistence.storageClassNameto aReadWriteManyclass, orstate.persistence.existingClaimto an existingReadWriteManyclaim. With neither set,helm installfails with:
state.persistence.storageClassName (a ReadWriteMany class) or state.persistence.existingClaim is required
The chart deletes the state claim it creates when you uninstall the release. It never deletes an existingClaim.
The efs-sc-ai-engine class keeps a volume when its claim is deleted (reclaimPolicy: Retain),
so after an uninstall the volume stays Released with its EFS access point.
Delete them with kubectl delete pv <name> and aws efs delete-access-point when you no longer need them.
Model Storage
By default, each pod copies the model files from the model image into an emptyDir volume every time it starts.
To keep the files between restarts, set model.persistence.enabled to true.
The chart then creates a claim, and a restart copies only the files that are missing or have a different size:
model:
persistence:
enabled: true
storageClassName: ebs-sc
accessMode: ReadWriteOnce
size: 10Gi
ebs-sc is the EBS gp3 class that the DataMasque chart creates by default (persistence.ebs.StorageClassName) in layout 2.
In layout 3, use an EBS class of your cluster.
Set model.persistence.existingClaim to use a claim you created. The chart never deletes it.
A ReadWriteOnce model claim allows one replica.
With replicaCount above 1, helm install fails with:
model.persistence.accessMode ReadWriteOnce allows one replica; use ReadWriteMany for replicaCount > 1
Use a ReadWriteMany class and accessMode: ReadWriteMany for more than one replica.
When the models are on EFS, the file system needs Elastic throughput.
Check the EFS throughput mode and set it with:
$ aws efs update-file-system --file-system-id fs-00000000000000000 --throughput-mode elastic
Installing the AI Engine
The values below use the output of ecr-image-push.sh from Preparing the Images.
Each value is required unless it is marked with a default or as optional.
Layout 2: Same Cluster as DataMasque
Add the following to the values.yaml of the DataMasque chart, as described in
DataMasque Deployment on EKS or
on ROSA.
The AI Engine values go under the aiEngine key:
aiEngine:
enabled: true
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
tag: 1.26.11.2
model:
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101 # optional: defaults to the two file names below without `.tar.enc`, joined with `_`
baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc
modelSize: base # default: base. The model served until Settings chooses another: base (Precision) or small (Express)
replicaCount: 1 # default: 1. One GPU per pod
If you changed namespace in the DataMasque chart, set aiEngine.namespaceOverride to the same value.
Otherwise the render fails with the following message, here for namespace set to dm and aiEngine.namespaceOverride left at datamasque:
aiEngine.namespaceOverride (datamasque) must equal namespace (dm): set aiEngine.namespaceOverride=dm
Then install or upgrade DataMasque as usual:
$ helm upgrade --install datamasque ./datamasque-helmchart -n datamasque -f values.yaml
On ROSA, run the helm and kubectl commands in this guide from the DataMasque project, as in
ROSA Deployment, with oc for kubectl and without -n datamasque.
The AI Engine pod requests one nvidia.com/gpu and tolerates the taint nvidia.com/gpu, so it runs on the GPU node.
To place it on a node with a different taint or label, set aiEngine.podPlacement.tolerations,
aiEngine.podPlacement.nodeSelector or aiEngine.podPlacement.affinity.
Check that the pod is ready:
$ kubectl get pods -n datamasque -l app.kubernetes.io/name=datamasque-ai-engine
Layout 3: Different Cluster from DataMasque
In the cluster with the GPU node, save the following as ai-engine-values.yaml.
The state.persistence.storageClassName must be a ReadWriteMany class of that cluster.
A load balancer makes the AI Engine reachable from the DataMasque cluster:
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine
tag: 1.26.11.2
model:
image:
repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/datamasque/local-ai-engine-model
tag: dm-ai-ner-base-1.0.0-20260101_dm-ai-ner-small-1.0.0-20260101 # optional: defaults to the two file names below without `.tar.enc`, joined with `_`
baseFile: dm-ai-ner-base-1.0.0-20260101.tar.enc
smallFile: dm-ai-ner-small-1.0.0-20260101.tar.enc
state:
persistence:
storageClassName: efs-sc # a ReadWriteMany class
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
The three annotations are for the AWS Load Balancer Controller, which creates an internal Network Load Balancer. A cluster without that controller uses the in-tree controller, which reads these annotations instead:
service:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
Install the chart from the datamasque-ai-engine-only.tgz package of the DataMasque Helm chart download:
$ helm install engine ./datamasque-helmchart/datamasque-ai-engine-only.tgz -f ai-engine-values.yaml \
--namespace datamasque-ai-engine --create-namespace
$ kubectl get service engine-ai-engine -n datamasque-ai-engine
The EXTERNAL-IP column shows the load balancer address once it exists.
The AI Engine has no login, so the chart refuses a LoadBalancer Service without the internal annotations above.
helm install then fails with:
service.type LoadBalancer needs an internal load balancer: set service.annotations service.beta.kubernetes.io/aws-load-balancer-type: external, service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip and service.beta.kubernetes.io/aws-load-balancer-scheme: internal (AWS Load Balancer Controller), or service.beta.kubernetes.io/aws-load-balancer-internal: "true" (in-tree controller)
Allow only the DataMasque cluster to reach port 9021.
The DataMasque cluster must have a network route to the load balancer, for example through VPC peering.
The AI Engine URL
- In layout 2, the chart gives DataMasque the URL of the AI Engine as its default.
For the release
datamasquein the namespacedatamasque, the URL ishttp://datamasque-ai-engine.datamasque.svc.cluster.local:9021. - In layout 3, enter
http://<load balancer address>:9021as the AI Engine URL on the Settings page.
A URL saved in Settings, or through the API, always wins over the chart's default, including an empty URL.
Choosing a model in Settings saves the URL with it, so the chart's default stops applying after that,
even when the saved URL is the same as the chart's.
A helm upgrade never changes a saved URL.
To use the chart's URL again after saving another or an empty one, enter the URL in Settings.
Running More than One AI Engine Pod
Set replicaCount above 1 in layout 2 to run more pods, each with its own GPU.
The pods share the state volume, so a model chosen in Settings reaches every pod.
Each pod notices the change within seconds, stops taking new requests, and restarts itself with the new model
once nothing is queued or running on it.
On a model switch, a pod waits for its clients to fetch their asynchronous /detect results, and drops a result that no client has asked for in 30 seconds.
The pods can restart at different times, and Settings shows the AI Engine as loading until a pod is ready.
A pod that restarts for another reason also serves the model that was chosen.
DataMasque's agent submits a job with POST /detect/async and polls GET /detect/async/{id}.
Each pod holds its jobs in memory, so a poll must reach the pod that took the job.
In layout 2, the ClusterIP Service keeps each client on one AI Engine pod.
In layout 3, a load balancer does not, so helm install fails for a replicaCount above 1 with:
service.type LoadBalancer allows one replica: a load balancer does not keep a client on one pod, so an async job's poll can reach another pod and get 404. Use replicaCount 1, or a ClusterIP Service for more
If the state volume is replaced with a different EFS file system or access point, the choice made in Settings is lost
and the AI Engine serves the model from modelSize until Settings chooses another.
Startup Times and Memory
The AI Engine pod is ready when the model is loaded.
The pod waits up to 15 minutes for that (startupProbe.periodSeconds of 10 seconds times startupProbe.failureThreshold of 90),
then restarts.
The times on an EKS g6.xlarge node:
| Step | emptyDir models |
Model claim on gp3 |
|---|---|---|
| Pull the images | About 3.5 minutes on a new node, none once cached | The same as emptyDir |
| Copy the models | About 30 seconds at every start | About 20 seconds at the first start, then none |
| Load the model | About 2 minutes for the base (Precision) model | About 2 minutes for the base (Precision) model |
The image pull is once per node: the engine image is about 5.3 GB and the model image about 4.6 GB.
With a model claim, a restart finds the files up to date and skips the copy.
The memory request of the pod is resources.requests.memory, which is 10Gi.
The pod used at most 8.4 GiB of memory on EKS.
The chart sets no memory or CPU limit, because the taint keeps the GPU node for the AI Engine.
Upgrading the AI Engine
The AI Engine image tag (image.tag, or aiEngine.image.tag in layout 2) is separate from the DataMasque tag.
Upgrading DataMasque does not upgrade the AI Engine.
An AI Engine build expires on the first day of the month after the month of its version.
The expiry_date field of its /health response shows the date.
Renew the image before then:
- Get the new AI Engine package, and run
ecr-image-push.shfrom it as in Preparing the Images. - Set the new
image.tag(aiEngine.image.tagin layout 2), and a newmodel.image.tagandmodel.baseFileormodel.smallFileif the script printed different ones. - Upgrade the release:
$ helm upgrade datamasque ./datamasque-helmchart -n datamasque -f values.yaml
For layout 3, run helm upgrade engine ./datamasque-helmchart/datamasque-ai-engine-only.tgz -f ai-engine-values.yaml --namespace datamasque-ai-engine.
With replicaCount 1, the pod is replaced without a second pod starting first, so the AI Engine is unavailable while the new pod loads, and helm upgrade --wait returns before the new pod is ready: check the pod with kubectl get pods as below.
With more replicas, the pods are replaced one at a time and the others keep serving new runs.
A masking run that starts while no engine pod is ready waits up to 10 minutes for it, as described in
Choosing the model.
A run that is already masking with ai_detect when the engine pod it uses is replaced fails, because that pod held the run's jobs.
With replicaCount 1 the run fails with Failed to connect to AI Engine, and with more replicas with
AI Engine async poll (request_id=<id>) returned HTTP error - Request <id> not found.
Upgrade when no such run is in progress, or start the run again once the pod is ready.
If the new pod does not become ready, go back to the previous release:
$ helm history datamasque -n datamasque
$ helm rollback datamasque -n datamasque
For layout 3, use helm history engine and helm rollback engine with --namespace datamasque-ai-engine.
A rollback in layout 2 restores the whole DataMasque release.
Check that the AI Engine is ready again with kubectl get pods.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
Pod Pending with Insufficient nvidia.com/gpu |
No node offers a free GPU. Check that the GPU node group exists and that the device plugin runs, with the GPU column check in GPU Nodes on EKS. Each pod needs its own GPU, so replicaCount cannot exceed the number of GPU nodes. |
Pod Pending with untolerated taint |
The GPU node has a taint other than nvidia.com/gpu. Set aiEngine.podPlacement.tolerations (layout 2) or podPlacement.tolerations (layout 3) to match it. |
State claim Pending, event reports Volume capabilities not supported |
The state storage class does not support ReadWriteMany, as with gp3. Use an EFS class, see State Storage. |
ImagePullBackOff |
The node cannot pull the image. If the node role lacks the AmazonEC2ContainerRegistryReadOnly policy, attach it. Otherwise check image.repository, image.tag and model.image.tag (aiEngine.image.repository and so on in layout 2) with kubectl describe pod. A tag that was never pushed gives the same state: roll back as in Upgrading the AI Engine. DataMasque pods are not affected. |
Init:Error |
The copy-model container found no model files. Read kubectl logs <pod> -c copy-model: no *.tar.enc in /models means the model image holds no model. Push it again with ecr-image-push.sh from a folder that download_model.sh filled. <file> is not in /models means model.baseFile or model.smallFile (aiEngine.model. in layout 2) names a file the model image does not hold: set them to the file names in the image, or push the image again. |
Settings shows No answer from the AI Engine, or a run fails with did not serve its model within 10 minutes |
DataMasque cannot reach the URL, or the pod is not ready. Check that the pod is ready and that the URL in Settings is the one in The AI Engine URL. |
Settings shows that a model file is damaged or incomplete; delete it and download it again |
The model image holds a damaged model file. On Kubernetes, run download_model.sh into a new folder and push the model image again with ecr-image-push.sh, as in Preparing the Images. The new image has the same tag, so set model.image.pullPolicy (aiEngine.model.image.pullPolicy in layout 2) to Always and upgrade the release; the new pod pulls the image and copies the new file. |
Pod CrashLoopBackOff soon after it starts, image.tag (aiEngine.image.tag in layout 2) below 1.26.11.2 |
AI Engine builds before 1.26.11.2 run as root and cannot read their files as the chart's user 1000 (the log shows PermissionError: [Errno 13] Permission denied: '/.cache'). Use 1.26.11.2 or later. |
Settings shows Build has expired |
The AI Engine build is past its expiry. Renew the image as in Upgrading the AI Engine. |