Containers are not starting on ARM Instances

Containers failing to start on ARM nodes can often be attributed to architecture mismatches. This occurs when container images designed for x86 architecture are deployed on ARM-based nodes.

The binary instructions in the x86 image are incompatible with ARM processors, causing the kernel on the ARM node to encounter illegal or unknown instructions, resulting in errors.

The most common symptom of this issue is the occurrence of CrashLoopBackOff errors in Kubernetes.

Inspecting container image architecture

To inspect the architecture of container images, you can use the following steps:

1. Pull the container image from the registry using docker pull.
2. Extract the image using docker save or similar tools.
3. Within the extracted files, locate the image manifest file (usually named manifest.json). Open this file and look for the architecture field to determine the architecture the image is built for.

Possible solutions

Rebuilding ARM images

If you have control over the container image source, consider rebuilding the image specifically for the ARM architecture.

Tools like docker buildx support multi-platform builds, allowing you to build images for multiple architectures simultaneously. Ensure that your Dockerfile file specifies the target architecture correctly.

Example of Dockerfile snippet for ARM:

FROM aarch64/ubuntu:latest
# Add your application setup steps here

Additionally, ensure that any libraries or dependencies used in your application are compatible with ARM architecture.

Using multi-arch images

Use multi-architecture images that support both x86 and ARM architectures. Docker supports manifest lists, which are a collection of different image manifests for different architectures.

Kubernetes can pull the appropriate image based on the architecture of the node where it is scheduled to run.

Example of manifest list usage:

docker manifest create <multi-arch-image>:<tag> <x86-image>:<tag> <arm-image>:<tag>
docker manifest push <multi-arch-image>:<tag>

Kubernetes will then pull the correct architecture of the image based on the node’s architecture.

Additional tips

• Ensure that all dependencies and libraries used in your application are compatible with the ARM architecture. Tools like the file command can help identify the architecture of binary files.
• Use Kubernetes node selectors or affinity rules to ensure that containers are scheduled on nodes with compatible architectures. This prevents scheduling x86 containers on ARM nodes and vice versa.

Refer to our documentation to find out how to deploy both x86 and ARM images in Kubernetes.