[Bug]: Incomplete aarch64 support

[middelkoopt]

Describe the bug

the openchami package needs the following versions to be bumped (along with the CMD exec bug fixed) in order to fully support aarch64 on OpenHPC (OpenCHAMI/local-ca#5)

• Bump opaal + opaal-idp to v0.3.12
• Bump step-ca to local-ca:v0.2.3

Steps to reproduce

With the following patch OpenCHAMI fully deploys with a OpenHPC recipe.

# --- aarch64 patch: fix container image versions until RPM is updated ---
sed -i 's|ghcr.io/openchami/opaal:v0.3.10|ghcr.io/openchami/opaal:v0.3.12|g' /etc/containers/systemd/opaal.container /etc/containers/systemd/opaal-idp.container
sed -i 's|ghcr.io/openchami/local-ca:v0.2.2|ghcr.io/openchami/local-ca:v0.2.3|g' /etc/containers/systemd/step-ca.container
sed -i '/^Image=ghcr.io\/openchami\/local-ca/a Exec=\/usr\/bin\/step-ca --password-file \/home\/step\/secrets\/password \/home\/step\/config\/ca.json' /etc/containers/systemd/step-ca.container
# --- end aarch64 patch ---

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[synackd]

v0.2.4 of the release incorporates these changes. There's currently a signing issue with the RPM that is preventing an official release (will have to look into it tomorrow). However, it can be built manually on the v0.2.4 tag for testing purposes.

[middelkoopt]

0.7.1 fails on aarch64 support.

Mar 25 18:50:48 head step-ca[6783]: Error: unable to copy from source docker://ghcr.io/openchami/local-ca:v0.2.2: choosing an image from manifest list docker://ghcr.io/openchami/local-ca:v0.2.2: no image found in image index for architecture "arm64", variant "v8", OS "linux"
Mar 25 18:50:48 head podman[6783]: 2026-03-25 18:50:48.271367687 +0000 UTC m=+10.644050797 image pull-error  ghcr.io/openchami/local-ca:v0.2.2 unable to copy from source docker://ghcr.io/openchami/local-ca:v0.2.2: choosing an image from manifest list docker://ghcr.io/openchami/local-ca:v0.2.2: no image found in image index for architecture "arm64", variant "v8", OS "linux"

Mar 25 18:52:42 head opaal-idp[9475]: {"msg":"exec container process `/sbin/tini`: Exec format error","level":"error","time":"2026-03-25T18:52:42.80063>
Mar 25 18:52:11 head opaal[8400]: {"msg":"exec container process `/sbin/tini`: Exec format error","level":"error","time":"2026-03-25T18:52:11.090615Z"}

[synackd]

Looks like specifically ARMv8 is missing. For local-ca, I've created a PR to add it as well as an image inspection step for debugging: OpenCHAMI/local-ca#7

[synackd]

0.7.1 fails on aarch64 support.

v0.7.1 is the ochami CLI latest version, not the release repository's latest version. I am still waiting on an update key so we can tag a new release with a signed RPM.

Mar 25 18:50:48 head step-ca[6783]: Error: unable to copy from source docker://ghcr.io/openchami/local-ca:v0.2.2: choosing an image from manifest list docker://ghcr.io/openchami/local-ca:v0.2.2: no image found in image index for architecture "arm64", variant "v8", OS "linux"
Mar 25 18:50:48 head podman[6783]: 2026-03-25 18:50:48.271367687 +0000 UTC m=+10.644050797 image pull-error  ghcr.io/openchami/local-ca:v0.2.2 unable to copy from source docker://ghcr.io/openchami/local-ca:v0.2.2: choosing an image from manifest list docker://ghcr.io/openchami/local-ca:v0.2.2: no image found in image index for architecture "arm64", variant "v8", OS "linux"

Looks like you are using v0.2.2 of local-ca when the current was v0.2.4. I've merged the linked PR in the local-ca repo and tagged v0.2.5. Can you try it pulling ghcr.io/openchami/local-ca:v0.2.5?

[middelkoopt]

Ah, I confused ochami and openchami...

I was using local-ca v0.2.2 because I removed my patches, I thought I was testing a new release, sorry about that.

I get a failure with that.

+ systemctl start openchami.target
A dependency job for openchami.target failed. See 'journalctl -xe' for details.

hard to tease out the errors, but these stood out.

Mar 25 19:52:38 head podman[7306]: 2026-03-25 19:52:38.738536002 +0000 UTC m=+0.034628113 container start 3a7b70a4d23f71446f844ba5212ea2f265a7ee785100cb13de02e8369f245438 (image=ghcr.io/openchami/local-ca:v0.2.5, name=step-ca, org.opencontainers.image.url=https://github.com/OpenCHAMI/local-ca, org.opencontainers.image.source=https://github.com/OpenCHAMI/local-ca, org.opencontainers.image.title=local-ca, org.opencontainers.image.authors=Chainguard Team https://www.chainguard.dev/, org.opencontainers.image.created=2026-03-25T19:27:50.665Z, dev.chainguard.image.title=wolfi-base, org.opencontainers.image.revision=582a6aecf4b5f10c11b0197a957bdf82fc39a641, org.opencontainers.image.version=v0.2.5, org.opencontainers.image.description=, PODMAN_SYSTEMD_UNIT=step-ca.service, org.opencontainers.image.vendor=Chainguard, dev.chainguard.package.main=, org.opencontainers.image.licenses=MIT)
Mar 25 19:52:38 head systemd[1]: Started step-ca.service - The step-ca container.
Mar 25 19:52:38 head step-ca[7306]: 3a7b70a4d23f71446f844ba5212ea2f265a7ee785100cb13de02e8369f245438
Mar 25 19:52:38 head podman[7306]: 2026-03-25 19:52:38.710880854 +0000 UTC m=+0.006973007 image pull 47146166f076762bee3d1a94ae5ae35b38a7fcec008a131a666b8566f69bc810 ghcr.io/openchami/local-ca:v0.2.5
Mar 25 19:52:38 head systemd[1]: Starting openchami-cert-trust.service - Extract and trust OpenCHAMI root CA certificate...
Mar 25 19:52:38 head step-ca[7325]: step-ca can't find or open the configuration file for your CA.
Mar 25 19:52:38 head step-ca[7325]: You may need to create a CA first by running `step ca init`.
Mar 25 19:52:38 head step-ca[7325]: Documentation: https://u.step.sm/docs/ca
Mar 25 19:52:38 head sh[7338]: Error: 1 error occurred:
Mar 25 19:52:38 head sh[7338]:         * copying from container: open /proc/7329/ns/mnt: no such file or directory

and

Mar 25 19:52:38 head step-ca[7170]: Generating intermediate certificate... done!
Mar 25 19:52:38 head step-ca[7170]: 
Mar 25 19:52:38 head step-ca[7170]: ✔ Root certificate: /home/step/certs/root_ca.crt
Mar 25 19:52:38 head step-ca[7170]: ✔ Root private key: /home/step/secrets/root_ca_key
Mar 25 19:52:38 head step-ca[7170]: ✔ Root fingerprint: 3b9059021d2d4e818a9434d59b9e1b4572dbc8e3b00a7cd51074bf7b6109a91d
Mar 25 19:52:38 head step-ca[7170]: ✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
Mar 25 19:52:38 head step-ca[7170]: ✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
Mar 25 19:52:38 head step-ca[7170]: ✔ Database folder: /home/step/db
Mar 25 19:52:38 head step-ca[7170]: ✔ Default configuration: /home/step/config/defaults.json
Mar 25 19:52:38 head step-ca[7170]: ✔ Certificate Authority configuration: /home/step/config/ca.json
Mar 25 19:52:38 head step-ca[7170]: 
Mar 25 19:52:38 head step-ca[7170]: Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.
Mar 25 19:52:38 head step-ca[7170]: 
Mar 25 19:52:38 head step-ca[7170]: FEEDBACK 😍 🍻
Mar 25 19:52:38 head step-ca[7170]:   The step utility is not instrumented for usage statistics. It does not phone
Mar 25 19:52:38 head step-ca[7170]:   home. But your feedback is extremely valuable. Any information you can provide
Mar 25 19:52:38 head step-ca[7170]:   regarding how you’re using `step` helps. Please send us a sentence or two,
Mar 25 19:52:38 head step-ca[7170]:   good or bad at feedback@smallstep.com or join GitHub Discussions
Mar 25 19:52:38 head step-ca[7170]:   https://github.com/smallstep/certificates/discussions and our Discord 
Mar 25 19:52:38 head step-ca[7170]:   https://u.step.sm/discord.
Mar 25 19:52:38 head step-ca[7170]: 
Mar 25 19:52:38 head step-ca[7170]: 👉 Your CA administrative password is: "supersecretpassword"
Mar 25 19:52:38 head step-ca[7170]: 🤫 This will only be displayed once.
Mar 25 19:52:38 head step-ca[7170]: 🔒 Your CA is ready to go!
Mar 25 19:52:38 head step-ca[7170]: step-ca can't find or open the configuration file for your CA.
Mar 25 19:52:38 head step-ca[7170]: You may need to create a CA first by running `step ca init`.
Mar 25 19:52:38 head step-ca[7170]: Documentation: https://u.step.sm/docs/ca

[synackd]

Thanks for the detailed logs. It can be easy to do...

So the container is pulling now, which means it's finding the right architecture. Quick check, what is the output of:

grep local-ca /etc/containers/systemd/step-ca

?

Should be:

Image=ghcr.io/openchami/local-ca:v0.2.5

If not, it should be changed to that. After that, sudo systemctl daemon-reload and then restart openchami.target.

If you've already done all of these things, let me know and we can troubleshoot further.

Since we aren't using the local-ca acme.sh container, I'm thinking we can probably get rid of the local-ca container in a future OpenCHAMI release...

[middelkoopt]

[root@head ~]# grep local-ca /etc/containers/systemd/step-ca
grep: /etc/containers/systemd/step-ca: No such file or directory
[root@head ~]# grep local-ca /etc/containers/systemd/step-ca.*
Image=ghcr.io/openchami/local-ca:v0.2.5
[root@head ~]# 

[middelkoopt]

Mar 26 12:02:53 head step-ca[6738]: 7c67589669aa2626ac52db4703bc891129a19b4c1c2bdae90f5af28b4250b92f
Mar 26 12:02:53 head NetworkManager[656]: <info>  [1774526573.9094] device (podman4): Activation: successful, device activated.
Mar 26 12:02:53 head sh[7003]: Error: "/root_ca/root_ca.crt" could not be found on container step-ca: no such file or directory

Mar 26 12:02:53 head step-ca[6979]: 
Mar 26 12:02:53 head step-ca[6979]: Generating root certificate... done!
Mar 26 12:02:54 head sh[7038]: Error: "/root_ca/root_ca.crt" could not be found on container step-ca: no such file or directory
Mar 26 12:02:54 head step-ca[6979]: Generating intermediate certificate... done!
Mar 26 12:02:54 head step-ca[6979]: 
Mar 26 12:02:54 head step-ca[6979]: ✔ Root certificate: /home/step/certs/root_ca.crt
Mar 26 12:02:54 head step-ca[6979]: ✔ Root private key: /home/step/secrets/root_ca_key
Mar 26 12:02:54 head step-ca[6979]: ✔ Root fingerprint: 194f6a23b56484b6961463f9128206cfb7be94cecd36c489a45265f8d7b23043
Mar 26 12:02:54 head step-ca[6979]: ✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
Mar 26 12:02:54 head step-ca[6979]: ✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
Mar 26 12:02:54 head step-ca[6979]: ✔ Database folder: /home/step/db
Mar 26 12:02:54 head step-ca[6979]: ✔ Default configuration: /home/step/config/defaults.json
Mar 26 12:02:54 head step-ca[6979]: ✔ Certificate Authority configuration: /home/step/config/ca.json
Mar 26 12:02:54 head step-ca[6979]: 
Mar 26 12:02:54 head step-ca[6979]: Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.
Mar 26 12:02:54 head step-ca[6979]: 
Mar 26 12:02:54 head step-ca[6979]: FEEDBACK 😍 🍻
Mar 26 12:02:54 head step-ca[6979]:   The step utility is not instrumented for usage statistics. It does not phone
Mar 26 12:02:54 head step-ca[6979]:   home. But your feedback is extremely valuable. Any information you can provide
Mar 26 12:02:54 head step-ca[6979]:   regarding how you’re using `step` helps. Please send us a sentence or two,
Mar 26 12:02:54 head step-ca[6979]:   good or bad at feedback@smallstep.com or join GitHub Discussions
Mar 26 12:02:54 head step-ca[6979]:   https://github.com/smallstep/certificates/discussions and our Discord 
Mar 26 12:02:54 head step-ca[6979]:   https://u.step.sm/discord.
Mar 26 12:02:54 head step-ca[6979]: 
Mar 26 12:02:54 head step-ca[6979]: 👉 Your CA administrative password is: "supersecretpassword"
Mar 26 12:02:54 head step-ca[6979]: 🤫 This will only be displayed once.
Mar 26 12:02:54 head step-ca[6979]: 🔒 Your CA is ready to go!
Mar 26 12:02:55 head step-ca[6979]: step-ca can't find or open the configuration file for your CA.
Mar 26 12:02:55 head step-ca[6979]: You may need to create a CA first by running `step ca init`.
Mar 26 12:02:55 head step-ca[6979]: Documentation: https://u.step.sm/docs/ca

Mar 26 12:02:55 head podman[7064]: 2026-03-26 12:02:55.207684216 +0000 UTC m=+0.155633207 container remove 7c67589669aa2626ac52db4703bc891129a19b4c1c2bdae90f5af28b4250b92f (image=ghcr.io/openchami/local-ca:v0.2.5, name=step-ca, org.opencontainers.image.title=local-ca, org.opencontainers.image.url=https://github.com/OpenCHAMI/local-ca, PODMAN_SYSTEMD_UNIT=step-ca.service, org.opencontainers.image.authors=Chainguard Team https://www.chainguard.dev/, org.opencontainers.image.source=https://github.com/OpenCHAMI/local-ca, org.opencontainers.image.revision=582a6aecf4b5f10c11b0197a957bdf82fc39a641, org.opencontainers.image.vendor=Chainguard, dev.chainguard.package.main=, org.opencontainers.image.created=2026-03-25T19:27:50.665Z, org.opencontainers.image.version=v0.2.5, dev.chainguard.image.title=wolfi-base, org.opencontainers.image.description=, org.opencontainers.image.licenses=MIT)
Mar 26 12:02:55 head podman[7064]: 2026-03-26 12:02:55.208554768 +0000 UTC m=+0.156503717 volume remove 62635fc00e79dd9935169313e262d64d68ff26a8cf08f4d3cf943aebfe0aa535
Mar 26 12:02:55 head systemd[1]: step-ca.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 12:02:55 head systemd[1]: step-ca.service: Failed with result 'exit-code'.
Mar 26 12:02:55 head systemd[1]: step-ca.service: Consumed 1.567s CPU time, 213.2M memory peak.
Mar 26 12:02:55 head postgres[6746]: Copying blob sha256:926abe7126acf3dcc8f1e85c13d01ac01ee5c64101268f38e9ca901e07cc1f14
Mar 26 12:02:55 head systemd[1]: step-ca.service: Scheduled restart job, restart counter is at 1.
Mar 26 12:02:55 head systemd[1]: openchami-cert-trust.service: Main process exited, code=killed, status=15/TERM
Mar 26 12:02:55 head systemd[1]: openchami-cert-trust.service: Failed with result 'signal'.
Mar 26 12:02:55 head systemd[1]: Stopped openchami-cert-trust.service - Extract and trust OpenCHAMI root CA certificate.
Mar 26 12:02:55 head systemd[1]: Starting step-ca.service - The step-ca container...

[root@head ~]# for I in /etc/containers/systemd/step-ca* ; do echo === $I ; cat $I ; done
=== /etc/containers/systemd/step-ca.container
[Unit]
Description=The step-ca container
PartOf=openchami.target
Requires=openchami-cert-internal-network.service step-ca-home-volume.service step-ca-db-volume.service step-root-ca-volume.service
After=openchami-cert-internal-network.service step-ca-home-volume.service step-ca-db-volume.service step-root-ca-volume.service

[Container]
ContainerName=step-ca
HostName=step-ca
Image=ghcr.io/openchami/local-ca:v0.2.5

# Volumes
Volume=step-ca-home.volume:/home/step
Volume=step-ca-db.volume:/home/step/db
Volume=step-root-ca.volume:/root_ca/

# Environment Variables
EnvironmentFile=/etc/openchami/configs/openchami.env

# Networks for the Container to use
Network=openchami-cert-internal.network

# Proxy settings
PodmanArgs=--http-proxy=false
PodmanArgs=--no-hosts

[Service]
Restart=always
=== /etc/containers/systemd/step-ca-db.volume
[Unit]
Description=step-ca-db Volume

[Volume]
VolumeName=step-ca-db
=== /etc/containers/systemd/step-ca-home.volume
[Unit]
Description=step-ca-home Volume

[Volume]
VolumeName=step-ca-home

[middelkoopt]

Claude and I did some debugging.

This is the patch to make it work.

# FIXME: local-ca CMD uses unexpanded $PWDPATH/$CONFIGPATH; override with literal paths
sed -i '/^PodmanArgs=--no-hosts$/a Exec=/usr/bin/step-ca --password-file /home/step/secrets/password /home/step/config/ca.json' \
	/etc/containers/systemd/step-ca.container

From Claude:

To summarize the root cause for the bug report: local-ca v0.2.5's Dockerfile defines CMD in exec form — ["/usr/bin/step-ca", "--password-file", "$PWDPATH", "$CONFIGPATH"] — with the env var names as literal strings. The entrypoint's exec "${@}" passes them through unexpanded, so step-ca receives $PWDPATH and $CONFIGPATH as literal filenames rather than the paths defined by the image's ENV. The fix in the Dockerfile would be to use shell form (CMD /usr/bin/step-ca --password-file $PWDPATH $CONFIGPATH) or to replace exec "${@}" with exec /usr/bin/step-ca --password-file "$PWDPATH" "$CONFIGPATH" directly.

[synackd]

@middelkoopt v0.2.6 of local-ca contains a fix that I've tested on a test system. Can you pull it and try it out?

[middelkoopt]

Works. OpenHPC builds with this patch/versions on aarch64

sed -i 's|ghcr.io/openchami/opaal:v0.3.10|ghcr.io/openchami/opaal:v0.3.12|g' \
	/etc/containers/systemd/opaal.container \
	/etc/containers/systemd/opaal-idp.container
sed -i 's|ghcr.io/openchami/local-ca:v0.2.2|ghcr.io/openchami/local-ca:v0.2.6|g' \
	/etc/containers/systemd/step-ca.container

# OpenCHAMI component versions
openchami_version: "0.1.2"
ochami_version: "0.7.0"

[synackd]

Excellent. I'll bump the versions in the release to match these.