cameleer-server/docs/handoff/2026-04-28-runtime-base-hardening.md

Handoff — Runtime Base Image Hardening (cameleer-saas)

Audience: cameleer-saas team. Owner repo to change: cameleer-saas (docker/runtime-base/). Owner repo of this handoff: cameleer-server (multi-tenant orchestration consumer of the image).

TL;DR

Replace eclipse-temurin:21-jre-alpine with cgr.dev/chainguard/jre:openjdk-21 (Chainguard, Wolfi-based, glibc) and remove the dead ENTRYPOINT from docker/runtime-base/Dockerfile. One-line FROM change plus deletion. Pin by digest in production. Net effect:

• Smaller CVE surface — Chainguard rebuilds daily; baseline CVE count is near zero by design, with signed images and SBOMs.
• glibc instead of musl — fixes a hidden compatibility risk (Netty tcnative, Snappy, LZ4, Zstd, RocksDB, oshi, JNA-using libs are glibc-only and fail at load on Alpine/musl). Tenant apps haven't tripped this yet only because no one's tried.
• Same operational shape — non-root, has sh (Wolfi/busybox), works with the existing DeploymentExecutor sh -c entrypoint construction. No orchestrator change required.

Why now

cameleer-server's DockerRuntimeOrchestrator enforces a hardening contract for tenant containers (cap_drop ALL, no-new-privileges, apparmor=docker-default, readonly rootfs, per-container /tmp tmpfs nosuid 256 MB, pids_limit=512, userns_mode=host:1000:65536). The base image is the one piece outside that contract — and is the largest source of CVEs in a tenant container's attack surface today. Switching the base is the highest-leverage remaining hardening move.

Current state

cameleer-saas/docker/runtime-base/Dockerfile:

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app

COPY agent.jar /app/agent.jar
COPY cameleer-log-appender.jar /app/cameleer-log-appender.jar

ENTRYPOINT exec java \
  -Dcameleer.export.type=${CAMELEER_EXPORT_TYPE:-HTTP} \
  -Dcameleer.export.endpoint=${CAMELEER_SERVER_URL} \
  -Dcameleer.agent.name=${HOSTNAME} \
  -Dcameleer.agent.application=${CAMELEER_APPLICATION_ID:-default} \
  -Dcameleer.agent.environment=${CAMELEER_ENVIRONMENT_ID:-default} \
  -Dcameleer.routeControl.enabled=${CAMELEER_ROUTE_CONTROL_ENABLED:-false} \
  -Dcameleer.replay.enabled=${CAMELEER_REPLAY_ENABLED:-false} \
  -Dcameleer.health.enabled=true \
  -Dcameleer.health.port=9464 \
  -javaagent:/app/agent.jar \
  -jar /app/app.jar

Two issues, addressed together:

1. Base = eclipse-temurin:21-jre-alpine — Alpine + musl. Already small, already non-root, but musl breaks any tenant pulling glibc-only JNI. Daily CVE refresh is on Eclipse's release cadence (slower).
2. ENTRYPOINT is dead code. cameleer-server's DeploymentExecutor constructs its own per-runtime-type entrypoint at deploy time and passes it to createContainerCmd().withCmd("sh", "-c", entrypoint), overriding whatever the base sets. The path /app/app.jar referenced here is also stale — actual deploys mount /app/jars/app.jar via the per-replica named volume populated by cameleer-runtime-loader. Keeping the dead ENTRYPOINT invites future maintainers to "fix" the wrong layer.

Target state

# Wolfi-based JRE, glibc, daily-rebuilt with near-zero baseline CVEs,
# signed images + SBOM published, non-root by default. Pin by digest in
# production overlays — see "Pinning" below.
FROM cgr.dev/chainguard/jre:openjdk-21

WORKDIR /app

# Agent + log appender are baked in; tenant JAR is delivered at deploy
# time by cameleer-runtime-loader into the RO-mounted /app/jars volume.
COPY agent.jar /app/agent.jar
COPY cameleer-log-appender.jar /app/cameleer-log-appender.jar

# No ENTRYPOINT here. cameleer-server's DeploymentExecutor builds the
# per-runtime-type entrypoint (spring-boot/quarkus: -jar; plain-java:
# -cp + main; native: exec) and overrides via withCmd("sh","-c",...).
# Setting one here only creates drift between this image and the actual
# runtime command.

That's it. No multi-stage needed, no extra packages.

Pinning (production)

Tag references (:openjdk-21) move when Chainguard rebuilds. That's the point — you get CVE refresh — but for reproducible deploys, pin by digest in the production CI run:

FROM cgr.dev/chainguard/jre:openjdk-21@sha256:<digest>

Resolve the current digest at build time:

crane digest cgr.dev/chainguard/jre:openjdk-21
# or:
docker buildx imagetools inspect cgr.dev/chainguard/jre:openjdk-21 \
  --format '{{json .Manifest.Digest}}'

Bump the pin on a regular cadence (monthly, or when a Chainguard advisory lands). The CI workflow that builds cameleer-runtime-base is the natural home for the bump — keep it as a deliberate commit so reviewers see the upgrade.

Verification

Before merging in cameleer-saas:

1. Build smoke:

   docker build -t cameleer-runtime-base:test docker/runtime-base/
   docker run --rm cameleer-runtime-base:test java -version
   docker run --rm cameleer-runtime-base:test sh -c 'id'
   

   Expect Java 21 banner, non-root id (uid=65532 or similar — Chainguard's default nonroot user).

2. End-to-end deploy through cameleer-server:

   • Build the new cameleer-runtime-base image.
   • Push to the dev registry.
   • Trigger a deployment of any tenant Spring Boot app via the cameleer-server UI / API.
   • Watch the deploy progress through PRE_FLIGHT → PULL_IMAGE → CREATE_NETWORK → START_REPLICAS → HEALTH_CHECK → SWAP_TRAFFIC → COMPLETE.
   • Confirm: container starts, /api/v1/health returns UP on the tenant, agent registers and heartbeats appear in cameleer-server logs.

3. Negative test (compatibility win, optional but recommended):

   • Build a tiny Camel app that uses camel-netty (which bundles netty-tcnative-boringssl-static).
   • Deploy it on the new base. With Alpine/musl this fails at native lib load; on Chainguard it should start clean.
   • This is the test that demonstrates the real user-visible win, not just the CVE numbers.

4. Rollback plan: revert the Dockerfile change, rebuild + push, retag deployments. The runtime contract on the cameleer-server side is unchanged — no migration, no data shape change, no orchestrator behaviour change. Failure at the base layer is reversible at the same speed as a normal deploy.

What you're NOT changing

• cameleer-server orchestrator code — no changes. The runtime base is opaque to it; only env vars, entrypoint construction, and the loader-volume mount matter, and none of those depend on the base.
• cameleer-runtime-loader image — separate, already minimal (busybox:1.37-musl, ~2.6 MB, runs only at deploy time, exits 0 on success). Loader runs wget once and is gone before the main container starts. Don't bundle it with the base.
• Hardening contract — orchestrator-side, unchanged. cap_drop ALL, readonly rootfs, /tmp tmpfs, etc. continue to apply on top of whatever base image is used.

Optional follow-ups (NOT required for this handoff)

These are deeper investments worth tracking but don't block the Chainguard switch:

1. Distroless — gcr.io/distroless/java21-debian12:nonroot is even smaller (~200 MB) and has the smallest attack surface of any pre-built option (no shell, no package manager). Adopting it requires cameleer-server's DeploymentExecutor to refactor its entrypoint construction from withCmd("sh","-c", "<string>") to a JSON-array form (withCmd("java","-javaagent:/app/agent.jar","-jar","/app/jars/app.jar", ...)). That's a small but non-trivial change because the orchestrator currently splices customArgs (freeform string) into the shell command — doable safely with a tokeniser, but worth discussing as its own ticket. Trade-off: lose docker exec -it sh for live debugging.

2. jlink-based custom JRE — explicitly not recommended for this base. jlink works when you control the app's JDK module set; tenant apps can use any standard module (AWT, JFR, sun.misc.Unsafe, etc.). A custom JRE base would silently break tenant code on JDK upgrades. Keep jlink for single-purpose images, not multi-tenant runtime bases.

3. From scratch — same reasoning as #2 plus you take on the burden of glibc + libfontconfig + libfreetype + every CA bundle update. Maintenance cost dwarfs the CVE win.

Cross-checks for the SaaS team

• cameleer-saas/.gitea/workflows/ci.yml "Build and push runtime base image" step: no change needed. Same docker buildx build --push docker/runtime-base/ invocation works against the new FROM.
• cameleer-saas/docker/runtime-base/agent.jar and cameleer-log-appender.jar are still pulled from the gitea Maven registry by the CI step. Unchanged.
• The runtime-base:latest tag consumers (cameleer-server's CAMELEER_SERVER_RUNTIME_BASEIMAGE env on tenant servers) keep pointing at the same logical image. Tenant servers pick up the new base on their next deploy because pullImage() runs at PRE_FLIGHT.

Sign-off checklist for the implementing engineer

• FROM swapped to cgr.dev/chainguard/jre:openjdk-21.
• Dead ENTRYPOINT block deleted.
• Production overlay pins by digest.
• Local docker build smoke green.
• One end-to-end tenant deploy through cameleer-server green (deploy reaches RUNNING, agent registers, healthcheck UP).
• Optional: Netty-tcnative tenant smoke shows the glibc compatibility win.
• CI registry cleanup loop already covers cameleer-runtime-base — confirm tag retention isn't disrupted (no change expected, but check).

Pointers

• cameleer-server/.claude/rules/docker-orchestration.md — the hardening contract on the cameleer-server side that this base sits underneath.
• cameleer-server/cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestrator.java — baseHardenedHostConfig() is the spec; the base image runs inside this contract.
• cameleer-server/cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DeploymentExecutor.java — entrypoint construction logic that overrides whatever ENTRYPOINT the base sets.
• Chainguard catalog: https://images.chainguard.dev/directory/image/jre/versions
• Chainguard image security model: https://www.chainguard.dev/unchained/the-chainguard-images-security-model