Per-app writeable volumes for stateful tenants (read-only rootfs follow-up) #153

New Issue

claude · 2026-04-25T20:51:56+02:00

claude commented

2026-04-25 20:51:56 +02:00

Follow-up to #152.

Context

The runtime-hardening PR sets read_only_rootfs=true on every tenant container and mounts a 256m tmpfs at /tmp (rw, nosuid). This works for stateless apps (vanilla Camel-Kafka producers/consumers, REST APIs, integrations) — but breaks anything that writes durable state outside /tmp:

App pattern	Breaks under read-only rootfs?
Kafka Streams with RocksDB state stores	✅ Needs writeable disk for state.dir
Apps writing log files to `/var/log/...`	✅ Needs volume or stdout reconfig
Apps writing heap dumps / flight recorder to `/opt/app/...`	✅ Needs volume
Hibernate L2 cache on disk, Lucene indexes	✅ Needs volume

A 256m tmpfs is enough for transient JVM scratch but not enough for production state stores, and tmpfs is wiped on container restart anyway.

Proposal

Add containerConfig.writeableVolumes: List<String> to ResolvedContainerConfig / ConfigMerger:

containerConfig:
  writeableVolumes:
    - /var/lib/kafka-streams
    - /opt/app/data

For each declared path, the orchestrator binds a Docker volume scoped to (tenant, app, env, path) so:

The volume name is deterministic: cameleer-{tenant}-{env}-{app}-{slug-of-path}.
Volumes survive deploys (state persists across blue/green / rolling).
Volumes never cross tenants (name carries tenant id).
Volumes never cross envs (name carries env slug).

UI

AppConfigurationPage → Resources tab → "Writeable volumes" list. Add path, remove path. Show resolved Docker volume name as read-only hint.

Implementation surface

cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ResolvedContainerConfig.java — add field
cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ConfigMerger.java — three-layer merge (global / env / app)
cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ContainerRequest.java — add field, plumb through
cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestrator.java — Volume binds for each declared path
cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DeploymentExecutor.java — name resolution + ensure-volume
UI: AppConfigurationPage.tsx resource tab
Lifecycle: when an app is deleted, prune its volumes (volume-cleanup hook in AppService.deleteApp)

Acceptance

Kafka Streams app deploys with writeableVolumes: [/tmp/kafka-streams] and survives a redeploy with state intact.
Volumes from tenant A are not visible to tenant B's containers.
Removing an app removes its volumes.
Existing stateless apps unaffected (default = empty list).

Follow-up to #152. ## Context The runtime-hardening PR sets `read_only_rootfs=true` on every tenant container and mounts a 256m tmpfs at `/tmp` (rw, nosuid). This works for stateless apps (vanilla Camel-Kafka producers/consumers, REST APIs, integrations) — but breaks anything that writes durable state outside `/tmp`: | App pattern | Breaks under read-only rootfs? | |---|---| | Kafka Streams with RocksDB state stores | ✅ Needs writeable disk for state.dir | | Apps writing log files to `/var/log/...` | ✅ Needs volume or stdout reconfig | | Apps writing heap dumps / flight recorder to `/opt/app/...` | ✅ Needs volume | | Hibernate L2 cache on disk, Lucene indexes | ✅ Needs volume | A 256m tmpfs is enough for transient JVM scratch but not enough for production state stores, and tmpfs is wiped on container restart anyway. ## Proposal Add `containerConfig.writeableVolumes: List<String>` to `ResolvedContainerConfig` / `ConfigMerger`: ```yaml containerConfig: writeableVolumes: - /var/lib/kafka-streams - /opt/app/data ``` For each declared path, the orchestrator binds a Docker volume scoped to `(tenant, app, env, path)` so: - The volume name is deterministic: `cameleer-{tenant}-{env}-{app}-{slug-of-path}`. - Volumes survive deploys (state persists across blue/green / rolling). - Volumes never cross tenants (name carries tenant id). - Volumes never cross envs (name carries env slug). ## UI `AppConfigurationPage` → Resources tab → "Writeable volumes" list. Add path, remove path. Show resolved Docker volume name as read-only hint. ## Implementation surface - `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ResolvedContainerConfig.java` — add field - `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ConfigMerger.java` — three-layer merge (global / env / app) - `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ContainerRequest.java` — add field, plumb through - `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestrator.java` — `Volume` binds for each declared path - `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DeploymentExecutor.java` — name resolution + ensure-volume - UI: `AppConfigurationPage.tsx` resource tab - Lifecycle: when an app is deleted, prune its volumes (volume-cleanup hook in `AppService.deleteApp`) ## Acceptance - Kafka Streams app deploys with `writeableVolumes: [/tmp/kafka-streams]` and survives a redeploy with state intact. - Volumes from tenant A are not visible to tenant B's containers. - Removing an app removes its volumes. - Existing stateless apps unaffected (default = empty list).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: cameleer/cameleer-server#153