cameleer/cameleer-server
1
0
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Wiki Activity

Secret delivery option 2: Tmpfs-mounted secret files #134

New Issue
Open
opened 2026-04-15 00:35:48 +02:00 by claude · 0 comments
claude commented 2026-04-15 00:35:48 +02:00
Owner
Copy Link

Parent epic: #129

Overview

Replace plaintext environment variables with secret files mounted into containers at /run/secrets/. The server writes secrets to a tmpfs-backed directory, bind-mounts read-only into the container. Mirrors Docker Swarm's native convention.

Docker API Support (docker-java 3.4.1)

Three Approaches

Feature withTmpFs() withMounts(TMPFS) Bind from /dev/shm
Pre-populated with secrets No No Yes
RAM-backed Yes Yes Yes (host is tmpfs)
noexec,nosuid Yes (string opts) Via mode only Via host mount opts
Read-only Yes (ro) Yes (withReadOnly) Yes (AccessMode.ro)
Size limit Yes (size=Nm) Yes (sizeBytes) Via host tmpfs mount

Key finding: You cannot combine bind mount with tmpfs properties in a single mount — they are mutually exclusive MountType values. But you can bind-mount a directory on host tmpfs (/dev/shm), achieving RAM-backed indirectly.

Recommendation: Bind from /dev/shm (Option C) — only approach that pre-populates secret files before container process reads them.

Lifecycle

1. Server creates temp dir: /dev/shm/cameleer-secrets-<uuid>/
2. Server writes secret files (mode 0400)
3. CreateContainerCmd with bind mount: dir → /run/secrets (ro)
4. StartContainerCmd
5. Delete /dev/shm/cameleer-secrets-<uuid>/ from host
6. Container still has access (bind mount holds inode reference)

Race Conditions

Scenario Risk Mitigation
Container reads before files written Impossible (files written before create) N/A
Host cleanup before container reads Low Delete after start returns; inode ref held
Concurrent deployments None UUID in dir name
Server crash between write and cleanup Low /dev/shm cleared on reboot + startup cleanup
Container restart HIGH Docker re-mounts from original path — if deleted, mount is empty/broken

Container Restart Problem (Primary Weakness)

Docker's restart policy re-mounts from the original source path. If the orchestrator deleted the host-side files, restarts will fail.

Mitigations:

  1. Don't delete host files while container is running (accept longer exposure)
  2. Handle restarts in orchestrator (Cameleer already uses onFailureRestart)
  3. Cleanup daemon only deletes dirs for stopped containers
  4. Re-populate secret files on observed restart (via DockerEventMonitor)

Security Analysis

Exposure Duration Comparison

Approach Duration on Host Can Reduce to Zero?
Environment variables Entire container lifetime No
Bind from /dev/shm Milliseconds to seconds Effectively yes
Docker Swarm secrets Encrypted at rest; decrypted only in container tmpfs Managed by Swarm

Attack Surface Comparison

Vector Environment Variables Tmpfs Secret Files
docker inspect Exposed (all plaintext) Not exposed (paths visible, not contents)
/proc/<pid>/environ Exposed Not exposed (secrets in files, not env)
Container escape Immediately available Only if /run/secrets accessible
Host compromise Visible via Docker API Only during brief write window
Child process inheritance All env vars inherited Files must be explicitly opened
Log/crash dump leakage Often logged by frameworks Only if app explicitly logs file contents
docker commit Embedded in image ENV layer Not captured (tmpfs excluded)

Tmpfs Security Caveat

Swap risk: Docker docs warn "Data may be written to swap and thereby persisted to the filesystem." Under memory pressure, tmpfs contents can be swapped to disk. Mitigate: disable swap (swapoff -a) or use encrypted swap.

vs Docker Swarm Internal Mechanism

Property Swarm Secrets Our Approach
Encrypted at rest Yes (AES-256 in Raft) No (plaintext in RAM)
Encrypted in transit Yes (mTLS) N/A (local only)
Mount location /run/secrets/<name> /run/secrets/<name> (compatible!)
Standalone Docker No (Swarm only) Yes
K8s support No Via emptyDir: {medium: Memory}

Docker-in-Docker (SaaS Mode) — Critical

Our SaaS server runs inside Docker and creates sibling containers. This creates a path resolution problem:

  • Server writes to /dev/shm/foo (container's tmpfs)
  • Docker bind mount resolves /dev/shm/foo on the host, not the server container
  • Host's /dev/shm/foo doesn't exist → mount fails

Solutions (same pattern as existing JARDOCKERVOLUME):

  1. Docker volume staging — create named volume, write files, mount into child container
  2. Shared host-path volume mounted in both server and child containers
  3. withTmpFs() + entrypoint wrapper that reads env vars into files then unsets them

Platform Compatibility

Platform Approach Notes
Docker standalone Bind from /dev/shm Primary target, works naturally
Docker Swarm Complementary (Swarm has native secrets) Use native for server creds, tmpfs for customer containers
Kubernetes emptyDir: {medium: Memory} or native Secret volumes Both are tmpfs-backed automatically
Docker-in-Docker Volume staging or entrypoint wrapper Needs the same pattern as JAR volume

K8s note: emptyDir with medium: Memory does NOT set nosuid,nodev,noexec by default (kubernetes/kubernetes#48912, open since 2017). Workaround: securityContext.readOnlyRootFilesystem: true.

State of the Art

Who Uses This Pattern

Platform Approach
Docker Swarm Native tmpfs secrets at /run/secrets/
Kubernetes Secret volumes (tmpfs-backed)
HashiCorp Vault Agent Sidecar writes to shared emptyDir: {medium: Memory}
Podman --secret flag (standalone, mounts to /run/secrets/)
Docker Compose v3.8+ secrets: key with file source

CIS Docker Benchmark (v1.7.0, 2025)

Control Recommendation
5.10 Do not store secrets in environment variables (our current approach violates this)
5.12 Mount container root filesystem as read-only
5.31 Do not mount sensitive host directories; use tmpfs

OWASP Secrets Management

The _FILE convention (e.g., POSTGRES_PASSWORD_FILE=/run/secrets/db-password) is widely adopted by official Docker images and recommended by OWASP.

Implementation Plan

Phase 1: Tmpfs + entrypoint wrapper (low effort, immediate benefit)

  • Classify env vars: CAMELEER_AGENT_AUTH_TOKEN is a secret; CAMELEER_AGENT_APPLICATION is config
  • Use withTmpFs("/run/secrets", "rw,noexec,nosuid,size=1m") + entrypoint wrapper to write secrets from env vars to files, then unset vars
  • Not perfect (secrets briefly in env vars) but eliminates docker inspect and /proc/environ exposure

Phase 2: _FILE convention in agent (medium effort)

  • Agent supports CAMELEER_AGENT_AUTH_TOKEN_FILE=/run/secrets/auth-token convention
  • Agent reads from file path if _FILE suffix set
  • Requires change in cameleer3 agent repo

Phase 3: Platform-adaptive injection (higher effort, production-grade)

  • Docker standalone: bind from /dev/shm or Docker volume staging
  • Kubernetes: native Secret resources
  • DinD mode: Docker volume + cleanup daemon
  • @Scheduled cleanup task every 60s for orphaned secret dirs

Recommendation

Verdict: ⭐⭐⭐½ (3.5/5)

Criterion Rating Notes
Security improvement over env vars 5/5 Eliminates docker inspect, /proc/environ, log leakage, docker commit exposure
Implementation complexity 3/5 Cleanup daemon, DinD path mapping, agent _FILE support
Platform compatibility 3/5 Works everywhere but needs per-platform adaptation
Restart resilience 2/5 Weakest point — container restarts break if secret files cleaned up
Industry alignment 4/5 Matches Swarm, K8s, OWASP, CIS Benchmark
DinD compatibility 3/5 Requires volume staging, not simple bind mount

This is a solid security upgrade and aligns with industry practices. The restart resilience problem is the main caveat. Combines well with Option 6 (callback pattern) — the callback delivers secrets, the tmpfs mount stores them without env var exposure.

What NOT to Do

  • Don't try to combine bind mount + tmpfs properties in a single mount
  • Don't delete host-side secret files while container is running with restart policy
  • Don't use a separate secrets manager just for file-based delivery (Vault is overkill for this alone)

Sources

Parent epic: #129 ## Overview Replace plaintext environment variables with secret files mounted into containers at `/run/secrets/`. The server writes secrets to a tmpfs-backed directory, bind-mounts read-only into the container. Mirrors Docker Swarm's native convention. --- ## Docker API Support (docker-java 3.4.1) ### Three Approaches | Feature | `withTmpFs()` | `withMounts(TMPFS)` | Bind from `/dev/shm` | |---|---|---|---| | Pre-populated with secrets | No | No | **Yes** | | RAM-backed | Yes | Yes | Yes (host is tmpfs) | | `noexec,nosuid` | Yes (string opts) | Via `mode` only | Via host mount opts | | Read-only | Yes (`ro`) | Yes (`withReadOnly`) | Yes (`AccessMode.ro`) | | Size limit | Yes (`size=Nm`) | Yes (`sizeBytes`) | Via host tmpfs mount | **Key finding:** You **cannot combine bind mount with tmpfs properties** in a single mount — they are mutually exclusive `MountType` values. But you can bind-mount a directory on host tmpfs (`/dev/shm`), achieving RAM-backed indirectly. **Recommendation:** Bind from `/dev/shm` (Option C) — only approach that pre-populates secret files before container process reads them. --- ## Lifecycle ``` 1. Server creates temp dir: /dev/shm/cameleer-secrets-<uuid>/ 2. Server writes secret files (mode 0400) 3. CreateContainerCmd with bind mount: dir → /run/secrets (ro) 4. StartContainerCmd 5. Delete /dev/shm/cameleer-secrets-<uuid>/ from host 6. Container still has access (bind mount holds inode reference) ``` ### Race Conditions | Scenario | Risk | Mitigation | |---|---|---| | Container reads before files written | Impossible (files written before create) | N/A | | Host cleanup before container reads | Low | Delete after start returns; inode ref held | | Concurrent deployments | None | UUID in dir name | | Server crash between write and cleanup | Low | `/dev/shm` cleared on reboot + startup cleanup | | **Container restart** | **HIGH** | Docker re-mounts from original path — if deleted, mount is empty/broken | ### Container Restart Problem (Primary Weakness) Docker's restart policy re-mounts from the original source path. If the orchestrator deleted the host-side files, restarts will fail. **Mitigations:** 1. Don't delete host files while container is running (accept longer exposure) 2. Handle restarts in orchestrator (Cameleer already uses `onFailureRestart`) 3. Cleanup daemon only deletes dirs for stopped containers 4. Re-populate secret files on observed restart (via `DockerEventMonitor`) --- ## Security Analysis ### Exposure Duration Comparison | Approach | Duration on Host | Can Reduce to Zero? | |---|---|---| | Environment variables | **Entire container lifetime** | No | | Bind from `/dev/shm` | **Milliseconds to seconds** | Effectively yes | | Docker Swarm secrets | Encrypted at rest; decrypted only in container tmpfs | Managed by Swarm | ### Attack Surface Comparison | Vector | Environment Variables | Tmpfs Secret Files | |---|---|---| | `docker inspect` | **Exposed** (all plaintext) | **Not exposed** (paths visible, not contents) | | `/proc/<pid>/environ` | **Exposed** | **Not exposed** (secrets in files, not env) | | Container escape | Immediately available | Only if `/run/secrets` accessible | | Host compromise | Visible via Docker API | Only during brief write window | | Child process inheritance | **All env vars inherited** | Files must be explicitly opened | | Log/crash dump leakage | Often logged by frameworks | Only if app explicitly logs file contents | | `docker commit` | **Embedded in image** ENV layer | **Not captured** (tmpfs excluded) | ### Tmpfs Security Caveat **Swap risk:** Docker docs warn "Data may be written to swap and thereby persisted to the filesystem." Under memory pressure, tmpfs contents can be swapped to disk. Mitigate: disable swap (`swapoff -a`) or use encrypted swap. ### vs Docker Swarm Internal Mechanism | Property | Swarm Secrets | Our Approach | |---|---|---| | Encrypted at rest | Yes (AES-256 in Raft) | No (plaintext in RAM) | | Encrypted in transit | Yes (mTLS) | N/A (local only) | | Mount location | `/run/secrets/<name>` | `/run/secrets/<name>` (compatible!) | | Standalone Docker | **No** (Swarm only) | **Yes** | | K8s support | **No** | Via `emptyDir: {medium: Memory}` | --- ## Docker-in-Docker (SaaS Mode) — Critical Our SaaS server runs inside Docker and creates sibling containers. This creates a path resolution problem: - Server writes to `/dev/shm/foo` (container's tmpfs) - Docker bind mount resolves `/dev/shm/foo` on the **host**, not the server container - Host's `/dev/shm/foo` doesn't exist → mount fails **Solutions (same pattern as existing `JARDOCKERVOLUME`):** 1. Docker volume staging — create named volume, write files, mount into child container 2. Shared host-path volume mounted in both server and child containers 3. `withTmpFs()` + entrypoint wrapper that reads env vars into files then unsets them --- ## Platform Compatibility | Platform | Approach | Notes | |---|---|---| | Docker standalone | Bind from `/dev/shm` | Primary target, works naturally | | Docker Swarm | Complementary (Swarm has native secrets) | Use native for server creds, tmpfs for customer containers | | Kubernetes | `emptyDir: {medium: Memory}` or native `Secret` volumes | Both are tmpfs-backed automatically | | Docker-in-Docker | Volume staging or entrypoint wrapper | Needs the same pattern as JAR volume | **K8s note:** `emptyDir` with `medium: Memory` does NOT set `nosuid,nodev,noexec` by default (kubernetes/kubernetes#48912, open since 2017). Workaround: `securityContext.readOnlyRootFilesystem: true`. --- ## State of the Art ### Who Uses This Pattern | Platform | Approach | |---|---| | Docker Swarm | Native tmpfs secrets at `/run/secrets/` | | Kubernetes | Secret volumes (tmpfs-backed) | | HashiCorp Vault Agent | Sidecar writes to shared `emptyDir: {medium: Memory}` | | Podman | `--secret` flag (standalone, mounts to `/run/secrets/`) | | Docker Compose v3.8+ | `secrets:` key with file source | ### CIS Docker Benchmark (v1.7.0, 2025) | Control | Recommendation | |---|---| | 5.10 | **Do not store secrets in environment variables** (our current approach violates this) | | 5.12 | Mount container root filesystem as read-only | | 5.31 | Do not mount sensitive host directories; use tmpfs | ### OWASP Secrets Management The `_FILE` convention (e.g., `POSTGRES_PASSWORD_FILE=/run/secrets/db-password`) is widely adopted by official Docker images and recommended by OWASP. --- ## Implementation Plan ### Phase 1: Tmpfs + entrypoint wrapper (low effort, immediate benefit) - Classify env vars: `CAMELEER_AGENT_AUTH_TOKEN` is a secret; `CAMELEER_AGENT_APPLICATION` is config - Use `withTmpFs("/run/secrets", "rw,noexec,nosuid,size=1m")` + entrypoint wrapper to write secrets from env vars to files, then unset vars - Not perfect (secrets briefly in env vars) but eliminates `docker inspect` and `/proc/environ` exposure ### Phase 2: `_FILE` convention in agent (medium effort) - Agent supports `CAMELEER_AGENT_AUTH_TOKEN_FILE=/run/secrets/auth-token` convention - Agent reads from file path if `_FILE` suffix set - Requires change in cameleer3 agent repo ### Phase 3: Platform-adaptive injection (higher effort, production-grade) - Docker standalone: bind from `/dev/shm` or Docker volume staging - Kubernetes: native `Secret` resources - DinD mode: Docker volume + cleanup daemon - `@Scheduled` cleanup task every 60s for orphaned secret dirs --- ## Recommendation ### Verdict: ⭐⭐⭐½ (3.5/5) | Criterion | Rating | Notes | |-----------|:---:|-------| | Security improvement over env vars | 5/5 | Eliminates docker inspect, /proc/environ, log leakage, docker commit exposure | | Implementation complexity | 3/5 | Cleanup daemon, DinD path mapping, agent `_FILE` support | | Platform compatibility | 3/5 | Works everywhere but needs per-platform adaptation | | **Restart resilience** | **2/5** | **Weakest point — container restarts break if secret files cleaned up** | | Industry alignment | 4/5 | Matches Swarm, K8s, OWASP, CIS Benchmark | | DinD compatibility | 3/5 | Requires volume staging, not simple bind mount | **This is a solid security upgrade** and aligns with industry practices. The restart resilience problem is the main caveat. Combines well with Option 6 (callback pattern) — the callback delivers secrets, the tmpfs mount stores them without env var exposure. ### What NOT to Do - Don't try to combine bind mount + tmpfs properties in a single mount - Don't delete host-side secret files while container is running with restart policy - Don't use a separate secrets manager just for file-based delivery (Vault is overkill for this alone) ### Sources - [Docker tmpfs Mounts](https://docs.docker.com/engine/storage/tmpfs/) - [Docker Secrets Documentation](https://docs.docker.com/engine/swarm/secrets/) - [OWASP Docker Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) - [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) - [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker) - [Docker Secrets Security (Wiz)](https://www.wiz.io/academy/container-security/docker-secrets) - [Docker Secrets Without Swarm](https://www.mykolaaleksandrov.dev/posts/2025/08/secure-secrets-docker-windows-linux/) - [K8s emptyDir nosuid issue #48912](https://github.com/kubernetes/kubernetes/issues/48912) - [docker-java tmpfs Support #818](https://github.com/docker-java/docker-java/issues/818)
claude added the featuresecurity labels 2026-04-15 00:35:48 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
Accessibility (a11y) improvements
 bug
Something isn't working correctly
 cleanup
Code cleanup, dead code removal, refactoring
 epic
Epic: groups related issues
 feature
New functionality
 pmf
Product-market fit: critical for first market offer
 route-diagram
Route diagram page and execution overlay
 security
Authentication, authorization, RBAC
 ui
Frontend / React UI
 ux
UX/usability improvements
No Label feature security
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cameleer/cameleer-server#134
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?