Secret delivery option 5: Per-container JWT with encrypted secret claims (JWE) #133

New Issue

claude · 2026-04-15T00:34:40+02:00

claude commented

2026-04-15 00:34:40 +02:00

Parent epic: #129

Overview

Mint a short-lived JWE (JSON Web Encryption) token per container containing encrypted secret claims. Pass as a single env var CAMELEER_BOOTSTRAP_JWT. Container decrypts on startup to extract secrets. Leverages our existing Nimbus JOSE library.

Approach Design

Token Structure

JWE: {"alg":"dir","enc":"A256GCM"}
Payload: {
  "iss": "cameleer3-server",
  "sub": "deployment:<uuid>",
  "exp": <10min from now>,
  "jti": "<unique nonce>",
  "secrets": { "DB_PASSWORD": "s3cret", "API_KEY": "ak_live_..." },
  "config": { "CAMELEER_AGENT_EXPORT_ENDPOINT": "http://..." }
}

Direct encryption with AES-256-GCM (dir + A256GCM) — simplest symmetric JWE mode. Nimbus JOSE v9.47 (our current dep) has full support via DirectEncrypter / DirectDecrypter.

Token Size Capacity

Platform	Limit	Practical JWT capacity
Linux `execve()` single arg	128 KiB	~95 KiB payload after JWE overhead
Docker env var	No Docker-specific limit (kernel limit)	Same
K8s Secret (etcd)	1 MiB	Ample

~50-80 key-value pairs fit easily. Typical container has 5-15 secrets — not a constraint.

Security Analysis

Honest Comparison: JWE vs Plain Env Vars

Threat	Plain Env Vars	JWE Token
`docker inspect` reveals secrets	YES	NO (opaque ciphertext)
Container runtime API leaks	YES	NO
Logging/monitoring captures secrets	YES	NO
Process memory exposure (`/proc`)	YES	YES (after decryption)
Attacker with Docker socket	Full access	Full access (can exec into container)
Attacker with JWE key	N/A	Can decrypt any token
Rotation requires redeployment	YES	YES
Debugging ease	Easy	Hard (need JWE tooling)

Verdict: More than obscurity (real protection against metadata leakage, log exposure) but less than a secrets manager. Does NOT protect against shell access to running container or key compromise.

The Fatal Chicken-and-Egg Problem

The container needs the decryption key to read the JWE. Options:

Derive from CAMELEER_AGENT_AUTH_TOKEN (already an env var — circular)
Embed key in agent JAR (baked into artifact — bad)
Second env var CAMELEER_BOOTSTRAP_KEY (defeats the purpose)

You still need at least one cleartext secret as env var. If an attacker reads docker inspect, they get the bootstrap token, derive the AES key, and decrypt the JWE. Security improvement against docker inspect threat is marginal — protecting N secrets by exposing 1 key that unlocks all N.

Replay & Revocation

Mitigation	Mechanism	Practicality
`jti` + short expiry	Primary defense — 10-min window	Good
Deployment ID binding	`sub: "deployment:<uuid>"`	Natural
One-time use enforcement	Requires callback before agent has auth	Complex, marginal gain
Post-delivery revocation	Not achievable — token consumed before network available	Same as env vars

Accidental Logging

JWE token in logs appears as opaque base64url — genuine improvement. But once agent decrypts and sets env vars from the JWT contents, those values are in process memory and potentially re-exposed.

Industry Assessment

Does Anyone Use JWTs for Secret Delivery?

No. After extensive research, no production systems use JWE tokens for secret delivery to containers. Closest patterns:

Vault response wrapping — single-use wrapping token, but Vault is the backend
K8s service account tokens — JWT for identity, not secret transport
SPIFFE/SPIRE SVIDs — workload identity, not secrets

The absence of this pattern in production is itself a signal.

What Standards Say

Source	Guidance
RFC 8725 (JWT Best Practices)	"Avoid placing sensitive or business data in JWTs"
OWASP JWT Cheat Sheet	"Ensure no sensitive information exposed in the payload"
Security researchers	"Secrets in JWTs" consistently categorized as anti-pattern

While this guidance targets JWS (readable payloads), the spirit applies: JWTs are designed for authorization claims, not secret transport.

CVE Concerns

JWE implementations have demonstrated subtle vulnerabilities:

CVE-2026-29000 (pac4j): Nested JWT parsing bypass where inner alg=none tokens bypass signature verification
Adding JWE to the critical path of container startup introduces attack surface that doesn't exist with simpler approaches

Vault Response Wrapping Comparison

Feature	JWE Bootstrap Token	Vault Response Wrapping
External dependency	None	Vault server required
Single-use guarantee	No (replay within TTL)	Yes (cubbyhole destroyed)
Tamper detection	No	Yes
Malfeasance detection	No	Yes (second unwrap fails, alerts)
Secret rotation	Requires redeployment	Dynamic (lease renewal)
Offline operation	Yes	No
Complexity	Medium	High

Platform Compatibility

Platform	Delivery	Security Notes
Docker standalone	Env var via `withEnv()`	Visible in `docker inspect` (as ciphertext)
Docker Swarm	Swarm secret file	Encrypted at rest, memory-only mount
Kubernetes	K8s Secret mounted as file	Encrypted at rest (if configured)

If mounting as a file anyway (Swarm/K8s), the JWE encryption becomes redundant — platform-native encryption already protects it.

Implementation Complexity

Component	Effort	Risk
Server: JWE minting in DeploymentExecutor	Low (~50 lines, Nimbus already in deps)	Low
Server: Key derivation	Low (~20 lines, same pattern as Ed25519)	Low
Agent (Java): JWE decode in bootstrap	Medium (~100 lines + Nimbus dep)	Medium
Non-Java containers: JWE bootstrap	High (per-language SDK, init wrapper)	High
Testing: round-trip, expiry, error cases	Medium	Low
Migration: backward compat	Medium (dual-mode)	Medium

Total: 2-3 days Java-only, 5-7 days with non-Java support.

Recommendation

Verdict: ⭐⭐ (2/5) — Do Not Implement

The per-container JWE approach is technically sound but strategically misguided for our scenario:

Chicken-and-egg is fatal — still need one cleartext secret to bootstrap decryption. If attacker reads docker inspect, they get the key and decrypt everything. Marginal security improvement.
Industry has not adopted this — no major platform, runtime, or security vendor uses JWE for secret delivery. Universal recommendation: platform-native secrets or external vault.
RFC 8725 and OWASP advise against it — putting sensitive data in JWT claims explicitly cautioned.
CVE surface area increases — JWE implementations have subtle vulnerabilities. Adds attack surface to critical startup path.
Cross-language tax is permanent — every non-Java container needs JWE bootstrap code forever.

What To Do Instead

The better version of this idea is the API-based secret fetch (Option 6, #TBD): mint a short-lived identity JWT (not containing secrets), container calls GET /api/v1/bootstrap/{id}/secrets with it, receives secrets over TLS, server marks token consumed. This gives single-use guarantees, audit logging, and separation of identity from secrets — the Vault response-wrapping pattern without Vault.

Sources

Parent epic: #129 ## Overview Mint a short-lived JWE (JSON Web Encryption) token per container containing encrypted secret claims. Pass as a single env var `CAMELEER_BOOTSTRAP_JWT`. Container decrypts on startup to extract secrets. Leverages our existing Nimbus JOSE library. --- ## Approach Design ### Token Structure ``` JWE: {"alg":"dir","enc":"A256GCM"} Payload: { "iss": "cameleer3-server", "sub": "deployment:<uuid>", "exp": <10min from now>, "jti": "<unique nonce>", "secrets": { "DB_PASSWORD": "s3cret", "API_KEY": "ak_live_..." }, "config": { "CAMELEER_AGENT_EXPORT_ENDPOINT": "http://..." } } ``` Direct encryption with AES-256-GCM (`dir + A256GCM`) — simplest symmetric JWE mode. Nimbus JOSE v9.47 (our current dep) has full support via `DirectEncrypter` / `DirectDecrypter`. ### Token Size Capacity | Platform | Limit | Practical JWT capacity | |----------|-------|----------------------| | Linux `execve()` single arg | 128 KiB | ~95 KiB payload after JWE overhead | | Docker env var | No Docker-specific limit (kernel limit) | Same | | K8s Secret (etcd) | 1 MiB | Ample | **~50-80 key-value pairs** fit easily. Typical container has 5-15 secrets — not a constraint. --- ## Security Analysis ### Honest Comparison: JWE vs Plain Env Vars | Threat | Plain Env Vars | JWE Token | |--------|:-:|:-:| | `docker inspect` reveals secrets | **YES** | **NO** (opaque ciphertext) | | Container runtime API leaks | **YES** | **NO** | | Logging/monitoring captures secrets | **YES** | **NO** | | Process memory exposure (`/proc`) | **YES** | **YES** (after decryption) | | Attacker with Docker socket | Full access | Full access (can exec into container) | | Attacker with JWE key | N/A | Can decrypt any token | | Rotation requires redeployment | YES | YES | | Debugging ease | Easy | Hard (need JWE tooling) | **Verdict:** More than obscurity (real protection against metadata leakage, log exposure) but less than a secrets manager. Does NOT protect against shell access to running container or key compromise. ### The Fatal Chicken-and-Egg Problem The container needs the decryption key to read the JWE. Options: - Derive from `CAMELEER_AGENT_AUTH_TOKEN` (already an env var — **circular**) - Embed key in agent JAR (baked into artifact — **bad**) - Second env var `CAMELEER_BOOTSTRAP_KEY` (**defeats the purpose**) **You still need at least one cleartext secret as env var.** If an attacker reads `docker inspect`, they get the bootstrap token, derive the AES key, and decrypt the JWE. Security improvement against `docker inspect` threat is **marginal** — protecting N secrets by exposing 1 key that unlocks all N. ### Replay & Revocation | Mitigation | Mechanism | Practicality | |------------|-----------|--------------| | `jti` + short expiry | Primary defense — 10-min window | Good | | Deployment ID binding | `sub: "deployment:<uuid>"` | Natural | | One-time use enforcement | Requires callback before agent has auth | Complex, marginal gain | | Post-delivery revocation | **Not achievable** — token consumed before network available | Same as env vars | ### Accidental Logging JWE token in logs appears as opaque base64url — genuine improvement. But once agent decrypts and sets env vars from the JWT contents, those values are in process memory and potentially re-exposed. --- ## Industry Assessment ### Does Anyone Use JWTs for Secret Delivery? **No.** After extensive research, no production systems use JWE tokens for secret delivery to containers. Closest patterns: - **Vault response wrapping** — single-use wrapping token, but Vault is the backend - **K8s service account tokens** — JWT for identity, not secret transport - **SPIFFE/SPIRE SVIDs** — workload identity, not secrets **The absence of this pattern in production is itself a signal.** ### What Standards Say | Source | Guidance | |--------|---------| | **RFC 8725** (JWT Best Practices) | "Avoid placing sensitive or business data in JWTs" | | **OWASP JWT Cheat Sheet** | "Ensure no sensitive information exposed in the payload" | | **Security researchers** | "Secrets in JWTs" consistently categorized as anti-pattern | While this guidance targets JWS (readable payloads), the spirit applies: JWTs are designed for authorization claims, not secret transport. ### CVE Concerns JWE implementations have demonstrated subtle vulnerabilities: - **CVE-2026-29000** (pac4j): Nested JWT parsing bypass where inner `alg=none` tokens bypass signature verification - Adding JWE to the critical path of container startup introduces attack surface that doesn't exist with simpler approaches --- ## Vault Response Wrapping Comparison | Feature | JWE Bootstrap Token | Vault Response Wrapping | |---------|:-:|:-:| | External dependency | None | Vault server required | | Single-use guarantee | No (replay within TTL) | **Yes** (cubbyhole destroyed) | | Tamper detection | No | **Yes** | | Malfeasance detection | No | **Yes** (second unwrap fails, alerts) | | Secret rotation | Requires redeployment | Dynamic (lease renewal) | | Offline operation | Yes | No | | Complexity | Medium | High | --- ## Platform Compatibility | Platform | Delivery | Security Notes | |----------|----------|---------------| | Docker standalone | Env var via `withEnv()` | Visible in `docker inspect` (as ciphertext) | | Docker Swarm | Swarm secret file | Encrypted at rest, memory-only mount | | Kubernetes | K8s Secret mounted as file | Encrypted at rest (if configured) | **If mounting as a file anyway** (Swarm/K8s), the JWE encryption becomes redundant — platform-native encryption already protects it. --- ## Implementation Complexity | Component | Effort | Risk | |-----------|--------|------| | Server: JWE minting in DeploymentExecutor | Low (~50 lines, Nimbus already in deps) | Low | | Server: Key derivation | Low (~20 lines, same pattern as Ed25519) | Low | | Agent (Java): JWE decode in bootstrap | Medium (~100 lines + Nimbus dep) | Medium | | **Non-Java containers: JWE bootstrap** | **High** (per-language SDK, init wrapper) | **High** | | Testing: round-trip, expiry, error cases | Medium | Low | | Migration: backward compat | Medium (dual-mode) | Medium | **Total: 2-3 days Java-only, 5-7 days with non-Java support.** --- ## Recommendation ### Verdict: ⭐⭐ (2/5) — Do Not Implement The per-container JWE approach is **technically sound** but **strategically misguided** for our scenario: 1. **Chicken-and-egg is fatal** — still need one cleartext secret to bootstrap decryption. If attacker reads `docker inspect`, they get the key and decrypt everything. Marginal security improvement. 2. **Industry has not adopted this** — no major platform, runtime, or security vendor uses JWE for secret delivery. Universal recommendation: platform-native secrets or external vault. 3. **RFC 8725 and OWASP advise against it** — putting sensitive data in JWT claims explicitly cautioned. 4. **CVE surface area increases** — JWE implementations have subtle vulnerabilities. Adds attack surface to critical startup path. 5. **Cross-language tax is permanent** — every non-Java container needs JWE bootstrap code forever. ### What To Do Instead The better version of this idea is the **API-based secret fetch** (Option 6, #TBD): mint a short-lived identity JWT (not containing secrets), container calls `GET /api/v1/bootstrap/{id}/secrets` with it, receives secrets over TLS, server marks token consumed. This gives single-use guarantees, audit logging, and separation of identity from secrets — the Vault response-wrapping pattern without Vault. ### Sources - [RFC 8725: JWT Best Current Practices](https://www.rfc-editor.org/rfc/rfc8725.html) - [RFC 7516: JSON Web Encryption](https://www.rfc-editor.org/rfc/rfc7516) - [OWASP JWT for Java Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/JSON_Web_Token_for_Java_Cheat_Sheet.html) - [CVE-2026-29000: pac4j JWE bypass](https://nvd.nist.gov/vuln/detail/CVE-2026-29000) - [Nimbus JOSE+JWT Library](https://connect2id.com/products/nimbus-jose-jwt) - [Vault Response Wrapping](https://developer.hashicorp.com/vault/docs/concepts/response-wrapping) - [SPIFFE/SPIRE Comparisons](https://spiffe.io/docs/latest/spire-about/comparisons/) - [Docker Secrets Explained (Wiz)](https://www.wiz.io/academy/container-security/docker-secrets) - [Linux Env Var Size Constraints](https://www.baeldung.com/linux/environment-variable-size-constraints)

claude added the feature security labels 2026-04-15 00:34:40 +02:00

claude referenced this issue

2026-04-15 00:37:56 +02:00

Epic: Secure secret delivery to provisioned containers #129

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: cameleer/cameleer-server#133