cameleer/cameleer-server
1
0
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Wiki Activity

Secret delivery option 1: Platform-native secrets (Docker Swarm + K8s Secrets) #130

New Issue
Open
opened 2026-04-15 00:31:01 +02:00 by claude · 0 comments
claude commented 2026-04-15 00:31:01 +02:00
Owner
Copy Link

Parent epic: #129

Overview

Use Docker Swarm's built-in secret management and Kubernetes Secrets to deliver secrets to provisioned containers, leveraging each platform's native encryption and access control.

Docker Swarm Secrets

How They Work

Aspect Detail
Storage Secrets stored in the Raft log, encrypted at rest, replicated across all manager nodes
Encryption at rest NaCl Secretbox (XSalsa20-Poly1305). FIPS mode uses AES-128-CBC via Fernet
Encryption in transit All manager-to-manager and manager-to-worker communication uses mutual TLS
Delivery to containers Decrypted secret mounted as tmpfs file at /run/secrets/<name>
Size limit 500 KB per secret
Access control Secret only accessible to services explicitly granted access
Autolock Optional --autolock requires unlock key after manager restart

docker-java Library Support

The docker-java library (v3.4.1, our current version) does support Swarm secret operations:

Command Class Docker API
Create secret CreateSecretCmd 1.25+
List secrets ListSecretsCmd 1.25+
Remove secret RemoveSecretCmd 1.25+

Critical Limitation: Services Only

Docker Swarm secrets are only available to Swarm services, NOT standalone containers. dockerClient.createContainerCmd() cannot attach secrets — only dockerClient.createServiceCmd() with ServiceSpec can.

This means our current architecture using createContainerCmd() in DockerRuntimeOrchestrator cannot use Swarm secrets without a major refactor to the Swarm service model (different lifecycle, networking, replica management).

Rotation

No built-in versioning. Pattern: create new secret (db_password_v2), update service to swap old→new, delete old. Requires rolling service updates.

Kubernetes Secrets

Encryption at Rest

Configuration Detail
Default Base64-encoded plaintext in etcd — no encryption
EncryptionConfiguration YAML config via --encryption-provider-config on kube-apiserver
Providers identity (none), secretbox (XSalsa20-Poly1305), aescbc, aesgcm, kms v1/v2
KMS v2 (recommended) Envelope encryption: K8s generates DEK per resource, KMS encrypts the DEK
K3s Must start with --secrets-encryption flag. Default: aescbc. Cannot enable after initial start without re-provisioning

Delivery Mechanisms

Method Security Hot Reload Best Practice
Env vars (secretKeyRef) Visible in kubectl describe pod, /proc/*/environ No (pod restart) Not recommended
Volume mounts (secretVolumeSource) Files on tmpfs, kubelet-managed lifecycle Yes (kubelet auto-updates) Recommended
CSI Secret Store Driver Integrates with external vaults, mounts as volumes Yes (periodic) Best practice for production

External Secrets Operator (ESO)

Aspect Status
CNCF status Sandbox (July 2022)
Current state Development paused since August 2025. Team too small. Merging community PRs but no new releases
Risk High adoption but sustainability concerns. Not recommended for new projects

Sealed Secrets (Bitnami)

Actively maintained (v0.36.5, April 2026). Solves GitOps secret storage, not runtime secret delivery — low relevance for our use case. Recent CVE-2026-22728 (scope widening) fixed in v0.36.0+.

Programmatic Creation (Java)

Fabric8 Kubernetes Client (most popular Java K8s client, good Spring Boot integration):

client.secrets().inNamespace("tenant-ns")
    .createOrReplace(new SecretBuilder()
        .withNewMetadata().withName("app-db-creds").endMetadata()
        .withStringData(Map.of("password", "s3cret"))
        .build());

Natural fit for Cameleer3's deployment flow: create Secret at PRE_FLIGHT → reference in PodSpec as volume mount.

Cross-Platform Analysis

Abstraction Feasibility

Platform Mechanism Container Creation Abstraction Complexity
Docker standalone No native secrets. Bind-mount files or env vars createContainerCmd() Low — can only bind-mount files
Docker Swarm Swarm secrets (Raft-encrypted, tmpfs) Must use createServiceCmd() High — requires container→service migration
Kubernetes K8s Secrets (etcd, optional encryption) Secret + Pod with volume mount Moderate — requires fabric8 dependency

Single-codebase abstraction is difficult. Swarm's service-only requirement is the main obstacle.

What Major Platforms Do

Platform Approach
Portainer Swarm secrets UI for Swarm mode. Standalone: falls back to env vars
Rancher Full K8s secrets via UI/API. External managers via helm charts
Coolify Secrets encrypted in its database, injected as env vars at runtime. No Swarm integration
CapRover Environment variables only. No native secret management

Key insight: Most platforms at our scale (Coolify, CapRover) use encrypted-at-rest DB + env var injection. Only enterprise platforms use platform-native secrets.

Security Assessment

NIST / CIS Guidance

Standard Guidance
NIST SP 800-190 "Do not embed plaintext secrets in images. Use secure secret management."
CIS Docker Benchmark Section 3.22: "Do not store secrets in environment variables." Recommends Docker secrets or external vault
CIS Kubernetes Benchmark 5.4.1: "Prefer secrets as files over secrets as environment variables." Recommends EncryptionConfiguration
Google GKE Policy Enforces no-secrets-as-env-vars via Policy Controller

Env Var Attack Vectors

Vector Severity Detail
docker inspect HIGH Docker socket access reveals all env vars
/proc/*/environ HIGH Any process in container can read all env vars via procfs
Container logs MEDIUM Env vars leak into stack traces, debug logs, crash dumps
kubectl describe pod MEDIUM Shows env var values in plaintext
Child processes MEDIUM All child processes inherit environment

Delivery Mechanism Comparison

Criteria Env Vars File Mount (tmpfs) API Fetch (Vault/ESO)
Encryption at rest None Platform-dependent Yes (in vault)
Visibility to processes Full (/proc/*/environ) Only file readers Only fetching process
Leak in inspect/describe Yes No No
Leak in logs/crashes Common Rare Rare
Hot reload No Yes (K8s: automatic) Yes (TTL/lease)
Rotation Container restart File update Transparent
Complexity Trivial Low-Moderate High
Cross-platform Universal Per-platform impl Requires vault infra

Recommendation

Verdict: ⭐⭐ (2/5) as standalone strategy

Docker Swarm secrets are incompatible with our createContainerCmd() architecture. Migrating to createServiceCmd() is a major refactor with significant behavioral changes.

K8s Secrets are a good fit when on K8s (4/5 alone), but don't help on Docker standalone or Swarm without K8s.

Cannot serve as the sole strategy across all three platforms. Better as a Phase 2 enhancement for K8s deployments, combined with a file-mount approach for Docker standalone.

What To Do

  • Do NOT adopt Swarm secrets as primary strategy — service-only limitation is a blocker
  • Do NOT adopt ESO — development paused since August 2025
  • DO plan K8s Secrets as a Phase 2 enhancement (create Secret + volume mount via fabric8)
  • DO enable K3s --secrets-encryption on our cluster if not already done
  • File-mount approach (see #TBD) works identically on all platforms and is forward-compatible with both Swarm secrets and K8s volume mounts

Sources

Parent epic: #129 ## Overview Use Docker Swarm's built-in secret management and Kubernetes Secrets to deliver secrets to provisioned containers, leveraging each platform's native encryption and access control. --- ## Docker Swarm Secrets ### How They Work | Aspect | Detail | |--------|--------| | **Storage** | Secrets stored in the Raft log, encrypted at rest, replicated across all manager nodes | | **Encryption at rest** | NaCl Secretbox (XSalsa20-Poly1305). FIPS mode uses AES-128-CBC via Fernet | | **Encryption in transit** | All manager-to-manager and manager-to-worker communication uses mutual TLS | | **Delivery to containers** | Decrypted secret mounted as tmpfs file at `/run/secrets/<name>` | | **Size limit** | 500 KB per secret | | **Access control** | Secret only accessible to services explicitly granted access | | **Autolock** | Optional `--autolock` requires unlock key after manager restart | ### docker-java Library Support The docker-java library (v3.4.1, our current version) **does** support Swarm secret operations: | Command | Class | Docker API | |---------|-------|------------| | Create secret | `CreateSecretCmd` | 1.25+ | | List secrets | `ListSecretsCmd` | 1.25+ | | Remove secret | `RemoveSecretCmd` | 1.25+ | ### Critical Limitation: Services Only **Docker Swarm secrets are only available to Swarm services, NOT standalone containers.** `dockerClient.createContainerCmd()` cannot attach secrets — only `dockerClient.createServiceCmd()` with `ServiceSpec` can. This means our current architecture using `createContainerCmd()` in `DockerRuntimeOrchestrator` **cannot use Swarm secrets** without a major refactor to the Swarm service model (different lifecycle, networking, replica management). ### Rotation No built-in versioning. Pattern: create new secret (`db_password_v2`), update service to swap old→new, delete old. Requires rolling service updates. --- ## Kubernetes Secrets ### Encryption at Rest | Configuration | Detail | |---------------|--------| | **Default** | Base64-encoded plaintext in etcd — **no encryption** | | **EncryptionConfiguration** | YAML config via `--encryption-provider-config` on kube-apiserver | | **Providers** | `identity` (none), `secretbox` (XSalsa20-Poly1305), `aescbc`, `aesgcm`, `kms` v1/v2 | | **KMS v2 (recommended)** | Envelope encryption: K8s generates DEK per resource, KMS encrypts the DEK | | **K3s** | Must start with `--secrets-encryption` flag. Default: `aescbc`. Cannot enable after initial start without re-provisioning | ### Delivery Mechanisms | Method | Security | Hot Reload | Best Practice | |--------|----------|------------|---------------| | **Env vars** (`secretKeyRef`) | Visible in `kubectl describe pod`, `/proc/*/environ` | No (pod restart) | **Not recommended** | | **Volume mounts** (`secretVolumeSource`) | Files on tmpfs, kubelet-managed lifecycle | Yes (kubelet auto-updates) | **Recommended** | | **CSI Secret Store Driver** | Integrates with external vaults, mounts as volumes | Yes (periodic) | **Best practice for production** | ### External Secrets Operator (ESO) | Aspect | Status | |--------|--------| | CNCF status | Sandbox (July 2022) | | Current state | **Development paused** since August 2025. Team too small. Merging community PRs but no new releases | | Risk | High adoption but sustainability concerns. **Not recommended for new projects** | ### Sealed Secrets (Bitnami) Actively maintained (v0.36.5, April 2026). Solves GitOps secret storage, **not runtime secret delivery** — low relevance for our use case. Recent CVE-2026-22728 (scope widening) fixed in v0.36.0+. ### Programmatic Creation (Java) **Fabric8 Kubernetes Client** (most popular Java K8s client, good Spring Boot integration): ```java client.secrets().inNamespace("tenant-ns") .createOrReplace(new SecretBuilder() .withNewMetadata().withName("app-db-creds").endMetadata() .withStringData(Map.of("password", "s3cret")) .build()); ``` Natural fit for Cameleer3's deployment flow: create Secret at PRE_FLIGHT → reference in PodSpec as volume mount. --- ## Cross-Platform Analysis ### Abstraction Feasibility | Platform | Mechanism | Container Creation | Abstraction Complexity | |----------|-----------|-------------------|----------------------| | **Docker standalone** | No native secrets. Bind-mount files or env vars | `createContainerCmd()` | Low — can only bind-mount files | | **Docker Swarm** | Swarm secrets (Raft-encrypted, tmpfs) | Must use `createServiceCmd()` | **High — requires container→service migration** | | **Kubernetes** | K8s Secrets (etcd, optional encryption) | Secret + Pod with volume mount | Moderate — requires fabric8 dependency | **Single-codebase abstraction is difficult.** Swarm's service-only requirement is the main obstacle. ### What Major Platforms Do | Platform | Approach | |----------|----------| | **Portainer** | Swarm secrets UI for Swarm mode. Standalone: falls back to env vars | | **Rancher** | Full K8s secrets via UI/API. External managers via helm charts | | **Coolify** | Secrets **encrypted in its database**, injected as env vars at runtime. No Swarm integration | | **CapRover** | Environment variables only. No native secret management | **Key insight:** Most platforms at our scale (Coolify, CapRover) use encrypted-at-rest DB + env var injection. Only enterprise platforms use platform-native secrets. --- ## Security Assessment ### NIST / CIS Guidance | Standard | Guidance | |----------|----------| | **NIST SP 800-190** | "Do not embed plaintext secrets in images. Use secure secret management." | | **CIS Docker Benchmark** | Section 3.22: "Do not store secrets in environment variables." Recommends Docker secrets or external vault | | **CIS Kubernetes Benchmark** | 5.4.1: "Prefer secrets as files over secrets as environment variables." Recommends EncryptionConfiguration | | **Google GKE Policy** | Enforces `no-secrets-as-env-vars` via Policy Controller | ### Env Var Attack Vectors | Vector | Severity | Detail | |--------|----------|--------| | `docker inspect` | HIGH | Docker socket access reveals all env vars | | `/proc/*/environ` | HIGH | Any process in container can read all env vars via procfs | | Container logs | MEDIUM | Env vars leak into stack traces, debug logs, crash dumps | | `kubectl describe pod` | MEDIUM | Shows env var values in plaintext | | Child processes | MEDIUM | All child processes inherit environment | ### Delivery Mechanism Comparison | Criteria | Env Vars | File Mount (tmpfs) | API Fetch (Vault/ESO) | |----------|----------|-------------------|----------------------| | Encryption at rest | None | Platform-dependent | Yes (in vault) | | Visibility to processes | Full (`/proc/*/environ`) | Only file readers | Only fetching process | | Leak in inspect/describe | Yes | No | No | | Leak in logs/crashes | Common | Rare | Rare | | Hot reload | No | Yes (K8s: automatic) | Yes (TTL/lease) | | Rotation | Container restart | File update | Transparent | | Complexity | Trivial | Low-Moderate | High | | Cross-platform | Universal | Per-platform impl | Requires vault infra | --- ## Recommendation ### Verdict: ⭐⭐ (2/5) as standalone strategy **Docker Swarm secrets are incompatible** with our `createContainerCmd()` architecture. Migrating to `createServiceCmd()` is a major refactor with significant behavioral changes. **K8s Secrets are a good fit when on K8s** (4/5 alone), but don't help on Docker standalone or Swarm without K8s. **Cannot serve as the sole strategy** across all three platforms. Better as a Phase 2 enhancement for K8s deployments, combined with a file-mount approach for Docker standalone. ### What To Do - **Do NOT adopt Swarm secrets** as primary strategy — service-only limitation is a blocker - **Do NOT adopt ESO** — development paused since August 2025 - **DO plan K8s Secrets as a Phase 2** enhancement (create Secret + volume mount via fabric8) - **DO enable K3s `--secrets-encryption`** on our cluster if not already done - File-mount approach (see #TBD) works identically on all platforms and is forward-compatible with both Swarm secrets and K8s volume mounts ### Sources - [Docker Secrets Documentation](https://docs.docker.com/engine/swarm/secrets/) - [Kubernetes Secrets Good Practices](https://kubernetes.io/docs/concepts/security/secrets-good-practices/) - [Kubernetes Encrypting Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) - [K3s Secrets Encryption](https://docs.k3s.io/security/secrets-encryption) - [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/) - [ESO Development Paused](https://infisical.com/blog/external-secrets-operator-paused) - [OWASP Docker Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html) - [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker) - [NIST SP 800-190](https://csrc.nist.gov/publications/detail/sp/800-190/final)
claude added the featuresecurity labels 2026-04-15 00:31:01 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
Accessibility (a11y) improvements
 bug
Something isn't working correctly
 cleanup
Code cleanup, dead code removal, refactoring
 epic
Epic: groups related issues
 feature
New functionality
 pmf
Product-market fit: critical for first market offer
 route-diagram
Route diagram page and execution overlay
 security
Authentication, authorization, RBAC
 ui
Frontend / React UI
 ux
UX/usability improvements
No Label feature security
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cameleer/cameleer-server#130
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?