From 1427d58e00ef375a9cf7240e4595e4969d5c6673 Mon Sep 17 00:00:00 2001 From: hsiegeln <37154749+hsiegeln@users.noreply.github.com> Date: Mon, 27 Apr 2026 14:35:07 +0200 Subject: [PATCH] docs(plan): init-container JAR fetch + ArtifactStore abstraction 14-task TDD plan to replace bind-mount JAR delivery with init-container download from Cameleer over HTTP, sitting behind a new ArtifactStore abstraction. Lands withUsernsMode hardening (last open gap from #152) and gives storage a clean migration path to OCI (Zot) tracked separately in issue #158. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../2026-04-27-init-container-jar-fetch.md | 1616 +++++++++++++++++ 1 file changed, 1616 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-27-init-container-jar-fetch.md diff --git a/docs/superpowers/plans/2026-04-27-init-container-jar-fetch.md b/docs/superpowers/plans/2026-04-27-init-container-jar-fetch.md new file mode 100644 index 00000000..50201407 --- /dev/null +++ b/docs/superpowers/plans/2026-04-27-init-container-jar-fetch.md @@ -0,0 +1,1616 @@ +# Init-Container JAR Fetch + ArtifactStore Abstraction Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace host-bind-mounted JAR delivery with an init-container download from Cameleer over HTTP, sitting behind a new `ArtifactStore` abstraction. Closes the last open hardening gap from issue #152 (`withUsernsMode`) and gives the storage layer a clean migration path to OCI later (tracked separately). + +**Architecture:** +- New `ArtifactStore` interface in `cameleer-server-core` with one filesystem implementation today; the runtime never touches the filesystem path again. +- Cameleer serves JARs over HTTP via short-lived HMAC-signed URLs (no JWT/RBAC dependency on the loader). +- Tenant deploys now run a `cameleer-runtime-loader` container (busybox + 30-line `entrypoint.sh`) first, sharing a per-replica named Docker volume with the main container. Loader downloads `app.jar`, exits 0, main container starts with the volume mounted read-only at `/app/jars/`. +- `withUsernsMode("host:1000:65536")` lands once host file ownership is no longer in the picture (everything in the container's UID namespace inside the named volume). +- `agent.jar` stays baked into the runtime base image — out of scope for this change. + +**Tech Stack:** Java 17, Spring Boot 3.4.3, docker-java client, Postgres (Flyway), busybox-based loader image, HMAC-SHA256 URL signing. + +--- + +## File Structure + +### New files + +| Path | Responsibility | +|---|---| +| `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactStore.java` | Interface: put/get/exists/delete/downloadUrl over an opaque coordinate key | +| `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactCoordinates.java` | Value type: `(tenantId, appId, version)` — OCI-friendly so a future `OciArtifactStore` reuses it | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/FilesystemArtifactStore.java` | Filesystem impl preserving today's `{jarStoragePath}/{appId}/v{version}/app.jar` layout | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/web/ArtifactDownloadTokenSigner.java` | HMAC-SHA256 signer/verifier for download URLs; key = HMAC(jwtSecret, "cameleer-artifact-token-v1") | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/web/ArtifactDownloadController.java` | `GET /api/v1/artifacts/{appVersionId}?exp=...&sig=...` — token-validated stream | +| `cameleer-runtime-loader/Dockerfile` | busybox base, copies entrypoint.sh, runs as UID 1000 | +| `cameleer-runtime-loader/entrypoint.sh` | wget the signed URL into `/app/jars/app.jar`, verify size, exit | +| `cameleer-runtime-loader/README.md` | Build + push instructions (one-liner) | +| `cameleer-server-app/src/test/java/com/cameleer/server/app/storage/FilesystemArtifactStoreTest.java` | put/get round-trip, delete, listing, idempotency | +| `cameleer-server-app/src/test/java/com/cameleer/server/app/web/ArtifactDownloadTokenSignerTest.java` | sign + verify, expiry, tamper detection | +| `cameleer-server-app/src/test/java/com/cameleer/server/app/web/ArtifactDownloadControllerTest.java` | 200 with valid sig, 401 wrong sig, 410 expired, 404 missing | +| `cameleer-server-app/src/test/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestratorLoaderTest.java` | Loader-pattern launch contract: per-replica volume created, loader runs first, main mounts RO, userns_mode applied | + +### Modified files + +| Path | Why | +|---|---| +| `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ContainerRequest.java` | Drop `jarPath`/`jarVolumeName`/`jarVolumeMountPath`; add `appVersionId`, `artifactDownloadUrl`, `loaderImage` | +| `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/AppService.java` | `uploadJar` writes via `ArtifactStore.put`; `resolveJarPath` removed (callers now use store directly or the download URL flow) | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/retention/JarRetentionJob.java` | Deletes via `ArtifactStore.delete` | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DeploymentExecutor.java` | Generates signed URL, threads through `ContainerRequest`; preflight no longer touches filesystem | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestrator.java` | Per-replica named volume, run loader synchronously, start main with volume RO at `/app/jars/`, add `withUsernsMode` | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/config/RuntimeBeanConfig.java` | Wire `ArtifactStore` bean, drop `jarStoragePath` from `AppService` ctor | +| `cameleer-server-app/src/main/java/com/cameleer/server/app/security/SecurityConfig.java` | `permitAll` for `/api/v1/artifacts/**` (controller does HMAC validation) | +| `.claude/rules/docker-orchestration.md`, `.claude/rules/core-classes.md`, `.claude/rules/app-classes.md` | Update class/endpoint maps | +| `cameleer-server-app/src/main/resources/application.yml` | New properties: `cameleer.server.runtime.loaderimage`, `cameleer.server.runtime.artifacttokenttlseconds` | + +--- + +## Task 1: Add `ArtifactCoordinates` value type + +**Files:** +- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactCoordinates.java` +- Test: `cameleer-server-core/src/test/java/com/cameleer/server/core/storage/ArtifactCoordinatesTest.java` + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer.server.core.storage; + +import org.junit.jupiter.api.Test; +import java.util.UUID; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class ArtifactCoordinatesTest { + @Test + void rejectsNullsAndNonPositiveVersion() { + UUID appId = UUID.randomUUID(); + assertThatThrownBy(() -> new ArtifactCoordinates(null, appId, 1)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> new ArtifactCoordinates("default", null, 1)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> new ArtifactCoordinates("default", appId, 0)) + .isInstanceOf(IllegalArgumentException.class); + } + + @Test + void filesystemKeyMatchesLegacyLayout() { + UUID appId = UUID.fromString("11111111-1111-1111-1111-111111111111"); + ArtifactCoordinates c = new ArtifactCoordinates("default", appId, 7); + assertThat(c.filesystemKey()).isEqualTo("11111111-1111-1111-1111-111111111111/v7/app.jar"); + } + + @Test + void ociRefIsDeterministic() { + UUID appId = UUID.fromString("11111111-1111-1111-1111-111111111111"); + ArtifactCoordinates c = new ArtifactCoordinates("acme", appId, 7); + assertThat(c.ociRef()).isEqualTo("acme/11111111-1111-1111-1111-111111111111:v7"); + } +} +``` + +- [ ] **Step 2: Run the test** + +Run: `mvn -pl cameleer-server-core test -Dtest=ArtifactCoordinatesTest` +Expected: FAIL — class missing. + +- [ ] **Step 3: Implement `ArtifactCoordinates`** + +```java +package com.cameleer.server.core.storage; + +import java.util.Objects; +import java.util.UUID; + +/** + * Opaque coordinate that addresses one stored JAR. Designed so a future + * OCI-backed store can reuse the same coordinate without schema changes — + * see issue (Zot follow-up). + */ +public record ArtifactCoordinates(String tenantId, UUID appId, int version) { + public ArtifactCoordinates { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(appId, "appId"); + if (version <= 0) throw new IllegalArgumentException("version must be > 0, got " + version); + } + + /** Path-fragment used by FilesystemArtifactStore. Mirrors today's layout. */ + public String filesystemKey() { + return appId + "/v" + version + "/app.jar"; + } + + /** OCI reference used by a future registry-backed store. Stable so coordinates + * survive a backend swap. */ + public String ociRef() { + return tenantId + "/" + appId + ":v" + version; + } +} +``` + +- [ ] **Step 4: Re-run test — expect PASS** + +- [ ] **Step 5: Commit** + +```bash +git add cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactCoordinates.java \ + cameleer-server-core/src/test/java/com/cameleer/server/core/storage/ArtifactCoordinatesTest.java +git commit -m "feat(storage): add ArtifactCoordinates value type" +``` + +--- + +## Task 2: Add `ArtifactStore` interface + +**Files:** +- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactStore.java` + +No test file — pure interface; behavior is tested via `FilesystemArtifactStore` in Task 3. + +- [ ] **Step 1: Write the interface** + +```java +package com.cameleer.server.core.storage; + +import java.io.IOException; +import java.io.InputStream; + +/** + * Storage abstraction for deployable JAR artifacts. Implementations decide where + * the bytes live (filesystem today; OCI registry / S3 later — see Zot follow-up + * issue). Callers never construct filesystem paths themselves. + * + *

The interface is intentionally narrow: enough to upload, fetch, and delete. + * {@link #downloadUrl} returns the URL the loader container hits — for the + * filesystem store today this is a Cameleer-served signed URL; for an OCI store + * later it could be a pre-signed registry URL. + */ +public interface ArtifactStore { + + /** Persist {@code bytes} of {@code size} bytes under {@code coords}. Idempotent + * on identical content; overwrites on same coords with different content. */ + String put(ArtifactCoordinates coords, InputStream bytes, long size) throws IOException; + + /** Open the artifact for reading. Caller closes. */ + InputStream get(ArtifactCoordinates coords) throws IOException; + + boolean exists(ArtifactCoordinates coords); + + /** Remove the artifact and any backend-specific scaffolding (empty parent dirs, etc.). + * Silently no-ops if the artifact is already absent. */ + void delete(ArtifactCoordinates coords) throws IOException; + + /** Stable identifier for this storage location — written into + * {@code app_versions.jar_path} so existing rows still resolve. For the + * filesystem store this is the absolute path; for OCI it's the ref. */ + String locator(ArtifactCoordinates coords); +} +``` + +- [ ] **Step 2: Verify it compiles** + +Run: `mvn -pl cameleer-server-core compile` + +- [ ] **Step 3: Commit** + +```bash +git add cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ArtifactStore.java +git commit -m "feat(storage): add ArtifactStore interface" +``` + +--- + +## Task 3: Implement `FilesystemArtifactStore` + +**Files:** +- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/FilesystemArtifactStore.java` +- Test: `cameleer-server-app/src/test/java/com/cameleer/server/app/storage/FilesystemArtifactStoreTest.java` + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer.server.app.storage; + +import com.cameleer.server.core.storage.ArtifactCoordinates; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class FilesystemArtifactStoreTest { + + @Test + void roundTripsBytes(@TempDir Path tmp) throws Exception { + FilesystemArtifactStore store = new FilesystemArtifactStore(tmp.toString()); + ArtifactCoordinates c = new ArtifactCoordinates("default", UUID.randomUUID(), 1); + + byte[] payload = "hello-jar".getBytes(); + store.put(c, new ByteArrayInputStream(payload), payload.length); + + assertThat(store.exists(c)).isTrue(); + try (InputStream in = store.get(c)) { + assertThat(in.readAllBytes()).isEqualTo(payload); + } + } + + @Test + void deleteRemovesFileAndEmptyParents(@TempDir Path tmp) throws Exception { + FilesystemArtifactStore store = new FilesystemArtifactStore(tmp.toString()); + UUID appId = UUID.randomUUID(); + ArtifactCoordinates c = new ArtifactCoordinates("default", appId, 1); + store.put(c, new ByteArrayInputStream("x".getBytes()), 1); + + store.delete(c); + + assertThat(store.exists(c)).isFalse(); + // {tmp}/{appId}/v1/ should be gone too + assertThat(Files.exists(tmp.resolve(appId.toString()).resolve("v1"))).isFalse(); + } + + @Test + void deleteIsIdempotent(@TempDir Path tmp) throws Exception { + FilesystemArtifactStore store = new FilesystemArtifactStore(tmp.toString()); + ArtifactCoordinates c = new ArtifactCoordinates("default", UUID.randomUUID(), 1); + store.delete(c); // no exception + assertThat(store.exists(c)).isFalse(); + } + + @Test + void locatorMatchesLegacyAbsolutePath(@TempDir Path tmp) { + FilesystemArtifactStore store = new FilesystemArtifactStore(tmp.toString()); + UUID appId = UUID.fromString("11111111-1111-1111-1111-111111111111"); + ArtifactCoordinates c = new ArtifactCoordinates("default", appId, 7); + assertThat(store.locator(c)) + .isEqualTo(tmp.resolve("11111111-1111-1111-1111-111111111111/v7/app.jar").toString()); + } +} +``` + +- [ ] **Step 2: Run — expect FAIL (class missing)** + +Run: `mvn -pl cameleer-server-app test -Dtest=FilesystemArtifactStoreTest` + +- [ ] **Step 3: Implement** + +```java +package com.cameleer.server.app.storage; + +import com.cameleer.server.core.storage.ArtifactCoordinates; +import com.cameleer.server.core.storage.ArtifactStore; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; + +public class FilesystemArtifactStore implements ArtifactStore { + + private final Path root; + + public FilesystemArtifactStore(String root) { + this.root = Path.of(root); + } + + private Path pathOf(ArtifactCoordinates coords) { + return root.resolve(coords.filesystemKey()); + } + + @Override + public String put(ArtifactCoordinates coords, InputStream bytes, long size) throws IOException { + Path target = pathOf(coords); + Files.createDirectories(target.getParent()); + try (InputStream in = bytes) { + Files.copy(in, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING); + } + return target.toString(); + } + + @Override + public InputStream get(ArtifactCoordinates coords) throws IOException { + return Files.newInputStream(pathOf(coords)); + } + + @Override + public boolean exists(ArtifactCoordinates coords) { + return Files.exists(pathOf(coords)); + } + + @Override + public void delete(ArtifactCoordinates coords) throws IOException { + Path target = pathOf(coords); + Files.deleteIfExists(target); + // Sweep empty {appId}/v{n}/ then {appId}/ if now empty. + Path versionDir = target.getParent(); + if (versionDir != null && Files.isDirectory(versionDir) && isEmpty(versionDir)) { + Files.delete(versionDir); + Path appDir = versionDir.getParent(); + if (appDir != null && Files.isDirectory(appDir) && isEmpty(appDir)) { + Files.delete(appDir); + } + } + } + + @Override + public String locator(ArtifactCoordinates coords) { + return pathOf(coords).toString(); + } + + private static boolean isEmpty(Path dir) throws IOException { + try (var entries = Files.list(dir)) { + return entries.findFirst().isEmpty(); + } + } +} +``` + +- [ ] **Step 4: Re-run test — expect PASS** + +- [ ] **Step 5: Commit** + +```bash +git add cameleer-server-app/src/main/java/com/cameleer/server/app/storage/FilesystemArtifactStore.java \ + cameleer-server-app/src/test/java/com/cameleer/server/app/storage/FilesystemArtifactStoreTest.java +git commit -m "feat(storage): add FilesystemArtifactStore (one impl of ArtifactStore)" +``` + +--- + +## Task 4: Migrate `AppService.uploadJar` to use `ArtifactStore`; remove `resolveJarPath` + +**Files:** +- Modify: `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/AppService.java` +- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/config/RuntimeBeanConfig.java` +- Test: `cameleer-server-core/src/test/java/com/cameleer/server/core/runtime/AppServiceUploadTest.java` (new) + +The store is in `cameleer-server-app` but the interface is in `cameleer-server-core`, so `AppService` only depends on the interface. + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer.server.core.runtime; + +import com.cameleer.server.core.storage.ArtifactCoordinates; +import com.cameleer.server.core.storage.ArtifactStore; +import org.junit.jupiter.api.Test; + +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +class AppServiceUploadTest { + + @Test + void uploadJarStoresViaArtifactStoreAndRecordsLocator() throws Exception { + AppRepository appRepo = mock(AppRepository.class); + AppVersionRepository versionRepo = mock(AppVersionRepository.class); + ArtifactStore store = mock(ArtifactStore.class); + + UUID appId = UUID.randomUUID(); + UUID versionId = UUID.randomUUID(); + when(appRepo.findById(appId)).thenReturn(Optional.of( + new App(appId, UUID.randomUUID(), "demo", "Demo", null))); + when(versionRepo.findMaxVersion(appId)).thenReturn(2); + when(store.put(any(), any(InputStream.class), anyLong())).thenReturn("/data/jars/" + appId + "/v3/app.jar"); + when(versionRepo.create(eq(appId), eq(3), anyString(), anyString(), eq("a.jar"), eq(5L))) + .thenReturn(versionId); + when(versionRepo.findById(versionId)).thenReturn(Optional.of( + new AppVersion(versionId, appId, 3, "/data/jars/" + appId + "/v3/app.jar", + "deadbeef", "a.jar", 5L, null, null, null))); + + AppService svc = new AppService(appRepo, versionRepo, store, "default"); + + byte[] body = "hello".getBytes(); + AppVersion result = svc.uploadJar(appId, "a.jar", new ByteArrayInputStream(body), body.length); + + verify(store).put(eq(new ArtifactCoordinates("default", appId, 3)), + any(InputStream.class), eq(5L)); + assertThat(result.version()).isEqualTo(3); + } +} +``` + +- [ ] **Step 2: Run — expect FAIL (constructor mismatch)** + +Run: `mvn -pl cameleer-server-core test -Dtest=AppServiceUploadTest` + +- [ ] **Step 3: Refactor `AppService`** + +Replace the file contents: + +```java +package com.cameleer.server.core.runtime; + +import com.cameleer.server.core.storage.ArtifactCoordinates; +import com.cameleer.server.core.storage.ArtifactStore; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.security.MessageDigest; +import java.util.HexFormat; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.regex.Pattern; + +public class AppService { + private static final Logger log = LoggerFactory.getLogger(AppService.class); + + private static final Pattern SLUG_PATTERN = Pattern.compile("^[a-z0-9][a-z0-9-]{0,63}$"); + + private final AppRepository appRepo; + private final AppVersionRepository versionRepo; + private final ArtifactStore artifactStore; + private final String tenantId; + private final CreateGuard createGuard; + + public AppService(AppRepository appRepo, AppVersionRepository versionRepo, + ArtifactStore artifactStore, String tenantId) { + this(appRepo, versionRepo, artifactStore, tenantId, CreateGuard.NOOP); + } + + public AppService(AppRepository appRepo, AppVersionRepository versionRepo, + ArtifactStore artifactStore, String tenantId, CreateGuard createGuard) { + this.appRepo = appRepo; + this.versionRepo = versionRepo; + this.artifactStore = artifactStore; + this.tenantId = tenantId; + this.createGuard = createGuard; + } + + public List listAll() { return appRepo.findAll(); } + public List listByEnvironment(UUID environmentId) { return appRepo.findByEnvironmentId(environmentId); } + public App getById(UUID id) { return appRepo.findById(id).orElseThrow(() -> new IllegalArgumentException("App not found: " + id)); } + public App getBySlug(String slug) { return appRepo.findBySlug(slug).orElseThrow(() -> new IllegalArgumentException("App not found: " + slug)); } + public App getByEnvironmentAndSlug(UUID environmentId, String slug) { + return appRepo.findByEnvironmentIdAndSlug(environmentId, slug) + .orElseThrow(() -> new IllegalArgumentException("App not found in environment: " + slug)); + } + public List listVersions(UUID appId) { return versionRepo.findByAppId(appId); } + + public AppVersion getVersion(UUID versionId) { + return versionRepo.findById(versionId) + .orElseThrow(() -> new IllegalArgumentException("AppVersion not found: " + versionId)); + } + + public ArtifactCoordinates coordinatesFor(AppVersion version) { + return new ArtifactCoordinates(tenantId, version.appId(), version.version()); + } + + public void updateContainerConfig(UUID id, Map containerConfig) { + getById(id); + appRepo.updateContainerConfig(id, containerConfig); + } + + public UUID createApp(UUID environmentId, String slug, String displayName) { + if (slug == null || !SLUG_PATTERN.matcher(slug).matches()) { + throw new IllegalArgumentException( + "Invalid app slug: must match ^[a-z0-9][a-z0-9-]{0,63}$"); + } + createGuard.check(appRepo.count()); + if (appRepo.findByEnvironmentIdAndSlug(environmentId, slug).isPresent()) { + throw new IllegalArgumentException("App with slug '" + slug + "' already exists in this environment"); + } + return appRepo.create(environmentId, slug, displayName); + } + + public AppVersion uploadJar(UUID appId, String filename, InputStream jarData, long size) throws IOException { + getById(appId); + int nextVersion = versionRepo.findMaxVersion(appId) + 1; + ArtifactCoordinates coords = new ArtifactCoordinates(tenantId, appId, nextVersion); + + // Buffer once for hashing + storage so we can also detect runtime type without re-reading. + ByteArrayOutputStream buf = new ByteArrayOutputStream(Math.max(8192, (int) Math.min(size, 64 * 1024 * 1024))); + MessageDigest digest; + try { digest = MessageDigest.getInstance("SHA-256"); } + catch (Exception e) { throw new RuntimeException(e); } + try (InputStream in = jarData) { + byte[] tmp = new byte[8192]; + int n; + while ((n = in.read(tmp)) != -1) { + buf.write(tmp, 0, n); + digest.update(tmp, 0, n); + } + } + byte[] bytes = buf.toByteArray(); + + String locator = artifactStore.put(coords, new java.io.ByteArrayInputStream(bytes), bytes.length); + String checksum = HexFormat.of().formatHex(digest.digest()); + + UUID versionId = versionRepo.create(appId, nextVersion, locator, checksum, filename, size); + + RuntimeDetector.DetectionResult detection = RuntimeDetector.detect(new java.io.ByteArrayInputStream(bytes)); + if (detection.runtimeType() != null) { + versionRepo.updateDetectedRuntime(versionId, detection.runtimeType().toConfigValue(), detection.mainClass()); + log.info("Uploaded JAR for app {}: version={}, size={}, sha256={}, detected={}", + appId, nextVersion, size, checksum, detection.runtimeType().toConfigValue()); + } else { + log.info("Uploaded JAR for app {}: version={}, size={}, sha256={}, detected=unknown", + appId, nextVersion, size, checksum); + } + + return versionRepo.findById(versionId).orElseThrow(); + } + + public void deleteApp(UUID id) { + appRepo.delete(id); + } +} +``` + +> Note: `RuntimeDetector.detect(InputStream)` may not exist today — verify and add an `InputStream` overload if needed (it currently takes a `Path`). If adding the overload is non-trivial, write the bytes to a temp file in this method and pass the path; document the temp-file lifetime. + +- [ ] **Step 4: Update `RuntimeBeanConfig`** + +```java +// In cameleer-server-app/src/main/java/com/cameleer/server/app/config/RuntimeBeanConfig.java +// Replace the AppService bean: + +@Bean +public ArtifactStore artifactStore(@Value("${cameleer.server.runtime.jarstoragepath:/data/jars}") String jarStoragePath) { + return new FilesystemArtifactStore(jarStoragePath); +} + +@Bean +public AppService appService(AppRepository appRepo, + AppVersionRepository versionRepo, + ArtifactStore artifactStore, + @Value("${cameleer.server.tenant.id:default}") String tenantId, + LicenseEnforcer licenseEnforcer) { + return new AppService(appRepo, versionRepo, artifactStore, tenantId, + current -> licenseEnforcer.assertWithinCap("max_apps", current, 1)); +} +``` + +- [ ] **Step 5: Run all core tests — expect PASS** + +```bash +mvn -pl cameleer-server-core test +mvn -pl cameleer-server-app test -Dtest='*AppService*,*RuntimeBeanConfig*' +``` + +Fix any callers of removed `resolveJarPath` — should be only `DeploymentExecutor` (handled in Task 7) and the orphaned-app cleanup. + +- [ ] **Step 6: Commit** + +```bash +git add -A +git commit -m "refactor(core): AppService writes via ArtifactStore; remove resolveJarPath" +``` + +--- + +## Task 5: Migrate `JarRetentionJob` to `ArtifactStore` + +**Files:** +- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/retention/JarRetentionJob.java` +- Test: `cameleer-server-app/src/test/java/com/cameleer/server/app/retention/JarRetentionJobTest.java` (extend existing or new) + +- [ ] **Step 1: Write the failing test** + +```java +@Test +void cleanupDeletesViaArtifactStore() { + EnvironmentService envSvc = mock(EnvironmentService.class); + AppService appSvc = mock(AppService.class); + AppVersionRepository versionRepo = mock(AppVersionRepository.class); + DeploymentRepository deployRepo = mock(DeploymentRepository.class); + ArtifactStore store = mock(ArtifactStore.class); + + UUID envId = UUID.randomUUID(), appId = UUID.randomUUID(); + Environment env = new Environment(envId, "dev", "Dev", false, true, + null, 2, "slate", Instant.now(), 7, 7, 7); + App app = new App(appId, envId, "demo", "Demo", null); + AppVersion v1 = new AppVersion(UUID.randomUUID(), appId, 1, "ignored", "h", "a.jar", 1L, null, null, null); + AppVersion v2 = new AppVersion(UUID.randomUUID(), appId, 2, "ignored", "h", "a.jar", 1L, null, null, null); + AppVersion v3 = new AppVersion(UUID.randomUUID(), appId, 3, "ignored", "h", "a.jar", 1L, null, null, null); + + when(envSvc.listAll()).thenReturn(List.of(env)); + when(appSvc.listByEnvironment(envId)).thenReturn(List.of(app)); + when(versionRepo.findByAppId(appId)).thenReturn(List.of(v3, v2, v1)); // DESC + when(deployRepo.findByAppId(appId)).thenReturn(List.of()); + when(appSvc.coordinatesFor(v1)).thenReturn(new ArtifactCoordinates("default", appId, 1)); + + new JarRetentionJob(envSvc, appSvc, versionRepo, deployRepo, store).cleanupOldVersions(); + + verify(store).delete(new ArtifactCoordinates("default", appId, 1)); + verify(versionRepo).delete(v1.id()); + verifyNoMoreInteractions(store); +} +``` + +- [ ] **Step 2: Run — expect FAIL (constructor mismatch)** + +- [ ] **Step 3: Refactor** + +Replace `deleteJarFile` with a delegation to `store.delete(appSvc.coordinatesFor(version))`. Add `ArtifactStore` to the constructor. + +```java +private final ArtifactStore store; + +public JarRetentionJob(EnvironmentService environmentService, + AppService appService, + AppVersionRepository versionRepo, + DeploymentRepository deploymentRepo, + ArtifactStore store) { + this.environmentService = environmentService; + this.appService = appService; + this.versionRepo = versionRepo; + this.deploymentRepo = deploymentRepo; + this.store = store; +} + +private int cleanupApp(App app, int retentionCount) { + List versions = versionRepo.findByAppId(app.id()); + if (versions.size() <= retentionCount) return 0; + + Set deployedVersionIds = deploymentRepo.findByAppId(app.id()).stream() + .map(Deployment::appVersionId).collect(Collectors.toSet()); + + int deleted = 0; + for (int i = retentionCount; i < versions.size(); i++) { + AppVersion version = versions.get(i); + if (deployedVersionIds.contains(version.id())) continue; + try { + store.delete(appService.coordinatesFor(version)); + } catch (IOException e) { + log.warn("Failed to delete artifact for version {}: {}", version.id(), e.getMessage()); + } + versionRepo.delete(version.id()); + deleted++; + log.info("Deleted version v{} of app {} ({})", version.version(), app.slug(), version.id()); + } + return deleted; +} +``` + +Delete the old `deleteJarFile` method and unused `Files`/`Path` imports. + +- [ ] **Step 4: Re-run tests — expect PASS** + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "refactor(retention): JarRetentionJob deletes via ArtifactStore" +``` + +--- + +## Task 6: HMAC `ArtifactDownloadTokenSigner` + +**Files:** +- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/web/ArtifactDownloadTokenSigner.java` +- Test: `cameleer-server-app/src/test/java/com/cameleer/server/app/web/ArtifactDownloadTokenSignerTest.java` + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer.server.app.web; + +import org.junit.jupiter.api.Test; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class ArtifactDownloadTokenSignerTest { + + private final String secret = "test-secret-do-not-use"; + private final Instant now = Instant.parse("2026-04-27T10:00:00Z"); + private final Clock clock = Clock.fixed(now, ZoneOffset.UTC); + + @Test + void signedTokenVerifiesWithinTtl() { + var signer = new ArtifactDownloadTokenSigner(secret, clock); + UUID id = UUID.randomUUID(); + var token = signer.sign(id, Duration.ofMinutes(5)); + assertThat(signer.verify(id, token.exp(), token.sig())).isTrue(); + } + + @Test + void rejectsTamperedSignature() { + var signer = new ArtifactDownloadTokenSigner(secret, clock); + UUID id = UUID.randomUUID(); + var token = signer.sign(id, Duration.ofMinutes(5)); + assertThat(signer.verify(id, token.exp(), token.sig() + "x")).isFalse(); + } + + @Test + void rejectsMismatchedAppVersionId() { + var signer = new ArtifactDownloadTokenSigner(secret, clock); + UUID id = UUID.randomUUID(); + var token = signer.sign(id, Duration.ofMinutes(5)); + assertThat(signer.verify(UUID.randomUUID(), token.exp(), token.sig())).isFalse(); + } + + @Test + void rejectsExpired() { + var signer = new ArtifactDownloadTokenSigner(secret, clock); + UUID id = UUID.randomUUID(); + long pastExp = now.minusSeconds(1).getEpochSecond(); + String sig = signer.signRaw(id, pastExp); + assertThat(signer.verify(id, pastExp, sig)).isFalse(); + } +} +``` + +- [ ] **Step 2: Run — expect FAIL** + +- [ ] **Step 3: Implement** + +```java +package com.cameleer.server.app.web; + +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; +import java.nio.charset.StandardCharsets; +import java.time.Clock; +import java.time.Duration; +import java.util.Base64; +import java.util.UUID; + +/** + * HMAC-SHA256 signed URL tokens for artifact downloads. Key derivation is + * deterministic from the JWT signing secret (HMAC-SHA256(jwtSecret, + * "cameleer-artifact-token-v1")) so server restarts don't invalidate fresh + * tokens. The loader container does NOT carry the JWT or the bootstrap token — + * it only carries the signed URL, which is scoped to one appVersionId and one + * short TTL. + */ +public class ArtifactDownloadTokenSigner { + + private static final String DERIVATION = "cameleer-artifact-token-v1"; + + public record SignedToken(long exp, String sig) {} + + private final byte[] key; + private final Clock clock; + + public ArtifactDownloadTokenSigner(String jwtSecret, Clock clock) { + this.key = deriveKey(jwtSecret); + this.clock = clock; + } + + private static byte[] deriveKey(String jwtSecret) { + try { + Mac mac = Mac.getInstance("HmacSHA256"); + mac.init(new SecretKeySpec(jwtSecret.getBytes(StandardCharsets.UTF_8), "HmacSHA256")); + return mac.doFinal(DERIVATION.getBytes(StandardCharsets.UTF_8)); + } catch (Exception e) { + throw new IllegalStateException("HMAC init failed", e); + } + } + + public SignedToken sign(UUID appVersionId, Duration ttl) { + long exp = clock.instant().plus(ttl).getEpochSecond(); + return new SignedToken(exp, signRaw(appVersionId, exp)); + } + + public String signRaw(UUID appVersionId, long exp) { + try { + Mac mac = Mac.getInstance("HmacSHA256"); + mac.init(new SecretKeySpec(key, "HmacSHA256")); + String payload = appVersionId + ":" + exp; + byte[] tag = mac.doFinal(payload.getBytes(StandardCharsets.UTF_8)); + return Base64.getUrlEncoder().withoutPadding().encodeToString(tag); + } catch (Exception e) { + throw new IllegalStateException("sign failed", e); + } + } + + public boolean verify(UUID appVersionId, long exp, String sig) { + if (sig == null || sig.isBlank()) return false; + if (clock.instant().getEpochSecond() > exp) return false; + String expected = signRaw(appVersionId, exp); + // constant-time compare + return java.security.MessageDigest.isEqual( + expected.getBytes(StandardCharsets.UTF_8), + sig.getBytes(StandardCharsets.UTF_8)); + } +} +``` + +- [ ] **Step 4: Run — expect PASS** + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(web): add HMAC token signer for artifact downloads" +``` + +--- + +## Task 7: `ArtifactDownloadController` + +**Files:** +- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/web/ArtifactDownloadController.java` +- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/security/SecurityConfig.java` +- Test: `cameleer-server-app/src/test/java/com/cameleer/server/app/web/ArtifactDownloadControllerTest.java` + +- [ ] **Step 1: Write the failing test (`@WebMvcTest`)** + +```java +@WebMvcTest(ArtifactDownloadController.class) +class ArtifactDownloadControllerTest { + + @Autowired MockMvc mvc; + @MockBean AppService appService; + @MockBean ArtifactStore artifactStore; + @MockBean ArtifactDownloadTokenSigner signer; + + @Test + void streamsBytesWhenSignatureValid() throws Exception { + UUID id = UUID.randomUUID(); + when(signer.verify(eq(id), eq(123L), eq("good"))).thenReturn(true); + AppVersion v = new AppVersion(id, UUID.randomUUID(), 1, "loc", "h", "a.jar", 5L, null, null, null); + when(appService.getVersion(id)).thenReturn(v); + when(appService.coordinatesFor(v)).thenReturn(new ArtifactCoordinates("default", v.appId(), 1)); + when(artifactStore.get(any())).thenReturn(new ByteArrayInputStream("hello".getBytes())); + + mvc.perform(get("/api/v1/artifacts/{id}", id).param("exp", "123").param("sig", "good")) + .andExpect(status().isOk()) + .andExpect(header().string("Content-Type", "application/java-archive")) + .andExpect(content().bytes("hello".getBytes())); + } + + @Test + void rejectsBadSignature() throws Exception { + UUID id = UUID.randomUUID(); + when(signer.verify(any(), anyLong(), any())).thenReturn(false); + mvc.perform(get("/api/v1/artifacts/{id}", id).param("exp", "123").param("sig", "bad")) + .andExpect(status().isUnauthorized()); + } + + @Test + void returns404WhenArtifactMissing() throws Exception { + UUID id = UUID.randomUUID(); + when(signer.verify(any(), anyLong(), any())).thenReturn(true); + when(appService.getVersion(id)).thenThrow(new IllegalArgumentException("AppVersion not found")); + mvc.perform(get("/api/v1/artifacts/{id}", id).param("exp", "123").param("sig", "ok")) + .andExpect(status().isNotFound()); + } +} +``` + +- [ ] **Step 2: Run — expect FAIL** + +- [ ] **Step 3: Implement** + +```java +package com.cameleer.server.app.web; + +import com.cameleer.server.core.runtime.AppService; +import com.cameleer.server.core.runtime.AppVersion; +import com.cameleer.server.core.storage.ArtifactStore; +import org.springframework.core.io.InputStreamResource; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +import java.io.IOException; +import java.io.InputStream; +import java.util.UUID; + +/** + * Token-validated artifact download endpoint hit by the cameleer-runtime-loader + * init container. Auth is the HMAC sig + exp on the URL — NOT JWT/bootstrap + * token. permitAll'd in SecurityConfig because all auth is in the controller. + */ +@RestController +@RequestMapping("/api/v1/artifacts") +public class ArtifactDownloadController { + + private final AppService appService; + private final ArtifactStore artifactStore; + private final ArtifactDownloadTokenSigner signer; + + public ArtifactDownloadController(AppService appService, + ArtifactStore artifactStore, + ArtifactDownloadTokenSigner signer) { + this.appService = appService; + this.artifactStore = artifactStore; + this.signer = signer; + } + + @GetMapping("/{appVersionId}") + public ResponseEntity download(@PathVariable UUID appVersionId, + @RequestParam("exp") long exp, + @RequestParam("sig") String sig) throws IOException { + if (!signer.verify(appVersionId, exp, sig)) { + return ResponseEntity.status(401).build(); + } + AppVersion version; + try { + version = appService.getVersion(appVersionId); + } catch (IllegalArgumentException e) { + return ResponseEntity.notFound().build(); + } + InputStream in = artifactStore.get(appService.coordinatesFor(version)); + return ResponseEntity.ok() + .contentType(MediaType.parseMediaType("application/java-archive")) + .contentLength(version.jarSizeBytes()) + .body(new InputStreamResource(in)); + } +} +``` + +- [ ] **Step 4: Add `permitAll` to `SecurityConfig`** + +In `cameleer-server-app/src/main/java/com/cameleer/server/app/security/SecurityConfig.java`, add `/api/v1/artifacts/**` to the permitAll matchers (alongside `/api/v1/auth/**`, `/api/v1/health`, etc.). + +- [ ] **Step 5: Wire `ArtifactDownloadTokenSigner` bean** (in `SecurityBeanConfig` or `RuntimeBeanConfig`) + +```java +@Bean +public ArtifactDownloadTokenSigner artifactDownloadTokenSigner( + @Value("${cameleer.server.security.jwtsecret}") String jwtSecret) { + return new ArtifactDownloadTokenSigner(jwtSecret, java.time.Clock.systemUTC()); +} +``` + +- [ ] **Step 6: Run all tests — expect PASS** + +- [ ] **Step 7: Commit** + +```bash +git add -A +git commit -m "feat(web): add ArtifactDownloadController with HMAC URL auth" +``` + +--- + +## Task 8: Build the `cameleer-runtime-loader` image + +**Files:** +- Create: `cameleer-runtime-loader/Dockerfile` +- Create: `cameleer-runtime-loader/entrypoint.sh` +- Create: `cameleer-runtime-loader/README.md` + +- [ ] **Step 1: Write `entrypoint.sh`** + +```sh +#!/bin/sh +# cameleer-runtime-loader: fetches one JAR from a signed URL into the shared +# /app/jars/ volume, verifies size, exits. Runs in the same hardened sandbox as +# the main container (cap_drop ALL, read-only rootfs, etc.) — only /app/jars/ +# is writeable. +set -eu + +: "${ARTIFACT_URL:?ARTIFACT_URL is required}" +: "${ARTIFACT_EXPECTED_SIZE:?ARTIFACT_EXPECTED_SIZE is required}" + +OUT=/app/jars/app.jar +mkdir -p /app/jars + +echo "loader: fetching artifact (expected $ARTIFACT_EXPECTED_SIZE bytes)" +# -q quiet, -O output, --tries=3 retry transient network blips, +# --timeout=30 cap stalls. wget exits non-zero on HTTP >=400. +wget -q --tries=3 --timeout=30 -O "$OUT" "$ARTIFACT_URL" + +actual=$(wc -c < "$OUT") +if [ "$actual" -ne "$ARTIFACT_EXPECTED_SIZE" ]; then + echo "loader: size mismatch — expected $ARTIFACT_EXPECTED_SIZE, got $actual" >&2 + exit 2 +fi + +echo "loader: artifact written to $OUT ($actual bytes)" +``` + +- [ ] **Step 2: Write `Dockerfile`** + +```dockerfile +# Tiny init-container image. No app code, no shell-injection surface — script +# only sees env vars set by the orchestrator. +FROM busybox:1.37-musl + +# Run as non-root (UID 1000 inside the container; with userns_mode this is +# remapped to host UID ~101000 — fully unprivileged on the host). +RUN adduser -D -u 1000 loader + +COPY entrypoint.sh /usr/local/bin/loader +RUN chmod +x /usr/local/bin/loader + +USER loader +WORKDIR /app +ENTRYPOINT ["/usr/local/bin/loader"] +``` + +- [ ] **Step 3: Build + smoke-test locally** + +```bash +docker build -t gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest cameleer-runtime-loader/ +docker run --rm -v test-vol:/app/jars \ + -e ARTIFACT_URL=https://example.invalid/missing \ + -e ARTIFACT_EXPECTED_SIZE=0 \ + gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest || echo "expected fail OK" +``` + +- [ ] **Step 4: Push to gitea registry** + +```bash +docker push gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest +``` + +- [ ] **Step 5: Write `README.md`** + +```markdown +# cameleer-runtime-loader + +Init container that fetches the deployable JAR into a shared volume before the +main runtime container starts. Pairs with `DockerRuntimeOrchestrator` / +(future) K8s init-container deploys. + +## Build + + docker build -t gitea.siegeln.net/cameleer/cameleer-runtime-loader: . + docker push gitea.siegeln.net/cameleer/cameleer-runtime-loader: + +## Contract + +- Env: `ARTIFACT_URL` (signed download URL), `ARTIFACT_EXPECTED_SIZE` (bytes). +- Volume: writes `/app/jars/app.jar`. +- Exit 0 on success; non-zero on fetch/size failure. +- Runs as UID 1000 (loader user), drops all caps, read-only rootfs except `/app/jars`. + +See `docs/superpowers/plans/2026-04-27-init-container-jar-fetch.md`. +``` + +- [ ] **Step 6: Commit** + +```bash +git add cameleer-runtime-loader/ +git commit -m "feat(loader): add cameleer-runtime-loader image (busybox + entrypoint)" +``` + +--- + +## Task 9: Refactor `ContainerRequest` + +**Files:** +- Modify: `cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ContainerRequest.java` + +No test — record refactor; behavior tested in Task 10. + +- [ ] **Step 1: Replace the record** + +```java +package com.cameleer.server.core.runtime; + +import java.util.List; +import java.util.Map; +import java.util.UUID; + +public record ContainerRequest( + String containerName, + String baseImage, + UUID appVersionId, + String artifactDownloadUrl, + long artifactExpectedSize, + String loaderImage, + String network, + List additionalNetworks, + Map envVars, + Map labels, + long memoryLimitBytes, + Long memoryReserveBytes, + int cpuShares, + Long cpuQuota, + List exposedPorts, + int healthCheckPort, + String restartPolicyName, + int restartPolicyMaxRetries, + String runtimeType, + String customArgs, + String mainClass +) {} +``` + +- [ ] **Step 2: Compile (will break orchestrator + executor — fixed in next tasks)** + +Run: `mvn -pl cameleer-server-core compile` + +- [ ] **Step 3: Commit (broken intermediate — okay because next two tasks restore green)** + +```bash +git add cameleer-server-core/src/main/java/com/cameleer/server/core/runtime/ContainerRequest.java +git commit -m "refactor(core): replace ContainerRequest jar fields with appVersionId+downloadUrl" +``` + +> If you prefer never to commit a broken state, fold Tasks 9, 10, 11 into one commit. + +--- + +## Task 10: `DockerRuntimeOrchestrator` loader-pattern launch + +**Files:** +- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestrator.java` +- Test: `cameleer-server-app/src/test/java/com/cameleer/server/app/runtime/DockerRuntimeOrchestratorLoaderTest.java` + +This is the central change. The orchestrator now: (1) creates a per-replica named volume, (2) creates + starts + waits for the loader container, (3) checks loader exit code, (4) creates the main container with the volume mounted RO at `/app/jars/`, (5) cleans up the loader container on success or failure. + +- [ ] **Step 1: Write the failing test** + +```java +@Test +void startContainerCreatesVolumeRunsLoaderThenStartsMain() throws Exception { + DockerClient docker = mockDockerClientWithRuntimes(Map.of()); + // capture every createContainerCmd invocation in order + CreateContainerCmd loaderCreate = mock(CreateContainerCmd.class, Answers.RETURNS_SELF); + CreateContainerCmd mainCreate = mock(CreateContainerCmd.class, Answers.RETURNS_SELF); + when(docker.createContainerCmd(eq("gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest"))).thenReturn(loaderCreate); + when(docker.createContainerCmd(eq("base-image:latest"))).thenReturn(mainCreate); + + CreateContainerResponse loaderResp = mock(CreateContainerResponse.class); + when(loaderResp.getId()).thenReturn("loader-id"); + when(loaderCreate.exec()).thenReturn(loaderResp); + CreateContainerResponse mainResp = mock(CreateContainerResponse.class); + when(mainResp.getId()).thenReturn("main-id"); + when(mainCreate.exec()).thenReturn(mainResp); + + when(docker.startContainerCmd(any())).thenReturn(mock(StartContainerCmd.class)); + when(docker.removeContainerCmd(any())).thenReturn(mock(RemoveContainerCmd.class, Answers.RETURNS_SELF)); + // wait returns exit 0 + var waitCmd = mock(WaitContainerCmd.class); + when(docker.waitContainerCmd("loader-id")).thenReturn(waitCmd); + var waitCallback = mock(WaitContainerResultCallback.class); + when(waitCmd.exec(any())).thenReturn(waitCallback); + when(waitCallback.awaitStatusCode(anyLong(), any())).thenReturn(0); + // volume create + when(docker.createVolumeCmd()).thenReturn(mock(CreateVolumeCmd.class, Answers.RETURNS_SELF)); + + var orchestrator = new DockerRuntimeOrchestrator(docker, ""); + orchestrator.startContainer(sampleRequest("base-image:latest", "https://srv/api/v1/artifacts/uuid?exp=1&sig=x", 12345L)); + + // loader was created with volume RW at /app/jars and the right env + ArgumentCaptor loaderHc = ArgumentCaptor.forClass(HostConfig.class); + verify(loaderCreate).withHostConfig(loaderHc.capture()); + assertThat(loaderHc.getValue().getBinds()).anyMatch(b -> + b.getVolume().getPath().equals("/app/jars") && b.getAccessMode() == AccessMode.rw); + + // main was created with volume RO at /app/jars + ArgumentCaptor mainHc = ArgumentCaptor.forClass(HostConfig.class); + verify(mainCreate).withHostConfig(mainHc.capture()); + assertThat(mainHc.getValue().getBinds()).anyMatch(b -> + b.getVolume().getPath().equals("/app/jars") && b.getAccessMode() == AccessMode.ro); + + // loader started, waited for, removed; then main started + InOrder order = inOrder(docker); + order.verify(docker).startContainerCmd("loader-id"); + order.verify(docker).waitContainerCmd("loader-id"); + order.verify(docker).removeContainerCmd("loader-id"); + order.verify(docker).startContainerCmd("main-id"); +} + +@Test +void startContainerAbortsAndCleansUpWhenLoaderFails() { + // wire wait to return exit 2; expect RuntimeException; expect volume + loader removed + // and main container NEVER created. + // ... (use the same scaffolding; assert `docker.createContainerCmd("base-image:latest")` is never called) +} + +@Test +void startContainerAppliesUsernsModeToBothContainers() { + // assert hostConfig.getUsernsMode() == "host:1000:65536" on both loader and main HostConfig captures. +} +``` + +- [ ] **Step 2: Run — expect FAIL** + +- [ ] **Step 3: Implement** + +Replace the `startContainer` method body with the loader-then-main flow. Key code: + +```java +private static final String LOADER_VOLUME_MOUNT = "/app/jars"; +private static final String USERNS_MODE = "host:1000:65536"; + +@Override +public String startContainer(ContainerRequest request) { + String volumeName = "cameleer-jars-" + request.containerName(); + dockerClient.createVolumeCmd().withName(volumeName).exec(); + + // 1) Loader container — RW on the shared volume. + String loaderId = createAndStartLoader(request, volumeName); + int exitCode; + try { + exitCode = dockerClient.waitContainerCmd(loaderId) + .exec(new com.github.dockerjava.core.command.WaitContainerResultCallback()) + .awaitStatusCode(120, java.util.concurrent.TimeUnit.SECONDS); + } catch (Exception e) { + cleanup(loaderId, volumeName); + throw new RuntimeException("Loader wait failed for " + request.containerName(), e); + } finally { + try { dockerClient.removeContainerCmd(loaderId).withForce(true).exec(); } + catch (Exception e) { log.warn("Failed to remove loader {}: {}", loaderId, e.getMessage()); } + } + if (exitCode != 0) { + cleanupVolume(volumeName); + throw new RuntimeException("Loader exited " + exitCode + " for " + request.containerName()); + } + + // 2) Main container — RO on the shared volume. + return createAndStartMain(request, volumeName); +} + +private String createAndStartLoader(ContainerRequest request, String volumeName) { + HostConfig hc = baseHardenedHostConfig() + .withBinds(new Bind(volumeName, new Volume(LOADER_VOLUME_MOUNT), AccessMode.rw)) + .withUsernsMode(USERNS_MODE) + .withNetworkMode(request.network()); + if (!dockerRuntime.isBlank()) hc.withRuntime(dockerRuntime); + + var create = dockerClient.createContainerCmd(request.loaderImage()) + .withName(request.containerName() + "-loader") + .withEnv(List.of( + "ARTIFACT_URL=" + request.artifactDownloadUrl(), + "ARTIFACT_EXPECTED_SIZE=" + request.artifactExpectedSize())) + .withHostConfig(hc); + String id = create.exec().getId(); + dockerClient.startContainerCmd(id).exec(); + return id; +} + +private String createAndStartMain(ContainerRequest request, String volumeName) { + List envList = request.envVars().entrySet().stream() + .map(e -> e.getKey() + "=" + e.getValue()).toList(); + + HostConfig hc = baseHardenedHostConfig() + .withMemory(request.memoryLimitBytes()) + .withMemorySwap(request.memoryLimitBytes()) + .withCpuShares(request.cpuShares()) + .withNetworkMode(request.network()) + .withRestartPolicy(RestartPolicy.onFailureRestart(request.restartPolicyMaxRetries())) + .withUsernsMode(USERNS_MODE) + .withBinds(new Bind(volumeName, new Volume(LOADER_VOLUME_MOUNT), AccessMode.ro)); + if (!dockerRuntime.isBlank()) hc.withRuntime(dockerRuntime); + if (request.memoryReserveBytes() != null) hc.withMemoryReservation(request.memoryReserveBytes()); + if (request.cpuQuota() != null) hc.withCpuQuota(request.cpuQuota()); + + String appJarPath = LOADER_VOLUME_MOUNT + "/app.jar"; + String customArgs = request.customArgs() != null && !request.customArgs().isBlank() + ? " " + request.customArgs() : ""; + String entrypoint = switch (request.runtimeType()) { + case "plain-java" -> "exec java -javaagent:/app/agent.jar" + customArgs + + " -cp " + appJarPath + " " + request.mainClass(); + case "native" -> "exec " + appJarPath + customArgs; + default -> "exec java -javaagent:/app/agent.jar" + customArgs + " -jar " + appJarPath; + }; + + var create = dockerClient.createContainerCmd(request.baseImage()) + .withName(request.containerName()) + .withEnv(envList) + .withLabels(request.labels() != null ? request.labels() : Map.of()) + .withHostConfig(hc) + .withHealthcheck(new HealthCheck() + .withTest(List.of("CMD-SHELL", + "wget -qO- http://localhost:" + request.healthCheckPort() + "/cameleer/health || exit 1")) + .withInterval(10_000_000_000L) + .withTimeout(5_000_000_000L) + .withRetries(3) + .withStartPeriod(30_000_000_000L)) + .withEntrypoint("sh", "-c", entrypoint); + + if (request.exposedPorts() != null && !request.exposedPorts().isEmpty()) { + var ports = request.exposedPorts().stream() + .map(com.github.dockerjava.api.model.ExposedPort::tcp) + .toArray(com.github.dockerjava.api.model.ExposedPort[]::new); + create.withExposedPorts(ports); + } + + String id = create.exec().getId(); + dockerClient.startContainerCmd(id).exec(); + log.info("Started container {} ({})", request.containerName(), id); + return id; +} + +/** Hardening contract from issue #152 — applied uniformly to loader + main. */ +private HostConfig baseHardenedHostConfig() { + return HostConfig.newHostConfig() + .withCapDrop(Capability.values()) + .withSecurityOpts(List.of("no-new-privileges:true", "apparmor=docker-default")) + .withReadonlyRootfs(true) + .withPidsLimit(PIDS_LIMIT) + .withTmpFs(Map.of("/tmp", TMPFS_TMP_OPTS)); +} + +private void cleanup(String loaderId, String volumeName) { + try { dockerClient.removeContainerCmd(loaderId).withForce(true).exec(); } catch (Exception ignored) {} + cleanupVolume(volumeName); +} + +private void cleanupVolume(String volumeName) { + try { dockerClient.removeVolumeCmd(volumeName).exec(); } catch (Exception ignored) {} +} +``` + +- [ ] **Step 4: Volume cleanup on container removal** + +In `removeContainer(String containerId)`, also remove the per-replica volume. The volume name is derived from the container name; look up the container name first via `inspectContainerCmd`: + +```java +@Override +public void removeContainer(String containerId) { + String volumeName = null; + try { + var name = dockerClient.inspectContainerCmd(containerId).exec().getName(); + if (name != null) { + // Docker prefixes inspected names with '/' + volumeName = "cameleer-jars-" + name.replaceFirst("^/", ""); + } + } catch (Exception e) { log.warn("Could not inspect {} for volume name: {}", containerId, e.getMessage()); } + try { + dockerClient.removeContainerCmd(containerId).withForce(true).exec(); + log.info("Removed container {}", containerId); + } catch (Exception e) { + log.warn("Failed to remove container {}: {}", containerId, e.getMessage()); + } + if (volumeName != null) cleanupVolume(volumeName); +} +``` + +- [ ] **Step 5: Run all orchestrator tests — expect PASS** + +Run: `mvn -pl cameleer-server-app test -Dtest='DockerRuntimeOrchestrator*'` + +The existing `DockerRuntimeOrchestratorHardeningTest` will need updating because `startContainer` now creates a volume and runs the loader first. Either update it to mock the loader path, or rewrite its assertions to inspect the *main* container's `HostConfig` capture (which still has the full hardening contract). Both paths must keep `cap_drop ALL`, `no-new-privileges`, `apparmor`, `readonly rootfs`, `pids_limit`, `/tmp` tmpfs. + +- [ ] **Step 6: Commit** + +```bash +git add -A +git commit -m "feat(runtime): orchestrator launches loader init container then main; lands withUsernsMode" +``` + +--- + +## Task 11: `DeploymentExecutor` — generate signed URL, drop filesystem checks + +**Files:** +- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/runtime/DeploymentExecutor.java` + +- [ ] **Step 1: Inject `ArtifactDownloadTokenSigner` + new config** + +Add to constructor + @Value: + +```java +@Value("${cameleer.server.runtime.loaderimage:gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest}") +private String loaderImage; + +@Value("${cameleer.server.runtime.artifacttokenttlseconds:600}") +private long artifactTokenTtlSeconds; + +@Value("${cameleer.server.runtime.artifactbaseurl:}") +private String artifactBaseUrl; // e.g. http://cameleer-server:8081 — falls back to globalServerUrl + +@Autowired +private ArtifactDownloadTokenSigner artifactTokenSigner; +``` + +- [ ] **Step 2: Replace JAR resolution in `executeAsync`** + +Drop the `String jarPath = appService.resolveJarPath(deployment.appVersionId());` line. Replace with: + +```java +AppVersion appVersion = appService.getVersion(deployment.appVersionId()); +String artifactBase = artifactBaseUrl.isBlank() + ? (globalServerUrl.isBlank() ? "http://cameleer-server:8081" : globalServerUrl) + : artifactBaseUrl; +var token = artifactTokenSigner.sign(appVersion.id(), java.time.Duration.ofSeconds(artifactTokenTtlSeconds)); +String artifactUrl = artifactBase + "/api/v1/artifacts/" + appVersion.id() + + "?exp=" + token.exp() + "&sig=" + token.sig(); +long expectedSize = appVersion.jarSizeBytes() == null ? 0L : appVersion.jarSizeBytes(); +``` + +Pass `appVersion`, `artifactUrl`, `expectedSize` into the `DeployCtx`: + +```java +private record DeployCtx( + Deployment deployment, + App app, + Environment env, + ResolvedContainerConfig config, + UUID appVersionId, + String artifactUrl, + long artifactExpectedSize, + String resolvedRuntimeType, + String mainClass, + String generation, + String primaryNetwork, + List additionalNets, + Map baseEnvVars, + Map prometheusLabels, + long deployStart +) {} +``` + +- [ ] **Step 3: Drop the filesystem check in `preFlightChecks`** + +```java +private void preFlightChecks(AppVersion appVersion, ResolvedContainerConfig config) { + if (appVersion.jarSizeBytes() == null || appVersion.jarSizeBytes() <= 0) { + throw new IllegalStateException("AppVersion " + appVersion.id() + " has no recorded jarSizeBytes"); + } + if (config.memoryLimitMb() <= 0) throw new IllegalStateException("..."); + if (config.appPort() <= 0 || config.appPort() > 65535) throw new IllegalStateException("..."); + if (config.replicas() < 1) throw new IllegalStateException("..."); +} +``` + +Remove the `Files.exists` import. + +- [ ] **Step 4: Update `startReplica` to build the new `ContainerRequest`** + +```java +ContainerRequest request = new ContainerRequest( + containerName, baseImage, + ctx.appVersionId(), ctx.artifactUrl(), ctx.artifactExpectedSize(), + loaderImage, + ctx.primaryNetwork(), + ctx.additionalNets(), + replicaEnvVars, labels, + config.memoryLimitBytes(), config.memoryReserveBytes(), + config.dockerCpuShares(), config.dockerCpuQuota(), + config.exposedPorts(), agentHealthPort, + "on-failure", 3, + ctx.resolvedRuntimeType(), config.customArgs(), ctx.mainClass() +); +``` + +Drop the `volumeName`/`jarStoragePath` plumbing. + +- [ ] **Step 5: Drop `jarDockerVolume` + `jarStoragePath` @Value fields** + +They're unused now. Also drop the `cameleer.server.runtime.jardockervolume` doc references in CLAUDE.md. + +- [ ] **Step 6: Run executor tests — expect PASS** + +```bash +mvn -pl cameleer-server-app test -Dtest='DeploymentExecutor*' +``` + +- [ ] **Step 7: Commit** + +```bash +git add -A +git commit -m "feat(runtime): DeploymentExecutor generates signed URL; drops filesystem JAR plumbing" +``` + +--- + +## Task 12: Configuration + integration smoke test + +**Files:** +- Modify: `cameleer-server-app/src/main/resources/application.yml` +- New: `cameleer-server-app/src/test/java/com/cameleer/server/app/runtime/InitContainerDeployIT.java` + +- [ ] **Step 1: Add new properties to `application.yml`** + +```yaml +cameleer: + server: + runtime: + loaderimage: ${CAMELEER_SERVER_RUNTIME_LOADERIMAGE:gitea.siegeln.net/cameleer/cameleer-runtime-loader:latest} + artifacttokenttlseconds: ${CAMELEER_SERVER_RUNTIME_ARTIFACTTOKENTTLSECONDS:600} + artifactbaseurl: ${CAMELEER_SERVER_RUNTIME_ARTIFACTBASEURL:} +``` + +Drop `jardockervolume` and (keep `jarstoragepath` for the filesystem store, but mark it as filesystem-store-specific). + +- [ ] **Step 2: Write the integration test (Testcontainers + real Docker)** + +Drives the whole flow: spin up the server in MockMvc/full context, upload a tiny JAR, trigger a deployment against a local Docker daemon, assert: per-replica volume created → loader exited 0 → main container running → JAR contents in `/app/jars/app.jar` match. Use `AbstractPostgresIT` as a base. Build the loader image inside the test if not present in the local cache, OR skip when not present and emit a warning. + +(Detailed test code omitted — model after `cameleer-server-app/src/test/java/.../runtime/*IT.java`.) + +- [ ] **Step 3: Run with Docker daemon** + +```bash +mvn -pl cameleer-server-app verify -Dit.test=InitContainerDeployIT +``` + +- [ ] **Step 4: Commit** + +```bash +git add -A +git commit -m "test(runtime): end-to-end IT for init-container deploy" +``` + +--- + +## Task 13: Update class/endpoint maps + +**Files:** +- Modify: `.claude/rules/docker-orchestration.md` — describe the loader pattern, named volume per replica, `withUsernsMode`, drop the "JAR bind-mount" lines +- Modify: `.claude/rules/core-classes.md` — add `ArtifactStore`, `ArtifactCoordinates` to a new "storage/" subsection; update `ContainerRequest` field list; remove `AppService.resolveJarPath` +- Modify: `.claude/rules/app-classes.md` — add `ArtifactDownloadController` + `ArtifactDownloadTokenSigner` + `FilesystemArtifactStore`; update the flat-endpoint allow-list with `/api/v1/artifacts/{appVersionId}` (reason: "Content-addressed download. HMAC-signed URL, no JWT context") +- Modify: `CLAUDE.md` — drop the `CAMELEER_SERVER_RUNTIME_JARDOCKERVOLUME` mention, replace with the loader-image / token-TTL / artifact-base-url envs + +- [ ] **Step 1: Edit each rule file** + +(No code template — read the existing file and make targeted updates.) + +- [ ] **Step 2: Run `gitnexus_detect_changes` to verify scope** + +```bash +# This is project policy per CLAUDE.md +``` + +- [ ] **Step 3: Commit** + +```bash +git add -A +git commit -m "docs(rules): document init-container loader + ArtifactStore; drop bind-mount" +``` + +--- + +## Task 14: Regenerate OpenAPI schema + +The new `ArtifactDownloadController` adds a public endpoint — SPA types need to know about it (even if the SPA never calls it directly). + +- [ ] **Step 1: Start the backend** + +```bash +mvn -pl cameleer-server-app spring-boot:run +``` + +- [ ] **Step 2: Regenerate** + +```bash +cd ui && npm run generate-api:live +``` + +- [ ] **Step 3: Fix any TypeScript compile errors that surface** + +```bash +cd ui && npm run typecheck +``` + +- [ ] **Step 4: Commit** + +```bash +git add ui/src/api/openapi.json ui/src/api/schema.d.ts +git commit -m "chore(ui): regenerate openapi schema for /api/v1/artifacts" +``` + +--- + +## Self-Review + +- **Spec coverage** — all four "discussed" items map to tasks: + - ArtifactStore abstraction → Tasks 1–5 + - HMAC-signed URL + controller → Tasks 6–7 + - Loader image → Task 8 + - Container request + orchestrator + executor → Tasks 9–11 + - `withUsernsMode` → embedded in Task 10 (lands once volume model is in) + - Maven coordinates UX → **deferred**, separate plan (out of scope; documented as a follow-on in the parent issue discussion) +- **Type consistency** — `coordinatesFor(AppVersion)` is referenced in Tasks 5 and 7 with the same signature; `ArtifactDownloadTokenSigner.SignedToken(exp, sig)` reused unchanged in 6/7/11; `ContainerRequest`'s new fields used identically in 9/10/11. +- **Placeholder scan** — Task 12 step 2 ("Detailed test code omitted") is the only deliberate omission; flagged so the engineer doesn't miss it. Everything else has either runnable code or specific edits. +- **Hidden assumption to verify in Task 4 step 3:** `RuntimeDetector.detect(InputStream)` may need to be added — current code takes a `Path`. Either add the overload or stage to a temp file inside `uploadJar`. +- **Migration safety** — existing `app_versions.jar_path` rows continue to resolve because `FilesystemArtifactStore.locator()` returns the same absolute path the upload code wrote yesterday. No data migration required.