# Container Startup Log Capture

Capture Docker container stdout/stderr from the moment a container starts until the Cameleer agent inside fully registers (SSE connection established). Stores logs in ClickHouse for display in the deployment view and general log search.

## Problem

When a deployed application crashes during startup — before the Cameleer agent can connect and send logs via the normal ingestion pipeline — all diagnostic output is lost. The container may be removed before anyone can inspect it, leaving operators blind to the root cause.

## Solution

A `ContainerLogForwarder` component streams Docker log output in real-time for each managed container, batches lines, and flushes them to the existing ClickHouse `logs` table with `source = 'container'`. Capture stops when the agent establishes its SSE connection, at which point the agent's own log pipeline takes over.

## Architecture

### Core Interface Extension

Extend `RuntimeOrchestrator` (core module) with three new methods:

```java
// in RuntimeOrchestrator.java
void startLogCapture(String containerId, String appSlug, String envSlug, String tenantId);
void stopLogCapture(String containerId);
void stopLogCaptureByApp(String appSlug, String envSlug);
```

`DisabledRuntimeOrchestrator` implements these as no-ops. `DockerRuntimeOrchestrator` delegates to `ContainerLogForwarder`.

### ContainerLogForwarder

**Package:** `com.cameleer.server.app.runtime` (Docker-specific, alongside `DockerRuntimeOrchestrator`, `DockerEventMonitor`, etc.)

**Responsibilities:**
- Manages active capture sessions in a `ConcurrentHashMap<String, CaptureSession>` keyed by container ID
- Each `CaptureSession` holds: containerId, appSlug, envSlug, tenantId, a `Future<?>` for the streaming thread, and a buffer of pending log lines
- Uses a bounded thread pool (fixed size ~10 threads)

**Streaming logic:**
- Calls `dockerClient.logContainerCmd(containerId).withFollowStream(true).withStdOut(true).withStdErr(true).withTimestamps(true)`
- Callback `onNext(Frame)` appends to an in-memory buffer
- Every ~2 seconds (or every 50 lines, whichever comes first), flushes the buffer to ClickHouse via `ClickHouseLogStore.insertBufferedBatch()` — constructs `BufferedLogEntry` records with `source = "container"`, the deployment's app/env/tenant metadata, and container name as `instanceId`
- On `onComplete()` (container stopped) or `onError()` — final flush, remove session from map

**Safety:**
- Max capture duration: 5 minutes. A scheduled cleanup (every 30s) stops sessions exceeding this limit.
- `@PreDestroy` cleanup: stop all active captures on server shutdown.

### ClickHouse Field Mapping

Uses the existing `logs` table. No schema changes required.

| Field | Value |
|-------|-------|
| `source` | `'container'` |
| `application` | appSlug from deployment |
| `environment` | envSlug from deployment |
| `tenant_id` | tenantId from deployment |
| `instance_id` | containerName (e.g., `prod-orderservice-0`) |
| `timestamp` | Parsed from Docker timestamp prefix |
| `message` | Log line content (after timestamp) |
| `level` | Inferred by regex (see below) |
| `logger_name` | Empty string (not parseable from raw stdout) |
| `thread_name` | Empty string |
| `stack_trace` | Empty string (stack traces appear as consecutive message lines) |
| `exchange_id` | Empty string |
| `mdc` | Empty map |

### Log Level Inference

- Regex scan for common Java log patterns: ` ERROR `, ` WARN `, ` INFO `, ` DEBUG `, ` TRACE `
- Stack trace continuation lines (starting with `\tat ` or `Caused by:`) inherit ERROR level
- Lines matching no pattern default to INFO

## Integration Points

### Start Capture — DeploymentExecutor

After each replica container is started (inside the replica loop):

```java
orchestrator.startLogCapture(containerId, appSlug, envSlug, tenantId);
```

### Stop Capture — SseConnectionManager.connect()

When an agent connects SSE, look up its `AgentInfo` from the registry to get `application` + `environmentId`:

```java
orchestrator.stopLogCaptureByApp(application, environmentId);
```

Best-effort call — no-op if no capture exists for that app+env (e.g., non-Docker agent).

### Stop Capture — Container Death

`DockerEventMonitor` handles `die`/`oom` events. After updating replica state:

```java
orchestrator.stopLogCapture(containerId);
```

Triggers final flush of buffered lines before cleanup.

### Stop Capture — Deployment Failure Cleanup

No extra code needed. When `DeploymentExecutor` stops/removes containers on health check failure, the Docker `die` event flows through `DockerEventMonitor` which calls `stopLogCapture`. The event monitor path handles it.

## UI Changes

### 1. Deployment Startup Log Panel

A collapsible log panel below the `DeploymentProgress` component in the deployment detail view.

**Data source:** Queries `/api/v1/logs?application={appSlug}&environment={envSlug}&source=container&from={deployCreatedAt}`

**Polling behavior:**
- Auto-refreshes every 3 seconds while deployment status is STARTING
- Stops polling when status reaches RUNNING or FAILED
- Manual refresh button available in all states

**Status indicator:**
- Green "live" badge + "polling every 3s" text while STARTING
- Red "stopped" badge when FAILED
- No badge when RUNNING (panel remains visible with historical startup logs)

**Layout:** Uses existing `LogViewer` component from `@cameleer/design-system` and shared log panel styles from `ui/src/styles/log-panel.module.css`.

### 2. Source Badge in Log Views

Everywhere logs are displayed (AgentInstance page, LogTab, general log search), each log line gets a small source badge:
- `container` — slate/gray badge
- `app` — green badge
- `agent` — existing behavior

The `source` field already exists in `LogEntryResponse`. This is a rendering-only change in the LogViewer or its wrapper.

### 3. Source Filter Update

The log toolbar source filter (currently App vs Agent) adds `Container` as a third option. The backend `/api/v1/logs` endpoint already accepts `source` as a query parameter — no backend change needed for filtering.

## Edge Cases

**Multi-replica:** Each replica gets its own capture session keyed by container ID. `instance_id` in ClickHouse is the container name (e.g., `prod-orderservice-0`). `stopLogCaptureByApp()` stops all sessions for that app+env pair.

**Server restart during capture:** Active sessions are in-memory and lost on restart. Not a problem — containers likely restart too (Docker restart policy), and new captures start when `DeploymentExecutor` runs again. Already-flushed logs survive in ClickHouse.

**Container produces no output:** Follow stream stays open but idle (parked thread, no CPU cost). Cleaned up by the 5-minute timeout or container death.

**Rapid redeployment:** Old container dies -> `stopLogCapture(oldContainerId)`. New container starts -> `startLogCapture(newContainerId, ...)`. Different container IDs, no conflict.

**Log overlap:** When the agent connects and starts sending `source='app'` logs, there may be a brief overlap with `source='container'` logs for the same timeframe. Both are shown with source badges. Users can filter by source if needed.

## Files Changed

### Backend — New

| File | Description |
|------|-------------|
| `app/runtime/ContainerLogForwarder.java` | Docker log streaming, buffering, ClickHouse flush |

### Backend — Modified

| File | Change |
|------|--------|
| `core/runtime/RuntimeOrchestrator.java` | Add 3 log capture methods to interface |
| `app/runtime/DockerRuntimeOrchestrator.java` | Implement log capture methods, delegate to ContainerLogForwarder |
| `app/runtime/DisabledRuntimeOrchestrator.java` | No-op implementations of new methods |
| `app/runtime/DeploymentExecutor.java` | Call `startLogCapture()` after container start |
| `app/agent/SseConnectionManager.java` | Call `stopLogCaptureByApp()` on SSE connect |
| `app/runtime/DockerEventMonitor.java` | Call `stopLogCapture()` on die/oom events |
| `app/runtime/RuntimeOrchestratorAutoConfig.java` | Wire ContainerLogForwarder into DockerRuntimeOrchestrator |

### Frontend — Modified

| File | Change |
|------|--------|
| `ui/src/pages/AppsTab/AppsTab.tsx` | Add startup log panel below DeploymentProgress |
| `ui/src/api/queries/logs.ts` | Hook for deployment startup logs query |
| Log display components | Add source badge rendering |
| Log toolbar | Add Container to source filter options |

### No Changes

| File | Reason |
|------|--------|
| ClickHouse `init.sql` | Existing `logs` table with `source` column is sufficient |
| `LogQueryController.java` | Already accepts `source` filter parameter |
| `ClickHouseLogStore.java` | Already writes `source` field from log entries |