cameleer-server/HOWTO.md

# HOWTO — Cameleer Server

## Prerequisites

- Java 17+
- Maven 3.9+
- Node.js 22+ and npm
- Docker & Docker Compose
- Access to the Gitea Maven registry (for `cameleer-common` dependency)

## Build

```bash
# Build UI first (required for embedded mode)
cd ui && npm ci && npm run build && cd ..

# Backend
mvn clean compile          # compile only
mvn clean verify           # compile + run all tests (needs Docker for integration tests)
```

## Start a brand-new local environment (Docker)

The repo ships a `docker-compose.yml` with the full stack: PostgreSQL, ClickHouse, the Spring Boot server, and the nginx-served SPA. All dev defaults are baked into the compose file — no `.env` file or extra config needed for a first run.

```bash
# 1. Clean slate (safe if this is already a first run — noop when no volumes exist)
docker compose down -v

# 2. Build + start everything. First run rebuilds both images (~2–4 min).
docker compose up -d --build

# 3. Watch the server come up (health check goes green in ~60–90s after Flyway + ClickHouse init)
docker compose logs -f cameleer-server
#   ready when you see "Started CameleerServerApplication in ...".
#   Ctrl+C when ready — containers keep running.

# 4. Smoke test
curl -s http://localhost:8081/api/v1/health     # → {"status":"UP"}
```

Open the UI at **http://localhost:8080** (nginx) and log in with **admin / admin**.

| Service    | Host port | URL / notes |
|------------|-----------|-------------|
| Web UI (nginx) | 8080 | http://localhost:8080 — proxies `/api` to the server |
| Server API | 8081 | http://localhost:8081/api/v1/health, http://localhost:8081/api/v1/swagger-ui.html |
| PostgreSQL | 5432 | user `cameleer`, password `cameleer_dev`, db `cameleer` |
| ClickHouse | 8123 (HTTP), 9000 (native) | user `default`, no password, db `cameleer` |

**Dev credentials baked into compose (do not use in production):**

| Purpose | Value |
|---|---|
| UI login | `admin` / `admin` |
| Bootstrap token (agent registration) | `dev-bootstrap-token-for-local-agent-registration` |
| JWT secret | `dev-jwt-secret-32-bytes-min-0123456789abcdef0123456789abcdef` |
| `CAMELEER_SERVER_RUNTIME_ENABLED` | `false` (Docker-in-Docker app orchestration off for the local stack) |

Override any of these by editing `docker-compose.yml` or passing `-e KEY=value` to `docker compose run`.

### Common lifecycle commands

```bash
# Stop everything but keep volumes (quick restart later)
docker compose stop

# Start again after a stop
docker compose start

# Apply changes to the server code / UI — rebuild just what changed
docker compose up -d --build cameleer-server
docker compose up -d --build cameleer-ui

# Wipe the environment completely (drops PG + ClickHouse volumes — all data gone)
docker compose down -v

# Fresh Flyway run by dropping just the PG volume (keeps ClickHouse data)
docker compose down
docker volume rm cameleer-server_cameleer-pgdata
docker compose up -d
```

### Infra-only mode (backend via `mvn` / UI via Vite)

If you want to iterate on backend/UI code without rebuilding the server image on every change, start just the databases and run the server + UI locally:

```bash
# 1. Only infra containers
docker compose up -d cameleer-postgres cameleer-clickhouse

# 2. Build and run the server jar against those containers
mvn clean package -DskipTests
SPRING_DATASOURCE_URL="jdbc:postgresql://localhost:5432/cameleer?currentSchema=tenant_default&ApplicationName=tenant_default" \
SPRING_DATASOURCE_USERNAME=cameleer \
SPRING_DATASOURCE_PASSWORD=cameleer_dev \
SPRING_FLYWAY_USER=cameleer \
SPRING_FLYWAY_PASSWORD=cameleer_dev \
CAMELEER_SERVER_CLICKHOUSE_URL="jdbc:clickhouse://localhost:8123/cameleer" \
CAMELEER_SERVER_CLICKHOUSE_USERNAME=default \
CAMELEER_SERVER_CLICKHOUSE_PASSWORD= \
CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN=dev-bootstrap-token-for-local-agent-registration \
CAMELEER_SERVER_SECURITY_JWTSECRET=dev-jwt-secret-32-bytes-min-0123456789abcdef0123456789abcdef \
CAMELEER_SERVER_RUNTIME_ENABLED=false \
CAMELEER_SERVER_TENANT_ID=default \
java -jar cameleer-server-app/target/cameleer-server-app-1.0-SNAPSHOT.jar

# 3. In another terminal — Vite dev server on :5173 (proxies /api → :8081)
cd ui && npm install && npm run dev
```

Database schema is applied automatically: PostgreSQL via Flyway migrations on server startup, ClickHouse tables via `ClickHouseSchemaInitializer`. No manual DDL needed.

`CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` is **required** for agent registration — the server fails fast on startup if it's not set. For token rotation without downtime, set `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKENPREVIOUS` to the old token while rolling out the new one — the server accepts both during the overlap window.

## API Endpoints

### Authentication (Phase 4)

All endpoints except health, registration, and docs require a JWT Bearer token. The typical flow:

```bash
# 1. Register agent (requires bootstrap token)
curl -s -X POST http://localhost:8081/api/v1/agents/register \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer my-secret-token" \
  -d '{"agentId":"agent-1","name":"Order Service","group":"order-service-prod","version":"1.0.0","routeIds":["route-1"],"capabilities":["deep-trace","replay"]}'
# Response includes: accessToken, refreshToken, serverPublicKey (Ed25519, Base64)

# 2. Use access token for all subsequent requests
TOKEN="<accessToken from registration>"

# 3. Refresh when access token expires (1h default)
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/refresh \
  -H "Authorization: Bearer <refreshToken>"
# Response: { "accessToken": "new-jwt" }
```

**UI Login (for browser access):**
```bash
# Login with UI credentials (returns JWT tokens)
curl -s -X POST http://localhost:8081/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}'
# Response: { "accessToken": "...", "refreshToken": "..." }

# Refresh UI token
curl -s -X POST http://localhost:8081/api/v1/auth/refresh \
  -H "Content-Type: application/json" \
  -d '{"refreshToken":"<refreshToken>"}'
```

UI credentials are configured via `CAMELEER_SERVER_SECURITY_UIUSER` / `CAMELEER_SERVER_SECURITY_UIPASSWORD` env vars (default: `admin` / `admin`).

**Public endpoints (no JWT required):** `GET /api/v1/health`, `POST /api/v1/agents/register` (uses bootstrap token), `POST /api/v1/auth/**`, OpenAPI/Swagger docs.

**Protected endpoints (JWT required):** All other endpoints including ingestion, search, agent management, commands.

**SSE connections:** Authenticated via query parameter: `/agents/{id}/events?token=<jwt>` (EventSource API doesn't support custom headers).

**Ed25519 signatures:** All SSE command payloads (config-update, deep-trace, replay) include a `signature` field. Agents verify payload integrity using the `serverPublicKey` received during registration. The server generates a new ephemeral keypair on each startup — agents must re-register to get the new key.

### RBAC (Role-Based Access Control)

JWTs carry a `roles` claim. Endpoints are restricted by role:

| Role | Access |
|------|--------|
| `AGENT` | Data ingestion (`/data/**` — executions, diagrams, metrics, logs), heartbeat, SSE events, command ack |
| `VIEWER` | Search, execution detail, diagrams, agent list, app config (read-only) |
| `OPERATOR` | VIEWER + send commands to agents, route control, replay, edit app config |
| `ADMIN` | OPERATOR + user management, audit log, OIDC config, database admin (`/admin/**`) |

The env-var local user gets `ADMIN` role. Agents get `AGENT` role at registration.

**UI role gating:** The sidebar hides the Admin section for non-ADMIN users. Admin routes (`/admin/*`) redirect to `/` for non-admin. The diagram node toolbar and route control bar are hidden for VIEWER. Config is a main tab (`/config` shows all apps, `/config/:appId` filters to one app with detail panel; sidebar clicks stay on config tab, route clicks resolve to parent app). VIEWER sees read-only, OPERATOR+ can edit.

### OIDC Login (Optional)

OIDC configuration is stored in PostgreSQL and managed via the admin API or UI. The SPA checks if OIDC is available:

```bash
# 1. SPA checks if OIDC is available (returns 404 if not configured)
curl -s http://localhost:8081/api/v1/auth/oidc/config
# Returns: { "issuer": "...", "clientId": "...", "authorizationEndpoint": "..." }

# 2. After OIDC redirect, SPA sends the authorization code
curl -s -X POST http://localhost:8081/api/v1/auth/oidc/callback \
  -H "Content-Type: application/json" \
  -d '{"code":"auth-code-from-provider","redirectUri":"http://localhost:5173/callback"}'
# Returns: { "accessToken": "...", "refreshToken": "..." }
```

Local login remains available as fallback even when OIDC is enabled.

### OIDC Admin Configuration (ADMIN only)

OIDC settings are managed at runtime via the admin API. No server restart needed.

```bash
# Get current OIDC config
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/v1/admin/oidc

# Save OIDC config (client_secret: send "********" to keep existing, or new value to update)
curl -s -X PUT http://localhost:8081/api/v1/admin/oidc \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "enabled": true,
    "issuerUri": "http://cameleer-logto:3001/oidc",
    "clientId": "your-client-id",
    "clientSecret": "your-client-secret",
    "rolesClaim": "realm_access.roles",
    "defaultRoles": ["VIEWER"]
  }'

# Test OIDC provider connectivity
curl -s -X POST http://localhost:8081/api/v1/admin/oidc/test \
  -H "Authorization: Bearer $TOKEN"

# Delete OIDC config (disables OIDC)
curl -s -X DELETE http://localhost:8081/api/v1/admin/oidc \
  -H "Authorization: Bearer $TOKEN"
```

**Initial provisioning**: OIDC can also be seeded from `CAMELEER_SERVER_SECURITY_OIDC*` env vars on first startup (when DB is empty). After that, the admin API takes over.

### Logto Setup (OIDC Provider)

Logto is deployed alongside the Cameleer stack. After first deployment:

Logto is proxy-aware via `TRUST_PROXY_HEADER=1`. The `LOGTO_ENDPOINT` and `LOGTO_ADMIN_ENDPOINT` secrets define the public-facing URLs that Logto uses for OIDC discovery, issuer URI, and redirect URLs. When behind a reverse proxy (e.g., Traefik), set these to the external URLs (e.g., `https://auth.cameleer.my.domain`). Logto needs its own subdomain — it cannot be path-prefixed under another app.

1. **Initial setup**: Open the Logto admin console (the `LOGTO_ADMIN_ENDPOINT` URL) and create the admin account
2. **Create SPA application**: Applications → Create → Single Page App
   - Name: `Cameleer UI`
   - Redirect URI: your UI URL + `/oidc/callback`
   - Note the **Client ID**
3. **Create API Resource**: API Resources → Create
   - Name: `Cameleer Server API`
   - Indicator: your API URL (e.g., `https://cameleer.siegeln.net/api`)
   - Add permissions: `server:admin`, `server:operator`, `server:viewer`
4. **Create M2M application** (for SaaS platform): Applications → Create → Machine-to-Machine
   - Name: `Cameleer SaaS`
   - Assign the API Resource created above with `server:admin` scope
   - Note the **Client ID** and **Client Secret**
5. **Configure Cameleer OIDC login**: Use the admin API (`PUT /api/v1/admin/oidc`) or the admin UI. OIDC login configuration is stored in the database — no env vars needed for the SPA OIDC flow.
6. **Configure resource server** (for M2M token validation):
   ```
   CAMELEER_SERVER_SECURITY_OIDCISSUERURI=<LOGTO_ENDPOINT>/oidc
   CAMELEER_SERVER_SECURITY_OIDCJWKSETURI=http://cameleer-logto:3001/oidc/jwks
   CAMELEER_SERVER_SECURITY_OIDCAUDIENCE=<api-resource-indicator-from-step-3>
   CAMELEER_SERVER_SECURITY_OIDCTLSSKIPVERIFY=true   # optional — skip cert verification for self-signed CAs
   ```
   `OIDCJWKSETURI` is needed when the public issuer URL isn't reachable from inside containers — it fetches JWKS directly from the internal Logto service. `OIDCTLSSKIPVERIFY` disables certificate verification for all OIDC HTTP calls (discovery, token exchange, JWKS); use only when the provider has a self-signed CA.

### SSO Behavior

When OIDC is configured and enabled, the UI automatically redirects to the OIDC provider for silent SSO (`prompt=none`). Users with an active provider session are signed in without seeing a login form. On first login, the provider may show a consent screen (scopes), after which subsequent logins are seamless. If auto-signup is enabled, new users are automatically provisioned with the configured default roles.

- **Bypass SSO**: Navigate to `/login?local` to see the local login form
- **Subpath deployments**: The OIDC redirect_uri respects `BASE_PATH` (e.g., `https://host/server/oidc/callback`)
- **Role sync**: System roles (ADMIN/OPERATOR/VIEWER) are synced from OIDC scopes on every login — revoking a scope in the provider takes effect on next login. Manually assigned group memberships are preserved.

### User Management (ADMIN only)

```bash
# List all users
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/v1/admin/users

# Update user roles
curl -s -X PUT http://localhost:8081/api/v1/admin/users/{userId}/roles \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"roles":["VIEWER","OPERATOR"]}'

# Delete user
curl -s -X DELETE http://localhost:8081/api/v1/admin/users/{userId} \
  -H "Authorization: Bearer $TOKEN"
```

### Ingestion (POST, returns 202 Accepted)

```bash
# Post route execution data (JWT required)
curl -s -X POST http://localhost:8081/api/v1/data/executions \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"agentId":"agent-1","routeId":"route-1","executionId":"exec-1","status":"COMPLETED","startTime":"2026-03-11T00:00:00Z","endTime":"2026-03-11T00:00:01Z","processorExecutions":[]}'

# Post route diagram
curl -s -X POST http://localhost:8081/api/v1/data/diagrams \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"agentId":"agent-1","routeId":"route-1","version":1,"nodes":[],"edges":[]}'

# Post agent metrics
curl -s -X POST http://localhost:8081/api/v1/data/metrics \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -H "Authorization: Bearer $TOKEN" \
  -d '[{"agentId":"agent-1","metricName":"cpu","value":42.0,"timestamp":"2026-03-11T00:00:00Z","tags":{}}]'

# Post application log entries (raw JSON array — no wrapper)
curl -s -X POST http://localhost:8081/api/v1/data/logs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '[{
    "timestamp": "2026-03-25T10:00:00Z",
    "level": "INFO",
    "loggerName": "com.acme.MyService",
    "message": "Processing order #12345",
    "threadName": "main",
    "source": "app"
  }]'
```

**Note:** The `X-Protocol-Version: 1` header is required on all `/api/v1/data/**` endpoints. Missing or wrong version returns 400.

### Health & Docs

```bash
# Health check
curl -s http://localhost:8081/api/v1/health

# OpenAPI JSON
curl -s http://localhost:8081/api/v1/api-docs

# Swagger UI
open http://localhost:8081/api/v1/swagger-ui.html
```

### Search (Phase 2)

```bash
# Search by status (GET with basic filters)
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8081/api/v1/search/executions?status=COMPLETED&limit=10"

# Search by time range
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8081/api/v1/search/executions?timeFrom=2026-03-11T00:00:00Z&timeTo=2026-03-12T00:00:00Z"

# Advanced search (POST with full-text)
curl -s -X POST http://localhost:8081/api/v1/search/executions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"status":"FAILED","text":"NullPointerException","limit":20}'

# Transaction detail (nested processor tree)
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:8081/api/v1/executions/{executionId}

# Processor exchange snapshot
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:8081/api/v1/executions/{executionId}/processors/{index}/snapshot

# Render diagram as SVG
curl -s -H "Authorization: Bearer $TOKEN" \
  -H "Accept: image/svg+xml" \
  http://localhost:8081/api/v1/diagrams/{contentHash}/render

# Render diagram as JSON layout
curl -s -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/json" \
  http://localhost:8081/api/v1/diagrams/{contentHash}/render
```

**Search response format:** `{ "data": [...], "total": N, "offset": 0, "limit": 50 }`

**Supported search filters (GET):** `status`, `timeFrom`, `timeTo`, `correlationId`, `limit`, `offset`

**Additional POST filters:** `durationMin`, `durationMax`, `text` (global full-text), `textInBody`, `textInHeaders`, `textInErrors`

### Agent Registry & SSE (Phase 3)

```bash
# Register an agent (uses bootstrap token, not JWT — see Authentication section above)
curl -s -X POST http://localhost:8081/api/v1/agents/register \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer my-secret-token" \
  -d '{"agentId":"agent-1","name":"Order Service","group":"order-service-prod","version":"1.0.0","routeIds":["route-1","route-2"],"capabilities":["deep-trace","replay"]}'

# Heartbeat (call every 30s)
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/heartbeat \
  -H "Authorization: Bearer $TOKEN"

# List agents (optionally filter by status)
curl -s -H "Authorization: Bearer $TOKEN" "http://localhost:8081/api/v1/agents"
curl -s -H "Authorization: Bearer $TOKEN" "http://localhost:8081/api/v1/agents?status=LIVE"

# Connect to SSE event stream (JWT via query parameter)
curl -s -N "http://localhost:8081/api/v1/agents/agent-1/events?token=$TOKEN"

# Send command to single agent
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/commands \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"type":"config-update","payload":{"samplingRate":0.5}}'

# Send command to agent group
curl -s -X POST http://localhost:8081/api/v1/agents/groups/order-service-prod/commands \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"type":"deep-trace","payload":{"routeId":"route-1","durationSeconds":60}}'

# Send route control command to agent group (start/stop/suspend/resume)
curl -s -X POST http://localhost:8081/api/v1/agents/groups/order-service-prod/commands \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"type":"route-control","payload":{"routeId":"route-1","action":"stop","nonce":"unique-uuid"}}'

# Broadcast command to all live agents
curl -s -X POST http://localhost:8081/api/v1/agents/commands \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"type":"config-update","payload":{"samplingRate":1.0}}'

# Acknowledge command delivery
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/commands/{commandId}/ack \
  -H "Authorization: Bearer $TOKEN"
```

**Agent lifecycle:** LIVE (heartbeat within 90s) → STALE (missed 3 heartbeats) → DEAD (5min after STALE). DEAD agents kept indefinitely.

**Server restart resilience:** The agent registry is in-memory and lost on server restart. Agents auto-re-register on their next heartbeat or SSE connection — the server reconstructs registry entries from JWT claims (subject, application). Route catalog uses ClickHouse execution data as fallback until agents re-register with full route IDs. Agents should also handle 404 on heartbeat by triggering a full re-registration.

**SSE events:** `config-update`, `deep-trace`, `replay`, `route-control` commands pushed in real time. Server sends ping keepalive every 15s.

**Command expiry:** Unacknowledged commands expire after 60 seconds.

**Route control responses:** Route control commands return `CommandGroupResponse` with per-agent status, response count, and timed-out agent IDs.

### Backpressure

When the write buffer is full (default capacity: 50,000), ingestion endpoints return **503 Service Unavailable**. Already-buffered data is not lost.

## Configuration

Key settings in `cameleer-server-app/src/main/resources/application.yml`. All custom properties live under `cameleer.server.*`. Env vars are a mechanical 1:1 mapping (dots to underscores, uppercase).

**Security** (`cameleer.server.security.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.security.bootstraptoken` | *(required)* | `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` | Bootstrap token for agent registration |
| `cameleer.server.security.bootstraptokenprevious` | *(empty)* | `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKENPREVIOUS` | Previous bootstrap token for rotation |
| `cameleer.server.security.uiuser` | `admin` | `CAMELEER_SERVER_SECURITY_UIUSER` | UI login username |
| `cameleer.server.security.uipassword` | `admin` | `CAMELEER_SERVER_SECURITY_UIPASSWORD` | UI login password |
| `cameleer.server.security.uiorigin` | `http://localhost:5173` | `CAMELEER_SERVER_SECURITY_UIORIGIN` | CORS allowed origin for UI |
| `cameleer.server.security.corsallowedorigins` | *(empty)* | `CAMELEER_SERVER_SECURITY_CORSALLOWEDORIGINS` | Comma-separated CORS origins — overrides `uiorigin` when set |
| `cameleer.server.security.jwtsecret` | *(random)* | `CAMELEER_SERVER_SECURITY_JWTSECRET` | HMAC secret for JWT signing. If set, tokens survive restarts |
| `cameleer.server.security.accesstokenexpiryms` | `3600000` | `CAMELEER_SERVER_SECURITY_ACCESSTOKENEXPIRYMS` | JWT access token lifetime (1h) |
| `cameleer.server.security.refreshtokenexpiryms` | `604800000` | `CAMELEER_SERVER_SECURITY_REFRESHTOKENEXPIRYMS` | Refresh token lifetime (7d) |
| `cameleer.server.security.infrastructureendpoints` | `true` | `CAMELEER_SERVER_SECURITY_INFRASTRUCTUREENDPOINTS` | Show DB/ClickHouse admin endpoints. Set `false` in SaaS-managed mode |

**OIDC resource server** (`cameleer.server.security.oidc.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.security.oidc.issueruri` | *(empty)* | `CAMELEER_SERVER_SECURITY_OIDC_ISSUERURI` | OIDC issuer URI — enables resource server mode |
| `cameleer.server.security.oidc.jwkseturi` | *(empty)* | `CAMELEER_SERVER_SECURITY_OIDC_JWKSETURI` | Direct JWKS URL — bypasses OIDC discovery |
| `cameleer.server.security.oidc.audience` | *(empty)* | `CAMELEER_SERVER_SECURITY_OIDC_AUDIENCE` | Expected JWT audience |
| `cameleer.server.security.oidc.tlsskipverify` | `false` | `CAMELEER_SERVER_SECURITY_OIDC_TLSSKIPVERIFY` | Skip TLS cert verification for OIDC calls |

**Note:** OIDC *login* configuration (issuer, client ID, client secret, roles claim, default roles) is stored in the database and managed via the admin API (`PUT /api/v1/admin/oidc`) or admin UI. The env vars above are for resource server mode (M2M token validation) only.

**Ingestion** (`cameleer.server.ingestion.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.ingestion.buffercapacity` | `50000` | `CAMELEER_SERVER_INGESTION_BUFFERCAPACITY` | Max items in write buffer |
| `cameleer.server.ingestion.batchsize` | `5000` | `CAMELEER_SERVER_INGESTION_BATCHSIZE` | Items per batch insert |
| `cameleer.server.ingestion.flushintervalms` | `5000` | `CAMELEER_SERVER_INGESTION_FLUSHINTERVALMS` | Buffer flush interval (ms) |
| `cameleer.server.ingestion.bodysizelimit` | `16384` | `CAMELEER_SERVER_INGESTION_BODYSIZELIMIT` | Max body size per execution (bytes) |

**Agent registry** (`cameleer.server.agentregistry.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.agentregistry.heartbeatintervalms` | `30000` | `CAMELEER_SERVER_AGENTREGISTRY_HEARTBEATINTERVALMS` | Expected heartbeat interval (ms) |
| `cameleer.server.agentregistry.stalethresholdms` | `90000` | `CAMELEER_SERVER_AGENTREGISTRY_STALETHRESHOLDMS` | Time before agent marked STALE (ms) |
| `cameleer.server.agentregistry.deadthresholdms` | `300000` | `CAMELEER_SERVER_AGENTREGISTRY_DEADTHRESHOLDMS` | Time after STALE before DEAD (ms) |
| `cameleer.server.agentregistry.pingintervalms` | `15000` | `CAMELEER_SERVER_AGENTREGISTRY_PINGINTERVALMS` | SSE ping keepalive interval (ms) |
| `cameleer.server.agentregistry.commandexpiryms` | `60000` | `CAMELEER_SERVER_AGENTREGISTRY_COMMANDEXPIRYMS` | Pending command TTL (ms) |
| `cameleer.server.agentregistry.lifecyclecheckintervalms` | `10000` | `CAMELEER_SERVER_AGENTREGISTRY_LIFECYCLECHECKINTERVALMS` | Lifecycle monitor interval (ms) |

**Runtime** (`cameleer.server.runtime.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.runtime.enabled` | `true` | `CAMELEER_SERVER_RUNTIME_ENABLED` | Enable Docker orchestration |
| `cameleer.server.runtime.baseimage` | `cameleer-runtime-base:latest` | `CAMELEER_SERVER_RUNTIME_BASEIMAGE` | Base Docker image for app containers |
| `cameleer.server.runtime.dockernetwork` | `cameleer` | `CAMELEER_SERVER_RUNTIME_DOCKERNETWORK` | Primary Docker network |
| `cameleer.server.runtime.jarstoragepath` | `/data/jars` | `CAMELEER_SERVER_RUNTIME_JARSTORAGEPATH` | JAR file storage directory |
| `cameleer.server.runtime.jardockervolume` | *(empty)* | `CAMELEER_SERVER_RUNTIME_JARDOCKERVOLUME` | Docker volume for JAR sharing |
| `cameleer.server.runtime.routingmode` | `path` | `CAMELEER_SERVER_RUNTIME_ROUTINGMODE` | `path` or `subdomain` Traefik routing |
| `cameleer.server.runtime.routingdomain` | `localhost` | `CAMELEER_SERVER_RUNTIME_ROUTINGDOMAIN` | Domain for Traefik routing labels |
| `cameleer.server.runtime.serverurl` | *(empty)* | `CAMELEER_SERVER_RUNTIME_SERVERURL` | Server URL injected into app containers |
| `cameleer.server.runtime.certresolver` | *(empty)* | `CAMELEER_SERVER_RUNTIME_CERTRESOLVER` | Traefik TLS cert resolver name (e.g. `letsencrypt`). Blank = omit the `tls.certresolver` label and let Traefik serve the default TLS-store cert |
| `cameleer.server.runtime.agenthealthport` | `9464` | `CAMELEER_SERVER_RUNTIME_AGENTHEALTHPORT` | Agent health check port |
| `cameleer.server.runtime.healthchecktimeout` | `60` | `CAMELEER_SERVER_RUNTIME_HEALTHCHECKTIMEOUT` | Health check timeout (seconds) |
| `cameleer.server.runtime.container.memorylimit` | `512m` | `CAMELEER_SERVER_RUNTIME_CONTAINER_MEMORYLIMIT` | Default memory limit for app containers |
| `cameleer.server.runtime.container.cpushares` | `512` | `CAMELEER_SERVER_RUNTIME_CONTAINER_CPUSHARES` | Default CPU shares for app containers |

**Other** (`cameleer.server.*`):

| Setting | Default | Env var | Description |
|---------|---------|---------|-------------|
| `cameleer.server.catalog.discoveryttldays` | `7` | `CAMELEER_SERVER_CATALOG_DISCOVERYTTLDAYS` | Days before stale discovered apps auto-hide from sidebar |
| `cameleer.server.tenant.id` | `default` | `CAMELEER_SERVER_TENANT_ID` | Tenant identifier |
| `cameleer.server.indexer.debouncems` | `2000` | `CAMELEER_SERVER_INDEXER_DEBOUNCEMS` | Search indexer debounce delay (ms) |
| `cameleer.server.indexer.queuesize` | `10000` | `CAMELEER_SERVER_INDEXER_QUEUESIZE` | Search indexer queue capacity |
| `cameleer.server.license.token` | *(empty)* | `CAMELEER_SERVER_LICENSE_TOKEN` | License token |
| `cameleer.server.license.publickey` | *(empty)* | `CAMELEER_SERVER_LICENSE_PUBLICKEY` | License verification public key |
| `cameleer.server.clickhouse.url` | `jdbc:clickhouse://localhost:8123/cameleer` | `CAMELEER_SERVER_CLICKHOUSE_URL` | ClickHouse JDBC URL |
| `cameleer.server.clickhouse.username` | `default` | `CAMELEER_SERVER_CLICKHOUSE_USERNAME` | ClickHouse user |
| `cameleer.server.clickhouse.password` | *(empty)* | `CAMELEER_SERVER_CLICKHOUSE_PASSWORD` | ClickHouse password |

## Web UI Development

```bash
cd ui
npm install
npm run dev          # Vite dev server on http://localhost:5173 (proxies /api to :8081)
npm run build        # Production build to ui/dist/
```

Login with `admin` / `admin` (or whatever `CAMELEER_SERVER_SECURITY_UIUSER` / `CAMELEER_SERVER_SECURITY_UIPASSWORD` are set to).

The UI uses runtime configuration via `public/config.js`. In Kubernetes, a ConfigMap overrides this file to set the correct API base URL.

### Regenerate API Types

When the backend OpenAPI spec changes:
```bash
cd ui
npm run generate-api   # Requires backend running on :8081
```

## Running Tests

Integration tests use Testcontainers (starts PostgreSQL automatically — requires Docker):

```bash
# All tests
mvn verify

# Unit tests only (no Docker needed)
mvn test -pl cameleer-server-core

# Specific integration test
mvn test -pl cameleer-server-app -Dtest=ExecutionControllerIT
```

## Verify Database Data

After posting data and waiting for the flush interval (1s default):

```bash
docker exec -it cameleer-server-postgres-1 psql -U cameleer -d cameleer \
  -c "SELECT count(*) FROM route_executions"
```

## Kubernetes Deployment

The full stack is deployed to k3s via CI/CD on push to `main`. K8s manifests are in `deploy/`.

### Architecture

```
cameleer namespace:
  PostgreSQL (StatefulSet, 10Gi PVC)       ← cameleer-postgres:5432 (ClusterIP)
  ClickHouse (StatefulSet, 10Gi PVC)       ← cameleer-clickhouse:8123 (ClusterIP)
  cameleer-server (Deployment)            ← NodePort 30081
  cameleer-ui (Deployment, Nginx)         ← NodePort 30090
  cameleer-deploy-demo (Deployment)        ← NodePort 30092
  Logto Server (Deployment)               ← NodePort 30951/30952
  Logto PostgreSQL (StatefulSet, 1Gi)     ← ClusterIP

cameleer-demo namespace:
  (deployed Camel applications — managed by cameleer-deploy-demo)
```

### Access (from your network)

| Service | URL |
|---------|-----|
| Web UI | `http://192.168.50.86:30090` |
| Server API | `http://192.168.50.86:30081/api/v1/health` |
| Swagger UI | `http://192.168.50.86:30081/api/v1/swagger-ui.html` |
| Deploy Demo | `http://192.168.50.86:30092` |
| Logto API | `LOGTO_ENDPOINT` secret (NodePort 30951 direct, or behind reverse proxy) |
| Logto Admin | `LOGTO_ADMIN_ENDPOINT` secret (NodePort 30952 direct, or behind reverse proxy) |

### CI/CD Pipeline

Push to `main` triggers: **build** (UI npm + Maven, unit tests) → **docker** (buildx amd64 for server + UI, push to Gitea registry) → **deploy** (kubectl apply + rolling update).

Required Gitea org secrets: `REGISTRY_TOKEN`, `KUBECONFIG_BASE64`, `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN`, `CAMELEER_SERVER_SECURITY_JWTSECRET`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CAMELEER_SERVER_SECURITY_UIUSER` (optional), `CAMELEER_SERVER_SECURITY_UIPASSWORD` (optional), `LOGTO_PG_USER`, `LOGTO_PG_PASSWORD`, `LOGTO_ENDPOINT` (public-facing Logto URL, e.g., `https://auth.cameleer.my.domain`), `LOGTO_ADMIN_ENDPOINT` (admin console URL), `CAMELEER_SERVER_SECURITY_OIDCISSUERURI` (optional, for resource server M2M token validation), `CAMELEER_SERVER_SECURITY_OIDCAUDIENCE` (optional, API resource indicator), `CAMELEER_SERVER_SECURITY_OIDCTLSSKIPVERIFY` (optional, skip TLS cert verification for self-signed CAs).

### Manual K8s Commands

```bash
# Check pod status
kubectl -n cameleer get pods

# View server logs
kubectl -n cameleer logs -f deploy/cameleer-server

# View PostgreSQL logs
kubectl -n cameleer logs -f statefulset/cameleer-postgres

# View ClickHouse logs
kubectl -n cameleer logs -f statefulset/cameleer-clickhouse

# Restart server
kubectl -n cameleer rollout restart deployment/cameleer-server
```