cameleer-server/HOWTO.md

# HOWTO — Cameleer3 Server

## Prerequisites

- Java 17+
- Maven 3.9+
- Docker & Docker Compose
- Access to the Gitea Maven registry (for `cameleer3-common` dependency)

## Build

```bash
mvn clean compile          # compile only
mvn clean verify           # compile + run all tests (needs Docker for integration tests)
```

## Infrastructure Setup

Start ClickHouse:

```bash
docker compose up -d
```

This starts ClickHouse 25.3 and automatically runs the schema init scripts (`clickhouse/init/01-schema.sql`, `clickhouse/init/02-search-columns.sql`).

| Service    | Port | Purpose          |
|------------|------|------------------|
| ClickHouse | 8123 | HTTP API (JDBC)  |
| ClickHouse | 9000 | Native protocol  |

ClickHouse credentials: `cameleer` / `cameleer_dev`, database `cameleer3`.

## Run the Server

```bash
mvn clean package -DskipTests
java -jar cameleer3-server-app/target/cameleer3-server-app-1.0-SNAPSHOT.jar
```

The server starts on **port 8081**.

## API Endpoints

### Ingestion (POST, returns 202 Accepted)

```bash
# Post route execution data
curl -s -X POST http://localhost:8081/api/v1/data/executions \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -d '{"agentId":"agent-1","routeId":"route-1","executionId":"exec-1","status":"COMPLETED","startTime":"2026-03-11T00:00:00Z","endTime":"2026-03-11T00:00:01Z","processorExecutions":[]}'

# Post route diagram
curl -s -X POST http://localhost:8081/api/v1/data/diagrams \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -d '{"agentId":"agent-1","routeId":"route-1","version":1,"nodes":[],"edges":[]}'

# Post agent metrics
curl -s -X POST http://localhost:8081/api/v1/data/metrics \
  -H "Content-Type: application/json" \
  -H "X-Protocol-Version: 1" \
  -d '[{"agentId":"agent-1","metricName":"cpu","value":42.0,"timestamp":"2026-03-11T00:00:00Z","tags":{}}]'
```

**Note:** The `X-Protocol-Version: 1` header is required on all `/api/v1/data/**` endpoints. Missing or wrong version returns 400.

### Health & Docs

```bash
# Health check
curl -s http://localhost:8081/api/v1/health

# OpenAPI JSON
curl -s http://localhost:8081/api/v1/api-docs

# Swagger UI
open http://localhost:8081/api/v1/swagger-ui.html
```

### Search (Phase 2)

```bash
# Search by status (GET with basic filters)
curl -s "http://localhost:8081/api/v1/search/executions?status=COMPLETED&limit=10"

# Search by time range
curl -s "http://localhost:8081/api/v1/search/executions?timeFrom=2026-03-11T00:00:00Z&timeTo=2026-03-12T00:00:00Z"

# Advanced search (POST with full-text)
curl -s -X POST http://localhost:8081/api/v1/search/executions \
  -H "Content-Type: application/json" \
  -d '{"status":"FAILED","text":"NullPointerException","limit":20}'

# Transaction detail (nested processor tree)
curl -s http://localhost:8081/api/v1/executions/{executionId}

# Processor exchange snapshot
curl -s http://localhost:8081/api/v1/executions/{executionId}/processors/{index}/snapshot

# Render diagram as SVG
curl -s http://localhost:8081/api/v1/diagrams/{contentHash}/render \
  -H "Accept: image/svg+xml"

# Render diagram as JSON layout
curl -s http://localhost:8081/api/v1/diagrams/{contentHash}/render \
  -H "Accept: application/json"
```

**Search response format:** `{ "data": [...], "total": N, "offset": 0, "limit": 50 }`

**Supported search filters (GET):** `status`, `timeFrom`, `timeTo`, `correlationId`, `limit`, `offset`

**Additional POST filters:** `durationMin`, `durationMax`, `text` (global full-text), `textInBody`, `textInHeaders`, `textInErrors`

### Agent Registry & SSE (Phase 3)

```bash
# Register an agent
curl -s -X POST http://localhost:8081/api/v1/agents/register \
  -H "Content-Type: application/json" \
  -d '{"agentId":"agent-1","name":"Order Service","group":"order-service-prod","version":"1.0.0","routeIds":["route-1","route-2"],"capabilities":["deep-trace","replay"]}'

# Heartbeat (call every 30s)
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/heartbeat

# List agents (optionally filter by status)
curl -s "http://localhost:8081/api/v1/agents"
curl -s "http://localhost:8081/api/v1/agents?status=LIVE"

# Connect to SSE event stream (agent receives commands here)
curl -s -N http://localhost:8081/api/v1/agents/agent-1/events

# Send command to single agent
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/commands \
  -H "Content-Type: application/json" \
  -d '{"type":"config-update","payload":{"samplingRate":0.5}}'

# Send command to agent group
curl -s -X POST http://localhost:8081/api/v1/agents/groups/order-service-prod/commands \
  -H "Content-Type: application/json" \
  -d '{"type":"deep-trace","payload":{"routeId":"route-1","durationSeconds":60}}'

# Broadcast command to all live agents
curl -s -X POST http://localhost:8081/api/v1/agents/commands \
  -H "Content-Type: application/json" \
  -d '{"type":"config-update","payload":{"samplingRate":1.0}}'

# Acknowledge command delivery
curl -s -X POST http://localhost:8081/api/v1/agents/agent-1/commands/{commandId}/ack
```

**Agent lifecycle:** LIVE (heartbeat within 90s) → STALE (missed 3 heartbeats) → DEAD (5min after STALE). DEAD agents kept indefinitely.

**SSE events:** `config-update`, `deep-trace`, `replay` commands pushed in real time. Server sends ping keepalive every 15s.

**Command expiry:** Unacknowledged commands expire after 60 seconds.

### Backpressure

When the write buffer is full (default capacity: 50,000), ingestion endpoints return **503 Service Unavailable**. Already-buffered data is not lost.

## Configuration

Key settings in `cameleer3-server-app/src/main/resources/application.yml`:

| Setting | Default | Description |
|---------|---------|-------------|
| `server.port` | 8081 | Server port |
| `ingestion.buffer-capacity` | 50000 | Max items in write buffer |
| `ingestion.batch-size` | 5000 | Items per ClickHouse batch insert |
| `ingestion.flush-interval-ms` | 1000 | Buffer flush interval (ms) |
| `ingestion.data-ttl-days` | 30 | ClickHouse TTL for auto-deletion |
| `agent-registry.heartbeat-interval-seconds` | 30 | Expected heartbeat interval |
| `agent-registry.stale-threshold-seconds` | 90 | Time before agent marked STALE |
| `agent-registry.dead-threshold-seconds` | 300 | Time after STALE before DEAD |
| `agent-registry.command-expiry-seconds` | 60 | Pending command TTL |
| `agent-registry.keepalive-interval-seconds` | 15 | SSE ping keepalive interval |

## Running Tests

Integration tests use Testcontainers (starts ClickHouse automatically — requires Docker):

```bash
# All tests
mvn verify

# Unit tests only (no Docker needed)
mvn test -pl cameleer3-server-core

# Specific integration test
mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT
```

## Verify ClickHouse Data

After posting data and waiting for the flush interval (1s default):

```bash
docker exec -it cameleer3-server-clickhouse-1 clickhouse-client \
  --user cameleer --password cameleer_dev -d cameleer3 \
  -q "SELECT count() FROM route_executions"
```