cameleer-server/.planning/PROJECT.md

# Cameleer Server

## What This Is

An observability server that receives, stores, and serves Apache Camel route execution data from distributed Cameleer agents. Think njams Server (by Integration Matters) — but built incrementally, API-first, with a modern stack. Users can search through millions of recorded transactions by state, time, duration, full text, and correlate executions across multiple Camel instances. The server also pushes configuration, tracing controls, and ad-hoc commands to agents via SSE.

## Core Value

Users can reliably search and find any transaction across all connected Camel instances — by any combination of state, time, duration, or content — even at millions of transactions per day with 30-day retention.

## Requirements

### Validated

(None yet — ship to validate)

### Active

- [ ] Receive and ingest transaction/activity data from Cameleer agents via HTTP POST
- [ ] Store transactions in a high-volume, horizontally scalable data store with 30-day retention
- [ ] Search transactions by state, execution date/time, duration, and full-text content
- [ ] Correlate activities across multiple routes and Camel instances within a single transaction
- [ ] Store and version route diagrams so each transaction links to the diagram active at that time
- [ ] Render route diagrams server-side from stored definitions
- [ ] Maintain agent instance registry with lifecycle states (LIVE → STALE → DEAD)
- [ ] Push configuration updates, tracing controls, and ad-hoc commands to agents via SSE
- [ ] Expose a clean REST API that serves as the sole interface for both the UI and external consumers
- [ ] Build a web UI that consumes only the REST API
- [ ] Secure agent-server communication with JWT auth, Ed25519 config signing, and bootstrap token registration
- [ ] Support containerized deployment (Docker)

### Out of Scope

- Mobile app — web UI is sufficient for ops/dev users
- Log aggregation — this is transaction-level observability, not a log collector
- APM/metrics dashboards — focused on Camel route transactions, not general application metrics
- Multi-tenancy — single-tenant deployment per environment for now

## Context

- **Agent side**: cameleer agent (`https://gitea.siegeln.net/cameleer/cameleer`) is under active development; already supports creating diagrams and capturing executions
- **Shared library**: `com.cameleer:cameleer-common` contains shared models and the graph API; protocol defined in `cameleer-common/PROTOCOL.md`
- **Data model**: Hierarchical — a **transaction** represents a message's full journey, containing **activities** per route execution. Transactions can span multiple Camel instances (e.g., route A calls route B on another instance via endpoint)
- **Scale target**: Millions of transactions per day, 50+ connected agents, 30-day data retention
- **Query pattern**: Incident-driven — mostly recent data queries, with deep historical dives during incidents
- **Inspiration**: njams Server by Integration Matters — similar domain, building our own step by step
- **Existing scaffolding**: Maven multi-module project with Spring Boot 3.4.3, two modules (core + app), Gitea CI pipeline

## Constraints

- **Tech stack**: Java 17+, Spring Boot 3.4.3, Maven multi-module — already established
- **Dependency**: Must consume `com.cameleer:cameleer-common` from Gitea Maven registry
- **Protocol**: Agent protocol is still evolving — server must adapt as it stabilizes
- **Incremental delivery**: Build step by step; storage and search first, then layer features on top

## Key Decisions

| Decision | Rationale | Outcome |
|----------|-----------|---------|
| API-first architecture | REST API is the sole interface; UI and external consumers use the same API | — Pending |
| Storage engine selection | Must handle millions of tx/day, full-text search, time-series queries, 30-day TTL, horizontal scaling | — Pending |
| Versioned diagram storage | Each transaction references the route diagram definition active at execution time | — Pending |
| SSE for agent push | Server pushes config, tracing control, and commands to agents via Server-Sent Events | — Pending |
| JWT + Ed25519 auth model | Secure bidirectional communication; bootstrap token for initial agent registration | — Pending |

---
*Last updated: 2026-03-11 after initialization*