Cameleer Server is a write-heavy, read-occasional observability system that receives millions of transaction records per day from distributed Apache Camel agents, stores them with 30-day retention, and provides structured + full-text search. The architecture closely parallels established observability platforms like Jaeger, Zipkin, and njams Server, with the key differentiator being Camel route diagram visualization tied to individual transactions.
The recommended stack centers on **ClickHouse** as the primary data store. ClickHouse's columnar MergeTree engine provides the exact properties this project needs: massive batch insert throughput, excellent time-range query performance, native TTL-based retention, and 10-20x compression on structured observability data. This is a well-established pattern used by production observability platforms (SigNoz, Uptrace, PostHog all run on ClickHouse).
For full-text search, the recommendation is a **phased approach**: start with ClickHouse's built-in token bloom filter skip indexes (`tokenbf_v1`), which handle exact-token search (correlation IDs, error messages, specific values) well enough for MVP. When query patterns demand fuzzy matching or relevance scoring, add **OpenSearch** as a secondary search index. The architecture should be designed from the start to allow this swap transparently via the repository abstraction in the core module.
The critical architectural pattern is an **in-memory write buffer** between the HTTP ingestion endpoint and ClickHouse. ClickHouse performs best with batch inserts of 1K-10K rows; individual row inserts are the single most common and most damaging mistake when building on ClickHouse. The buffer also provides the backpressure mechanism (HTTP 429) that prevents the server from being overwhelmed during agent reconnection storms.
The two-module structure (core for domain logic + interfaces, app for Spring Boot wiring + implementations) enforces clean boundaries. Core defines repository interfaces, service implementations, and the write buffer. App provides ClickHouse repository implementations, Spring SseEmitter integration, REST controllers, and security filters. The boundary rule is strict: app depends on core, never the reverse.
## Key Findings
**Stack:** Java 17 / Spring Boot 3.4.3 + ClickHouse (primary store) + ClickHouse skip indexes for text search (phase 1), OpenSearch optional (phase 2+) + Caffeine cache + springdoc-openapi + Auth0 java-jwt. No Kafka, no Elasticsearch, no JPA.
**Architecture:** Write-heavy CQRS-lite with three data paths: (1) buffered ingestion pipeline to ClickHouse, (2) query engine combining structured ClickHouse queries with text search, (3) SSE connection registry for agent push. Repository abstraction keeps core module storage-agnostic. Content-addressable diagram versioning with async pre-rendering.
**Critical pitfall:** Row-by-row ClickHouse inserts and wrong ORDER BY design. These two mistakes together will make the system fail within hours under load and cannot be fixed without table recreation. Batch buffering and schema design must be correct from the first implementation.
- Phase 2 (Query): Standard patterns, unlikely to need research
- Phase 5 (Security): Standard patterns, unlikely to need research
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
| Stack (ClickHouse choice) | HIGH | Well-established pattern for observability; used by SigNoz, Uptrace, PostHog |
| Stack (version numbers) | LOW | Could not verify against live sources; all versions from training data (May 2025 cutoff) |
| Features | MEDIUM | Based on domain knowledge of njams, Jaeger, Zipkin; could not verify latest feature trends |
| Architecture | MEDIUM | Patterns are well-established; batch buffer, SSE registry, content-addressable storage are standard |
| Pitfalls | HIGH | ClickHouse pitfalls are well-documented; SSE lifecycle issues are common; ingestion backpressure is standard |
| Full-text search approach | MEDIUM | ClickHouse skip indexes vs OpenSearch is a legitimate decision point that needs hands-on evaluation |
## Gaps to Address
- **ClickHouse Java client API:** The clickhouse-java library has undergone significant changes. Exact API, connection pooling, and Spring Boot integration patterns need phase-specific research
- **cameleer-common PROTOCOL.md:** Must read the agent protocol definition before designing ClickHouse schema -- this defines the exact data structures being ingested
- **ClickHouse Docker setup:** Optimal ClickHouse Docker configuration (memory limits, merge settings) for development and production
- **Full-text search decision:** ClickHouse skip indexes may or may not meet the "search by any content" requirement. This needs prototyping with realistic data
- **Diagram rendering library:** Server-side route diagram rendering is a significant unknown; needs prototyping with actual Camel route graph data from cameleer-common
