cameleer-server/.planning/research/STACK.md

# Technology Stack

**Project:** Cameleer Server
**Researched:** 2026-03-11
**Overall confidence:** MEDIUM (no live source verification available; versions based on training data up to May 2025)

## Recommended Stack

### Core Framework (Already Decided)

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| Java | 17+ | Runtime | Already established; LTS, well-supported | HIGH |
| Spring Boot | 3.4.3 | Application framework | Already in POM; provides web, security, configuration | HIGH |
| Maven | 3.9+ | Build system | Already established; multi-module project | HIGH |

### Primary Data Store: ClickHouse

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| ClickHouse | 24.x+ | Transaction/activity storage | Column-oriented, built for billions of rows, native TTL, excellent time-range queries, MergeTree engine handles millions of inserts/day trivially | MEDIUM |
| clickhouse-java (HTTP) | 0.6.x+ | Java client | Official ClickHouse Java client; HTTP transport is simpler and more reliable than native TCP for Spring Boot apps | MEDIUM |

**Why ClickHouse over alternatives:**

- **vs Elasticsearch/OpenSearch:** ClickHouse is 5-10x more storage-efficient for structured columnar data. For time-series-like transaction data with known schema, ClickHouse drastically outperforms ES on aggregation queries (avg duration, count by state, time bucketing). ES is overkill when you don't need its inverted index for *every* field.
- **vs TimescaleDB:** TimescaleDB is PostgreSQL-based and good for moderate scale, but ClickHouse handles the "millions of inserts per day" tier with less operational overhead. TimescaleDB's row-oriented heritage means larger storage footprint for wide transaction records. ClickHouse's columnar compression achieves 10-20x compression on typical observability data.
- **vs PostgreSQL (plain):** PostgreSQL cannot efficiently handle this insert volume with 30-day retention and fast analytical queries. Partitioning and vacuuming become operational nightmares at this scale.

**ClickHouse key features for this project:**
- **TTL on tables:** `TTL executionDate + INTERVAL 30 DAY` — automatic 30-day retention with zero application code
- **MergeTree engine:** Handles high insert throughput; batch inserts of 10K+ rows are trivial
- **Materialized views:** Pre-aggregate common queries (transactions by state per hour, etc.)
- **Low storage cost:** 10-20x compression means 30 days of millions of transactions fits in modest disk

### Full-Text Search: OpenSearch

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| OpenSearch | 2.x | Full-text search over payloads, metadata, attributes | True inverted index for arbitrary text search; ClickHouse's full-text is rudimentary | MEDIUM |
| opensearch-java | 2.x | Java client | Official OpenSearch Java client; works well with Spring Boot | MEDIUM |

**Why a separate search engine instead of ClickHouse alone:**

ClickHouse has token-level bloom filter indexes and `hasToken()`/`LIKE` matching, but these are not true full-text search. For the requirement "search by any content in payloads, metadata, and attributes," you need an inverted index with:
- Tokenization and analysis (stemming, case folding)
- Relevance scoring
- Phrase matching
- Highlighting of matched terms in results

**Why OpenSearch over Elasticsearch:**
- Apache 2.0 licensed (no SSPL concerns for self-hosted deployment)
- API-compatible with Elasticsearch 7.x
- Active development, large community
- OpenSearch Dashboards available if needed later
- No licensing ambiguity for Docker deployment

**Dual-store pattern:**
- ClickHouse = source of truth for structured queries (time range, state, duration, aggregations)
- OpenSearch = search index for full-text queries
- Application writes to both; OpenSearch indexed asynchronously from an internal queue
- Structured filters (time, state) applied in ClickHouse; full-text queries in OpenSearch return transaction IDs, then ClickHouse fetches full records

### Caching Layer: Caffeine + Redis (phased)

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| Caffeine | 3.1.x | In-process cache for agent registry, diagram versions, hot config | Fastest JVM cache; zero network overhead; perfect for single-instance start | MEDIUM |
| Spring Cache (`@Cacheable`) | (Spring Boot) | Cache abstraction | Switch cache backends without code changes | HIGH |
| Redis | 7.x | Distributed cache (Phase 2+, when horizontal scaling) | Shared state across multiple server instances; SSE session coordination | MEDIUM |

**Phased approach:**
1. **Phase 1:** Caffeine only. Single server instance. Agent registry, diagram cache, recent query results all in-process.
2. **Phase 2 (horizontal scaling):** Add Redis for shared state. Agent registry must be consistent across instances. SSE sessions need coordination.

### Message Ingestion: Internal Buffer with Backpressure

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| LMAX Disruptor | 4.0.x | High-performance ring buffer for ingestion | Lock-free, single-writer principle, handles burst traffic without blocking HTTP threads | MEDIUM |
| *Alternative:* `java.util.concurrent.LinkedBlockingQueue` | (JDK) | Simpler bounded queue | Good enough for initial implementation; switch to Disruptor if profiling shows contention | HIGH |

**Why an internal buffer, not Kafka:**

Kafka is the standard answer for "high-volume ingestion," but it adds massive operational complexity for a system that:
- Has a single data producer type (Cameleer agents via HTTP POST)
- Does not need replay from an external topic
- Does not need multi-consumer fan-out
- Is already receiving data via HTTP (not streaming)

The right pattern here: **HTTP POST -> bounded in-memory queue -> batch writer to ClickHouse + async indexer to OpenSearch**. If the queue fills up, return HTTP 503 with `Retry-After` header — agents should implement exponential backoff.

**When to add Kafka:** Only if you need cross-datacenter replication, multi-consumer processing, or guaranteed exactly-once delivery beyond what the internal buffer provides. This is a "maybe Phase 3+" decision.

### API Documentation: springdoc-openapi

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| springdoc-openapi-starter-webmvc-ui | 2.x | OpenAPI 3.1 spec generation + Swagger UI | De facto standard for Spring Boot 3.x API docs; annotation-driven, zero-config for basic setup | MEDIUM |

**Why springdoc over alternatives:**
- **vs SpringFox:** SpringFox is effectively dead; no Spring Boot 3 support
- **vs manual OpenAPI:** Too much maintenance overhead; springdoc generates from code
- springdoc supports Spring Boot 3.x natively, including Spring Security integration

### Security

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| Spring Security | (Spring Boot 3.4.3) | Authentication/authorization framework | Already part of Spring Boot; JWT filter chain, method security | HIGH |
| java-jwt (Auth0) | 4.x | JWT creation and validation | Lightweight, well-maintained; simpler than Nimbus for this use case | MEDIUM |
| Ed25519 (JDK `java.security`) | (JDK 17) | Config signing | JDK 15+ has native EdDSA support; no external library needed | HIGH |

### Testing

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| JUnit 5 | (Spring Boot) | Unit/integration testing | Already in POM; standard | HIGH |
| Testcontainers | 1.19.x+ | Integration tests with ClickHouse and OpenSearch | Spin up real databases in Docker for tests; no mocking storage layer | MEDIUM |
| Spring Boot Test | (Spring Boot) | Controller/integration testing | `@SpringBootTest`, `MockMvc`, etc. | HIGH |
| Awaitility | 4.2.x | Async testing (SSE, queue processing) | Clean API for testing eventually-consistent behavior | MEDIUM |

### Containerization

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| Docker | - | Container runtime | Required per project constraints | HIGH |
| Docker Compose | - | Local dev + simple deployment | Single command to run server + ClickHouse + OpenSearch + Redis | HIGH |
| Eclipse Temurin JDK 17 | - | Base image | Official OpenJDK distribution; `eclipse-temurin:17-jre-alpine` for small image | HIGH |

### Monitoring (Server Self-Observability)

| Technology | Version | Purpose | Why | Confidence |
|------------|---------|---------|-----|------------|
| Micrometer | (Spring Boot) | Metrics facade | Built into Spring Boot; exposes ingestion rates, queue depth, query latencies | HIGH |
| Spring Boot Actuator | (Spring Boot) | Health checks, metrics endpoint | `/actuator/health` for Docker health checks, `/actuator/prometheus` for metrics | HIGH |

## Supporting Libraries

| Library | Version | Purpose | When to Use | Confidence |
|---------|---------|---------|-------------|------------|
| MapStruct | 1.5.x | DTO <-> entity mapping | Compile-time mapping; avoids reflection overhead in hot path | MEDIUM |
| Jackson JavaTimeModule | (already used) | `Instant` serialization | Already in project for `java.time` types | HIGH |
| SLF4J + Logback | (Spring Boot) | Logging | Default Spring Boot logging; structured JSON logging for production | HIGH |

## What NOT to Use

| Technology | Why Not |
|------------|---------|
| Elasticsearch | SSPL license; OpenSearch is API-compatible and Apache 2.0 |
| Kafka | Massive operational overhead for a system with single producer type; internal buffer is sufficient initially |
| MongoDB | Poor fit for time-series analytical queries; no native TTL with the efficiency of ClickHouse's MergeTree |
| PostgreSQL (as primary) | Cannot handle millions of inserts/day with fast analytical queries at 30-day retention |
| SpringFox | Dead project; no Spring Boot 3 support |
| Hibernate/JPA | ClickHouse is not a relational DB; JPA adds friction with no benefit. Use ClickHouse Java client directly. |
| Lombok | Controversial; Java 17 records cover most use cases; explicit code is clearer |
| gRPC | Agents already use HTTP POST; adding gRPC doubles protocol complexity for marginal throughput gain |

## Alternatives Considered

| Category | Recommended | Alternative | Why Not Alternative |
|----------|-------------|-------------|---------------------|
| Primary store | ClickHouse | TimescaleDB | Row-oriented heritage; larger storage footprint; less efficient for wide analytical queries |
| Primary store | ClickHouse | PostgreSQL + partitioning | Vacuum overhead; partition management; slower aggregations |
| Search | OpenSearch | Elasticsearch | SSPL license risk; functionally equivalent |
| Search | OpenSearch | ClickHouse full-text indexes | Not true full-text search; no relevance scoring, no phrase matching |
| Ingestion buffer | Internal queue | Apache Kafka | Operational complexity not justified; single producer type |
| Cache | Caffeine | Guava Cache | Caffeine is successor to Guava Cache with better performance |
| API docs | springdoc-openapi | SpringFox | SpringFox has no Spring Boot 3 support |
| JWT | java-jwt (Auth0) | Nimbus JOSE+JWT | Nimbus is more complex; java-jwt sufficient for symmetric/asymmetric JWT |

## Installation (Maven Dependencies)

```xml
<!-- ClickHouse -->
<dependency>
    <groupId>com.clickhouse</groupId>
    <artifactId>clickhouse-http-client</artifactId>
    <version>0.6.5</version> <!-- verify latest -->
</dependency>

<!-- OpenSearch -->
<dependency>
    <groupId>org.opensearch.client</groupId>
    <artifactId>opensearch-java</artifactId>
    <version>2.13.0</version> <!-- verify latest -->
</dependency>

<!-- Caffeine Cache -->
<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
    <!-- version managed by Spring Boot -->
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

<!-- Security -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
    <groupId>com.auth0</groupId>
    <artifactId>java-jwt</artifactId>
    <version>4.4.0</version> <!-- verify latest -->
</dependency>

<!-- API Documentation -->
<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.6.0</version> <!-- verify latest -->
</dependency>

<!-- Monitoring -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

<!-- Testing -->
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>clickhouse</artifactId>
    <version>1.19.8</version> <!-- verify latest -->
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>1.19.8</version> <!-- verify latest -->
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.awaitility</groupId>
    <artifactId>awaitility</artifactId>
    <scope>test</scope>
    <!-- version managed by Spring Boot -->
</dependency>

<!-- Mapping -->
<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct</artifactId>
    <version>1.5.5.Final</version> <!-- verify latest -->
</dependency>
```

## Version Verification Notes

All version numbers are from training data (up to May 2025). Before adding dependencies, verify the latest stable versions on Maven Central:
- `clickhouse-http-client`: check https://github.com/ClickHouse/clickhouse-java/releases
- `opensearch-java`: check https://github.com/opensearch-project/opensearch-java/releases
- `springdoc-openapi-starter-webmvc-ui`: check https://springdoc.org/
- `java-jwt`: check https://github.com/auth0/java-jwt/releases
- `testcontainers`: check https://github.com/testcontainers/testcontainers-java/releases

## Sources

- Training data knowledge (ClickHouse architecture, OpenSearch capabilities, Spring Boot ecosystem)
- Project POM analysis (Spring Boot 3.4.3, Jackson 2.17.3, existing module structure)
- CLAUDE.md project instructions (ClickHouse mentioned as storage target, JWT/Ed25519 security model)

**Note:** All external source verification was unavailable during this research session. Version numbers should be validated before implementation.