# ClickHouse Phase 4: Remaining Tables — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Migrate route diagrams, agent events, and application logs from PostgreSQL/OpenSearch to ClickHouse.
**Architecture:** Three new ClickHouse stores implement existing interfaces. `ClickHouseDiagramStore` uses ReplacingMergeTree for content-hash dedup. `ClickHouseAgentEventRepository` uses MergeTree for append-only events. `ClickHouseLogStore` replaces `OpenSearchLogIndex` with SQL + ngram indexes. Feature flags control each store independently.
Implements `DiagramStore` interface (5 methods). Read `PostgresDiagramStore.java` first and translate.
**Key differences from PG:**
-`INSERT INTO ... ON CONFLICT DO NOTHING` → just `INSERT INTO` (ReplacingMergeTree deduplicates by content_hash)
-`?::jsonb` → plain `?` (CH stores definition as String, not JSONB)
-`ORDER BY created_at DESC LIMIT 1` → `ORDER BY created_at DESC LIMIT 1` (same, but add `FINAL` for ReplacingMergeTree reads)
-`findProcessorRouteMapping`: PG uses `jsonb_array_elements()` — CH has no native JSON array functions. Instead, store the definition as a string and parse in Java, OR query `route_diagrams FINAL` and deserialize definitions application-side. **Recommended:** Fetch all definitions for the application, deserialize in Java, extract processor→route mappings. This is a small result set (one row per route).
- SHA-256 content hash computation stays in Java (same as PG store)
- Add `WHERE tenant_id = 'default'` to all queries
**Tests:**
-`store_insertsNewDiagram`
-`store_duplicateHashIgnored` (ReplacingMergeTree dedup after OPTIMIZE FINAL)
-`StorageBeanConfig.java` — add CH diagram, event, and log store beans (all default to clickhouse with `matchIfMissing = true`)
-`application.yml` — add `diagrams`, `events`, `logs` flags under `cameleer.storage`
-`deploy/base/server.yaml` — add env vars
**Feature flags (all default to clickhouse):**
```yaml
cameleer:
storage:
metrics: ${CAMELEER_STORAGE_METRICS:postgres}
search: ${CAMELEER_STORAGE_SEARCH:opensearch}
stats: ${CAMELEER_STORAGE_STATS:clickhouse}
diagrams: ${CAMELEER_STORAGE_DIAGRAMS:clickhouse}
events: ${CAMELEER_STORAGE_EVENTS:clickhouse}
logs: ${CAMELEER_STORAGE_LOGS:clickhouse}
```
**Important for LogStore wiring:** The `OpenSearchLogIndex` is a `@Repository` used directly by controllers (not via an interface). The `ClickHouseLogStore` must be injectable in the same way. Options:
- Extract a `LogIndex` interface with `search()` + `indexBatch()` methods, used by both controllers
- Or make `ClickHouseLogStore` extend/implement the same type
**Recommended:** Create a `LogIndex` interface in the core module with the two methods, have both `OpenSearchLogIndex` and `ClickHouseLogStore` implement it, and update `LogIngestionController` + `LogQueryController` to inject `LogIndex` instead of `OpenSearchLogIndex`.
---
## Verification Checklist
1.**Diagrams**: Store + retrieve RouteGraph via ClickHouse, verify content-hash dedup
2.**Events**: Insert + query events with time range and app/agent filters
3.**Logs**: Batch insert + search with all filter types (level, query, exchangeId, time range)
4.**Feature flags**: Each store independently switchable between PG/OS and CH
5.**Backward compat**: Default config uses ClickHouse for all Phase 4 stores
