--- phase: 01-ingestion-pipeline-api-foundation plan: 01 subsystem: database tags: [clickhouse, jdbc, docker-compose, write-buffer, backpressure] requires: - phase: none provides: greenfield project skeleton provides: - ClickHouse Docker Compose for local development - ClickHouse DDL with route_executions, route_diagrams, agent_metrics tables - WriteBuffer generic bounded buffer with backpressure signal - ExecutionRepository, DiagramRepository, MetricsRepository interfaces - IngestionConfig and ClickHouseConfig Spring configuration - Application.yml with datasource, ingestion, springdoc, actuator config affects: [01-02, 01-03, 02-search, 03-agent-registry] tech-stack: added: [clickhouse-jdbc 0.9.7, springdoc-openapi 2.8.6, spring-boot-starter-actuator, spring-boot-starter-jdbc, testcontainers-clickhouse 2.0.2, awaitility, slf4j-api] patterns: [ArrayBlockingQueue write buffer with offer/drain, all-or-nothing offerBatch, content-hash dedup for diagrams, daily partitioning with TTL + ttl_only_drop_parts] key-files: created: - docker-compose.yml - clickhouse/init/01-schema.sql - cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java - cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java - cameleer-server-core/src/main/java/com/cameleer/server/core/storage/DiagramRepository.java - cameleer-server-core/src/main/java/com/cameleer/server/core/storage/MetricsRepository.java - cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/MetricsSnapshot.java - cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java - cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java - cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/WriteBufferTest.java modified: - cameleer-server-core/pom.xml - cameleer-server-app/pom.xml - cameleer-server-app/src/main/resources/application.yml key-decisions: - "Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config rather than manual DataSource" - "Created MetricsSnapshot record in core module since cameleer-common has no metrics model" - "ClickHouseConfig exposes JdbcTemplate bean; relies on Spring Boot DataSource auto-config" patterns-established: - "WriteBuffer pattern: ArrayBlockingQueue with offer()/offerBatch()/drain() for decoupling HTTP from ClickHouse" - "Repository interfaces in core module, implementations will go in app module" - "ClickHouse schema: DateTime64(3, 'UTC') for all timestamps, daily partitioning, TTL with ttl_only_drop_parts" requirements-completed: [INGST-04, INGST-05, INGST-06] duration: 3min completed: 2026-03-11 --- # Phase 1 Plan 01: ClickHouse Infrastructure and WriteBuffer Summary **ClickHouse schema with three tables (daily partitioned, TTL), Docker Compose, WriteBuffer with backpressure, and repository interfaces** ## Performance - **Duration:** 3 min - **Started:** 2026-03-11T10:45:57Z - **Completed:** 2026-03-11T10:49:47Z - **Tasks:** 2 - **Files modified:** 13 ## Accomplishments - ClickHouse DDL with route_executions (MergeTree, bloom_filter + tokenbf_v1 skip indexes), route_diagrams (ReplacingMergeTree), agent_metrics (MergeTree with TTL) - Generic WriteBuffer with all-or-nothing batch semantics and 10 passing unit tests - Repository interfaces defining batch insert contracts for executions, diagrams, and metrics - Full application.yml with datasource, ingestion buffer config, springdoc, and actuator health endpoint ## Task Commits Each task was committed atomically: 1. **Task 1: Dependencies, Docker Compose, ClickHouse schema, and application config** - `96c52b8` (feat) 2. **Task 2 RED: WriteBuffer failing tests** - `f37009e` (test) 3. **Task 2 GREEN: WriteBuffer, repository interfaces, config classes** - `cc1c082` (feat) ## Files Created/Modified - `docker-compose.yml` - ClickHouse service with ports 8123/9000, init volume mount - `clickhouse/init/01-schema.sql` - DDL for route_executions, route_diagrams, agent_metrics - `cameleer-server-core/src/main/java/.../ingestion/WriteBuffer.java` - Bounded queue with offer/offerBatch/drain - `cameleer-server-core/src/main/java/.../storage/ExecutionRepository.java` - Batch insert interface for RouteExecution - `cameleer-server-core/src/main/java/.../storage/DiagramRepository.java` - Store/find interface for RouteGraph - `cameleer-server-core/src/main/java/.../storage/MetricsRepository.java` - Batch insert interface for MetricsSnapshot - `cameleer-server-core/src/main/java/.../storage/model/MetricsSnapshot.java` - Metrics data record - `cameleer-server-app/src/main/java/.../config/IngestionConfig.java` - Buffer capacity, batch size, flush interval - `cameleer-server-app/src/main/java/.../config/ClickHouseConfig.java` - JdbcTemplate bean - `cameleer-server-core/src/test/java/.../ingestion/WriteBufferTest.java` - 10 unit tests for WriteBuffer - `cameleer-server-core/pom.xml` - Added slf4j-api - `cameleer-server-app/pom.xml` - Added clickhouse-jdbc, springdoc, actuator, testcontainers, awaitility - `cameleer-server-app/src/main/resources/application.yml` - Full config with datasource, ingestion, springdoc, actuator ## Decisions Made - Used spring-boot-starter-jdbc to get JdbcTemplate and HikariCP auto-configuration rather than manually wiring a DataSource - Created MetricsSnapshot record in core module since cameleer-common does not include a metrics model - ClickHouseConfig is minimal -- relies on Spring Boot auto-configuring DataSource from spring.datasource properties ## Deviations from Plan None - plan executed exactly as written. ## Issues Encountered None ## User Setup Required None - no external service configuration required. ## Next Phase Readiness - ClickHouse infrastructure ready for Plan 02 (REST controllers + flush scheduler) - WriteBuffer and repository interfaces ready for implementation wiring - Docker Compose available for local development: `docker compose up -d` ## Self-Check: PASSED All 10 created files verified present. All 3 task commits verified in git log. --- *Phase: 01-ingestion-pipeline-api-foundation* *Completed: 2026-03-11*