docs(01-01): complete ClickHouse infrastructure and WriteBuffer plan
- Create 01-01-SUMMARY.md with execution results - Update STATE.md with plan progress and decisions - Update REQUIREMENTS.md marking INGST-04, INGST-05, INGST-06 complete - Update ROADMAP.md with phase 01 progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
hsiegeln
@@ -0,0 +1,125 @@
|
||||
---
|
||||
phase: 01-ingestion-pipeline-api-foundation
|
||||
plan: 01
|
||||
subsystem: database
|
||||
tags: [clickhouse, jdbc, docker-compose, write-buffer, backpressure]
|
||||
|
||||
requires:
|
||||
- phase: none
|
||||
provides: greenfield project skeleton
|
||||
|
||||
provides:
|
||||
- ClickHouse Docker Compose for local development
|
||||
- ClickHouse DDL with route_executions, route_diagrams, agent_metrics tables
|
||||
- WriteBuffer<T> generic bounded buffer with backpressure signal
|
||||
- ExecutionRepository, DiagramRepository, MetricsRepository interfaces
|
||||
- IngestionConfig and ClickHouseConfig Spring configuration
|
||||
- Application.yml with datasource, ingestion, springdoc, actuator config
|
||||
|
||||
affects: [01-02, 01-03, 02-search, 03-agent-registry]
|
||||
|
||||
tech-stack:
|
||||
added: [clickhouse-jdbc 0.9.7, springdoc-openapi 2.8.6, spring-boot-starter-actuator, spring-boot-starter-jdbc, testcontainers-clickhouse 2.0.2, awaitility, slf4j-api]
|
||||
patterns: [ArrayBlockingQueue write buffer with offer/drain, all-or-nothing offerBatch, content-hash dedup for diagrams, daily partitioning with TTL + ttl_only_drop_parts]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- docker-compose.yml
|
||||
- clickhouse/init/01-schema.sql
|
||||
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java
|
||||
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java
|
||||
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java
|
||||
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java
|
||||
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/MetricsSnapshot.java
|
||||
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
|
||||
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java
|
||||
- cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java
|
||||
modified:
|
||||
- cameleer3-server-core/pom.xml
|
||||
- cameleer3-server-app/pom.xml
|
||||
- cameleer3-server-app/src/main/resources/application.yml
|
||||
|
||||
key-decisions:
|
||||
- "Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config rather than manual DataSource"
|
||||
- "Created MetricsSnapshot record in core module since cameleer3-common has no metrics model"
|
||||
- "ClickHouseConfig exposes JdbcTemplate bean; relies on Spring Boot DataSource auto-config"
|
||||
|
||||
patterns-established:
|
||||
- "WriteBuffer pattern: ArrayBlockingQueue with offer()/offerBatch()/drain() for decoupling HTTP from ClickHouse"
|
||||
- "Repository interfaces in core module, implementations will go in app module"
|
||||
- "ClickHouse schema: DateTime64(3, 'UTC') for all timestamps, daily partitioning, TTL with ttl_only_drop_parts"
|
||||
|
||||
requirements-completed: [INGST-04, INGST-05, INGST-06]
|
||||
|
||||
duration: 3min
|
||||
completed: 2026-03-11
|
||||
---
|
||||
|
||||
# Phase 1 Plan 01: ClickHouse Infrastructure and WriteBuffer Summary
|
||||
|
||||
**ClickHouse schema with three tables (daily partitioned, TTL), Docker Compose, WriteBuffer with backpressure, and repository interfaces**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 3 min
|
||||
- **Started:** 2026-03-11T10:45:57Z
|
||||
- **Completed:** 2026-03-11T10:49:47Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 13
|
||||
|
||||
## Accomplishments
|
||||
- ClickHouse DDL with route_executions (MergeTree, bloom_filter + tokenbf_v1 skip indexes), route_diagrams (ReplacingMergeTree), agent_metrics (MergeTree with TTL)
|
||||
- Generic WriteBuffer<T> with all-or-nothing batch semantics and 10 passing unit tests
|
||||
- Repository interfaces defining batch insert contracts for executions, diagrams, and metrics
|
||||
- Full application.yml with datasource, ingestion buffer config, springdoc, and actuator health endpoint
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Dependencies, Docker Compose, ClickHouse schema, and application config** - `96c52b8` (feat)
|
||||
2. **Task 2 RED: WriteBuffer failing tests** - `f37009e` (test)
|
||||
3. **Task 2 GREEN: WriteBuffer, repository interfaces, config classes** - `cc1c082` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
- `docker-compose.yml` - ClickHouse service with ports 8123/9000, init volume mount
|
||||
- `clickhouse/init/01-schema.sql` - DDL for route_executions, route_diagrams, agent_metrics
|
||||
- `cameleer3-server-core/src/main/java/.../ingestion/WriteBuffer.java` - Bounded queue with offer/offerBatch/drain
|
||||
- `cameleer3-server-core/src/main/java/.../storage/ExecutionRepository.java` - Batch insert interface for RouteExecution
|
||||
- `cameleer3-server-core/src/main/java/.../storage/DiagramRepository.java` - Store/find interface for RouteGraph
|
||||
- `cameleer3-server-core/src/main/java/.../storage/MetricsRepository.java` - Batch insert interface for MetricsSnapshot
|
||||
- `cameleer3-server-core/src/main/java/.../storage/model/MetricsSnapshot.java` - Metrics data record
|
||||
- `cameleer3-server-app/src/main/java/.../config/IngestionConfig.java` - Buffer capacity, batch size, flush interval
|
||||
- `cameleer3-server-app/src/main/java/.../config/ClickHouseConfig.java` - JdbcTemplate bean
|
||||
- `cameleer3-server-core/src/test/java/.../ingestion/WriteBufferTest.java` - 10 unit tests for WriteBuffer
|
||||
- `cameleer3-server-core/pom.xml` - Added slf4j-api
|
||||
- `cameleer3-server-app/pom.xml` - Added clickhouse-jdbc, springdoc, actuator, testcontainers, awaitility
|
||||
- `cameleer3-server-app/src/main/resources/application.yml` - Full config with datasource, ingestion, springdoc, actuator
|
||||
|
||||
## Decisions Made
|
||||
- Used spring-boot-starter-jdbc to get JdbcTemplate and HikariCP auto-configuration rather than manually wiring a DataSource
|
||||
- Created MetricsSnapshot record in core module since cameleer3-common does not include a metrics model
|
||||
- ClickHouseConfig is minimal -- relies on Spring Boot auto-configuring DataSource from spring.datasource properties
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written.
|
||||
|
||||
## Issues Encountered
|
||||
None
|
||||
|
||||
## User Setup Required
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
- ClickHouse infrastructure ready for Plan 02 (REST controllers + flush scheduler)
|
||||
- WriteBuffer and repository interfaces ready for implementation wiring
|
||||
- Docker Compose available for local development: `docker compose up -d`
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
All 10 created files verified present. All 3 task commits verified in git log.
|
||||
|
||||
---
|
||||
*Phase: 01-ingestion-pipeline-api-foundation*
|
||||
*Completed: 2026-03-11*
|
||||
Reference in New Issue
Block a user