cameleer-server/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
01-ingestion-pipeline-api-foundation	01	execute	1		pom.xml cameleer3-server-core/pom.xml cameleer3-server-app/pom.xml docker-compose.yml clickhouse/init/01-schema.sql cameleer3-server-app/src/main/resources/application.yml cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java cameleer3-server-app/src/test/resources/application-test.yml cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java	true	INGST-04 INGST-05 INGST-06	truths artifacts key_links WriteBuffer accepts items and returns false when full (backpressure signal) WriteBuffer drains items in batches for scheduled flush ClickHouse schema creates route_executions, route_diagrams, and agent_metrics tables with correct column types TTL clause on tables removes data older than configured days Docker Compose starts ClickHouse and initializes the schema path provides min_lines cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java Generic bounded write buffer with offer/drain/isFull 30 path provides contains clickhouse/init/01-schema.sql ClickHouse DDL for all three tables CREATE TABLE route_executions path provides contains docker-compose.yml Local ClickHouse service clickhouse-server path provides exports cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java Repository interface for execution batch inserts insertBatch path provides min_lines cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java Shared Testcontainers base class for integration tests 20 from to via pattern cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java application.yml spring.datasource properties spring.datasource from to via pattern cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java application.yml ingestion.* properties ingestion.

Set up ClickHouse infrastructure, schema, WriteBuffer with backpressure, repository interfaces, and test infrastructure.

Purpose: Establishes the storage foundation that all ingestion endpoints and future search queries depend on. The WriteBuffer is the central throughput mechanism -- all data flows through it before reaching ClickHouse. Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with unit tests, repository interfaces, Testcontainers base class for integration tests.

<execution_context> @C:/Users/Hendrik/.claude/get-shit-done/workflows/execute-plan.md @C:/Users/Hendrik/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/01-ingestion-pipeline-api-foundation/01-RESEARCH.md

@pom.xml @cameleer3-server-core/pom.xml @cameleer3-server-app/pom.xml

Task 1: Dependencies, Docker Compose, ClickHouse schema, and application config pom.xml, cameleer3-server-core/pom.xml, cameleer3-server-app/pom.xml, docker-compose.yml, clickhouse/init/01-schema.sql, cameleer3-server-app/src/main/resources/application.yml - docker compose up -d starts ClickHouse on ports 8123/9000 - Connecting to ClickHouse and running SELECT 1 succeeds - Tables route_executions, route_diagrams, agent_metrics exist after init - route_executions has TTL clause with configurable interval - route_executions has PARTITION BY toYYYYMMDD(start_time) and ORDER BY (agent_id, status, start_time, execution_id) - route_diagrams uses ReplacingMergeTree with ORDER BY (content_hash) - agent_metrics has TTL and daily partitioning - Maven compile succeeds with new dependencies 1. Add dependencies to cameleer3-server-app/pom.xml per research: - clickhouse-jdbc 0.9.7 (classifier: all) - spring-boot-starter-actuator - springdoc-openapi-starter-webmvc-ui 2.8.6 - testcontainers-clickhouse 2.0.2 (test scope) - junit-jupiter from testcontainers 2.0.2 (test scope) - awaitility (test scope)

2. Add slf4j-api dependency to cameleer3-server-core/pom.xml.

3. Create docker-compose.yml at project root with ClickHouse service:
   - Image: clickhouse/clickhouse-server:25.3
   - Ports: 8123:8123, 9000:9000
   - Volume mount ./clickhouse/init to /docker-entrypoint-initdb.d
   - Environment: CLICKHOUSE_USER=cameleer, CLICKHOUSE_PASSWORD=cameleer_dev, CLICKHOUSE_DB=cameleer3
   - ulimits nofile 262144

4. Create clickhouse/init/01-schema.sql with the three tables from research:
   - route_executions: MergeTree, daily partitioning on start_time, ORDER BY (agent_id, status, start_time, execution_id), TTL start_time + INTERVAL 30 DAY, SETTINGS ttl_only_drop_parts=1. Include Array columns for processor executions (processor_ids, processor_types, processor_starts, processor_ends, processor_durations, processor_statuses). Include skip indexes for correlation_id (bloom_filter) and error_message (tokenbf_v1).
   - route_diagrams: ReplacingMergeTree(created_at), ORDER BY (content_hash). No TTL.
   - agent_metrics: MergeTree, daily partitioning on collected_at, ORDER BY (agent_id, metric_name, collected_at), TTL collected_at + INTERVAL 30 DAY, SETTINGS ttl_only_drop_parts=1.
   - All DateTime fields use DateTime64(3, 'UTC').

5. Create cameleer3-server-app/src/main/resources/application.yml with config from research:
   - server.port: 8081
   - spring.datasource: url=jdbc:ch://localhost:8123/cameleer3, username/password, driver-class-name
   - spring.jackson: write-dates-as-timestamps=false, fail-on-unknown-properties=false
   - ingestion: buffer-capacity=50000, batch-size=5000, flush-interval-ms=1000
   - clickhouse.ttl-days: 30
   - springdoc paths under /api/v1/
   - management endpoints (health under /api/v1/, show-details=always)

6. Ensure .gitattributes exists with `* text=auto eol=lf`.

mvn clean compile -q 2>&1 | tail -5 Maven compiles successfully with all new dependencies. Docker Compose file and ClickHouse DDL exist. application.yml configures datasource, ingestion buffer, and springdoc. Task 2: WriteBuffer, repository interfaces, IngestionConfig, ClickHouseConfig, and test infrastructure cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java, cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java, cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java, cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java, cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java, cameleer3-server-app/src/test/resources/application-test.yml, cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java - WriteBuffer(capacity=10): offer() returns true for first 10 items, false on 11th - WriteBuffer.drain(5) returns up to 5 items and removes them from the queue - WriteBuffer.isFull() returns true when at capacity - WriteBuffer.offerBatch(list) returns false without partial insert if buffer would overflow - WriteBuffer.size() tracks current queue depth - ExecutionRepository interface declares insertBatch(List of RouteExecution) - DiagramRepository interface declares store(RouteGraph) and findByContentHash(String) - MetricsRepository interface declares insertBatch(List of metric data) - AbstractClickHouseIT starts a Testcontainers ClickHouse, applies schema, provides JdbcTemplate 1. Create WriteBuffer in core module (no Spring dependency): - Constructor takes int capacity, creates ArrayBlockingQueue(capacity) - offer(T item): returns queue.offer(item) -- false when full - offerBatch(List items): check remainingCapacity() >= items.size() first, then offer each. If insufficient capacity, return false immediately without adding any items. - drain(int maxBatch): drainTo into ArrayList, return list - size(), capacity(), isFull(), remainingCapacity() accessors

2. Create WriteBufferTest (JUnit 5, no Spring):
   - Test offer succeeds until capacity
   - Test offer returns false when full
   - Test offerBatch all-or-nothing semantics
   - Test drain returns items and removes from queue
   - Test drain with empty queue returns empty list
   - Test isFull/size/remainingCapacity

3. Create repository interfaces in core module:
   - ExecutionRepository: void insertBatch(List<RouteExecution> executions)
   - DiagramRepository: void store(RouteGraph graph), Optional<RouteGraph> findByContentHash(String hash), Optional<String> findContentHashForRoute(String routeId, String agentId)
   - MetricsRepository: void insertBatch(List<MetricsSnapshot> metrics) -- use a generic type or the cameleer3-common metrics model if available; if not, create a simple MetricsData record in core module

4. Create IngestionConfig as @ConfigurationProperties("ingestion"):
   - bufferCapacity (int, default 50000)
   - batchSize (int, default 5000)
   - flushIntervalMs (long, default 1000)

5. Create ClickHouseConfig as @Configuration:
   - Exposes JdbcTemplate bean (Spring Boot auto-configures DataSource from spring.datasource)
   - No custom bean needed if relying on auto-config; only create if explicit JdbcTemplate customization required

6. Create application-test.yml for test profile:
   - Placeholder datasource config (overridden by Testcontainers in AbstractClickHouseIT)
   - ingestion: small buffer for tests (capacity=100, batch-size=10, flush-interval-ms=100)

7. Create AbstractClickHouseIT base class:
   - @Testcontainers + @Container with ClickHouseContainer("clickhouse/clickhouse-server:25.3")
   - @DynamicPropertySource to override spring.datasource.url/username/password
   - @SpringBootTest
   - @ActiveProfiles("test")
   - @BeforeAll: read clickhouse/init/01-schema.sql and execute it against the container via JDBC
   - Expose protected JdbcTemplate for subclasses

mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q 2>&1 | tail -10 WriteBuffer passes all unit tests. Repository interfaces exist with correct method signatures. IngestionConfig reads from application.yml. AbstractClickHouseIT base class exists and can start a Testcontainers ClickHouse instance with schema applied. - `mvn test -pl cameleer3-server-core -q` -- all WriteBuffer unit tests pass - `mvn clean compile -q` -- full project compiles with new dependencies - `docker compose config` -- validates Docker Compose file - clickhouse/init/01-schema.sql contains CREATE TABLE for all three tables with correct ENGINE, ORDER BY, PARTITION BY, and TTL

<success_criteria> WriteBuffer unit tests green. Project compiles. ClickHouse DDL defines all three tables with TTL and correct partitioning. Repository interfaces define batch insert contracts. Testcontainers base class ready for integration tests in Plan 02. </success_criteria>

After completion, create `.planning/phases/01-ingestion-pipeline-api-foundation/01-01-SUMMARY.md`