diff --git a/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md b/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md index 43dc0e2c..69762716 100644 --- a/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md +++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md @@ -18,8 +18,6 @@ files_modified: - cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java - cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java - cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java - - cameleer3-server-app/src/test/resources/application-test.yml - - cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java autonomous: true requirements: - INGST-04 @@ -46,9 +44,6 @@ must_haves: - path: "cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java" provides: "Repository interface for execution batch inserts" exports: ["insertBatch"] - - path: "cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java" - provides: "Shared Testcontainers base class for integration tests" - min_lines: 20 key_links: - from: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java" to: "application.yml" @@ -61,10 +56,10 @@ must_haves: --- -Set up ClickHouse infrastructure, schema, WriteBuffer with backpressure, repository interfaces, and test infrastructure. +Set up ClickHouse infrastructure, schema, WriteBuffer with backpressure, and repository interfaces. Purpose: Establishes the storage foundation that all ingestion endpoints and future search queries depend on. The WriteBuffer is the central throughput mechanism -- all data flows through it before reaching ClickHouse. -Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with unit tests, repository interfaces, Testcontainers base class for integration tests. +Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with unit tests, repository interfaces. @@ -147,7 +142,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un - Task 2: WriteBuffer, repository interfaces, IngestionConfig, ClickHouseConfig, and test infrastructure + Task 2: WriteBuffer, repository interfaces, IngestionConfig, and ClickHouseConfig cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java, cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java, @@ -155,9 +150,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java, cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java, - cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java, - cameleer3-server-app/src/test/resources/application-test.yml, - cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java + cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java - WriteBuffer(capacity=10): offer() returns true for first 10 items, false on 11th @@ -168,7 +161,6 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un - ExecutionRepository interface declares insertBatch(List of RouteExecution) - DiagramRepository interface declares store(RouteGraph) and findByContentHash(String) - MetricsRepository interface declares insertBatch(List of metric data) - - AbstractClickHouseIT starts a Testcontainers ClickHouse, applies schema, provides JdbcTemplate 1. Create WriteBuffer in core module (no Spring dependency): @@ -199,23 +191,11 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un 5. Create ClickHouseConfig as @Configuration: - Exposes JdbcTemplate bean (Spring Boot auto-configures DataSource from spring.datasource) - No custom bean needed if relying on auto-config; only create if explicit JdbcTemplate customization required - - 6. Create application-test.yml for test profile: - - Placeholder datasource config (overridden by Testcontainers in AbstractClickHouseIT) - - ingestion: small buffer for tests (capacity=100, batch-size=10, flush-interval-ms=100) - - 7. Create AbstractClickHouseIT base class: - - @Testcontainers + @Container with ClickHouseContainer("clickhouse/clickhouse-server:25.3") - - @DynamicPropertySource to override spring.datasource.url/username/password - - @SpringBootTest - - @ActiveProfiles("test") - - @BeforeAll: read clickhouse/init/01-schema.sql and execute it against the container via JDBC - - Expose protected JdbcTemplate for subclasses mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q 2>&1 | tail -10 - WriteBuffer passes all unit tests. Repository interfaces exist with correct method signatures. IngestionConfig reads from application.yml. AbstractClickHouseIT base class exists and can start a Testcontainers ClickHouse instance with schema applied. + WriteBuffer passes all unit tests. Repository interfaces exist with correct method signatures. IngestionConfig reads from application.yml. @@ -228,7 +208,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un -WriteBuffer unit tests green. Project compiles. ClickHouse DDL defines all three tables with TTL and correct partitioning. Repository interfaces define batch insert contracts. Testcontainers base class ready for integration tests in Plan 02. +WriteBuffer unit tests green. Project compiles. ClickHouse DDL defines all three tables with TTL and correct partitioning. Repository interfaces define batch insert contracts. diff --git a/.planning/phases/01-ingestion-pipeline-api-foundation/01-02-PLAN.md b/.planning/phases/01-ingestion-pipeline-api-foundation/01-02-PLAN.md index 59c7a00a..51c929db 100644 --- a/.planning/phases/01-ingestion-pipeline-api-foundation/01-02-PLAN.md +++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-02-PLAN.md @@ -60,7 +60,15 @@ must_haves: - from: "ClickHouseFlushScheduler.java" to: "ClickHouseExecutionRepository.java" via: "insertBatch call" - pattern: "repository\\.insertBatch" + pattern: "executionRepository\\.insertBatch" + - from: "ClickHouseFlushScheduler.java" + to: "ClickHouseDiagramRepository.java" + via: "store call after drain" + pattern: "diagramRepository\\.store" + - from: "ClickHouseFlushScheduler.java" + to: "ClickHouseMetricsRepository.java" + via: "insertBatch call after drain" + pattern: "metricsRepository\\.insertBatch" --- @@ -127,7 +135,7 @@ public class IngestionConfig { - + Task 1: IngestionService, ClickHouse repositories, and flush scheduler cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java, @@ -136,15 +144,6 @@ public class IngestionConfig { cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsRepository.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java - - - IngestionService.acceptExecutions(list) delegates to WriteBuffer.offerBatch, returns boolean - - IngestionService.acceptDiagram(graph) computes content hash, delegates to WriteBuffer.offer - - IngestionService.acceptMetrics(list) delegates to WriteBuffer.offerBatch - - ClickHouseExecutionRepository.insertBatch uses JdbcTemplate.batchUpdate with all columns including Array columns for processor executions - - ClickHouseDiagramRepository.store computes SHA-256 content hash and does INSERT with content_hash dedup - - ClickHouseFlushScheduler runs on @Scheduled(fixedDelayString="${ingestion.flush-interval-ms:1000}"), drains each buffer and calls respective repository.insertBatch - - ClickHouseFlushScheduler implements SmartLifecycle for graceful shutdown (flush remaining on stop) - 1. Create IngestionService in core module (no Spring annotations -- it's a plain class): - Constructor takes three WriteBuffer instances (executions, diagrams, metrics) diff --git a/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md b/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md index eff0bf69..b344a67d 100644 --- a/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md +++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md @@ -8,6 +8,8 @@ files_modified: - cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java - cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java - cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java + - cameleer3-server-app/src/test/resources/application-test.yml + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java - cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java @@ -38,6 +40,9 @@ must_haves: - path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java" provides: "Registers interceptor with path patterns" min_lines: 10 + - path: "cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java" + provides: "Shared Testcontainers base class for integration tests" + min_lines: 20 key_links: - from: "WebConfig.java" to: "ProtocolVersionInterceptor.java" @@ -54,10 +59,10 @@ must_haves: --- -Implement the API foundation: health endpoint, OpenAPI documentation, protocol version header validation, forward compatibility, and TTL verification. +Implement the API foundation: test infrastructure, health endpoint, OpenAPI documentation, protocol version header validation, forward compatibility, and TTL verification. -Purpose: Completes the API scaffolding so all endpoints follow the protocol v1 contract. Health endpoint enables monitoring. OpenAPI enables discoverability. Protocol version interceptor enforces compatibility. TTL verification confirms data retention. -Output: Working health, Swagger UI, protocol header enforcement, and verified TTL retention. +Purpose: Establishes the Testcontainers base class used by all integration tests across plans. Completes the API scaffolding so all endpoints follow the protocol v1 contract. Health endpoint enables monitoring. OpenAPI enables discoverability. Protocol version interceptor enforces compatibility. TTL verification confirms data retention. +Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol header enforcement, and verified TTL retention. @@ -75,34 +80,40 @@ Output: Working health, Swagger UI, protocol header enforcement, and verified TT - - Task 1: Protocol version interceptor, WebConfig, and Spring Boot application class + + Task 1: Test infrastructure, protocol version interceptor, WebConfig, and Spring Boot application class + cameleer3-server-app/src/test/resources/application-test.yml, + cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java, cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java - - - Request to /api/v1/data/* without X-Cameleer-Protocol-Version header returns 400 - - Request to /api/v1/data/* with X-Cameleer-Protocol-Version:2 returns 400 - - Request to /api/v1/data/* with X-Cameleer-Protocol-Version:1 passes through - - Request to /api/v1/health without the header succeeds (excluded from interceptor) - - Request to /api/v1/swagger-ui without the header succeeds (excluded) - - Request to /api/v1/api-docs without the header succeeds (excluded) - - 1. Create ProtocolVersionInterceptor implementing HandlerInterceptor: + 1. Create application-test.yml for test profile: + - Placeholder datasource config (overridden by Testcontainers in AbstractClickHouseIT) + - ingestion: small buffer for tests (capacity=100, batch-size=10, flush-interval-ms=100) + + 2. Create AbstractClickHouseIT base class: + - @Testcontainers + @Container with ClickHouseContainer("clickhouse/clickhouse-server:25.3") + - @DynamicPropertySource to override spring.datasource.url/username/password + - @SpringBootTest + - @ActiveProfiles("test") + - @BeforeAll: read clickhouse/init/01-schema.sql and execute it against the container via JDBC + - Expose protected JdbcTemplate for subclasses + + 3. Create ProtocolVersionInterceptor implementing HandlerInterceptor: - preHandle: read X-Cameleer-Protocol-Version header - If null or not "1": set response status 400, write JSON error body {"error": "Missing or unsupported X-Cameleer-Protocol-Version header"}, return false - If "1": return true - 2. Create WebConfig implementing WebMvcConfigurer: + 4. Create WebConfig implementing WebMvcConfigurer: - @Configuration - Inject ProtocolVersionInterceptor (declare it as @Component or @Bean) - Override addInterceptors: register interceptor with pathPatterns "/api/v1/data/**" and "/api/v1/agents/**" - Explicitly EXCLUDE: "/api/v1/health", "/api/v1/api-docs/**", "/api/v1/swagger-ui/**", "/api/v1/swagger-ui.html" - 3. Create or update Cameleer3ServerApplication: + 5. Create or update Cameleer3ServerApplication: - @SpringBootApplication in package com.cameleer3.server.app - @EnableScheduling (needed for ClickHouseFlushScheduler from Plan 02) - @EnableConfigurationProperties(IngestionConfig.class) @@ -112,7 +123,7 @@ Output: Working health, Swagger UI, protocol header enforcement, and verified TT mvn clean compile -pl cameleer3-server-app -q 2>&1 | tail -5 - ProtocolVersionInterceptor validates header on data/agent paths. Health, swagger, and api-docs paths excluded. Application class enables scheduling and config properties. + AbstractClickHouseIT base class ready for integration tests. ProtocolVersionInterceptor validates header on data/agent paths. Health, swagger, and api-docs paths excluded. Application class enables scheduling and config properties. @@ -133,6 +144,7 @@ Output: Working health, Swagger UI, protocol header enforcement, and verified TT - GET /api/v1/health without protocol header returns 200 (not intercepted) - POST /api/v1/data/executions with extra unknown JSON fields returns 202 (not 400/422) - ClickHouse route_executions table SHOW CREATE TABLE includes TTL + - ClickHouse agent_metrics table SHOW CREATE TABLE includes TTL 1. Create HealthControllerIT (extends AbstractClickHouseIT): @@ -155,18 +167,20 @@ Output: Working health, Swagger UI, protocol header enforcement, and verified TT - Test: POST /api/v1/data/executions with valid RouteExecution JSON plus extra unknown fields (e.g., "futureField": "value") -> 202 (Jackson does not fail on unknown properties) - This validates API-05 requirement explicitly. - 5. TTL verification (add to HealthControllerIT or separate test): - - Query ClickHouse: SHOW CREATE TABLE route_executions - - Assert result contains "TTL start_time + toIntervalDay(30)" - - Query: SHOW CREATE TABLE agent_metrics - - Assert result contains TTL clause + 5. TTL verification (add to HealthControllerIT as dedicated test methods): + - Test method: ttlConfiguredOnRouteExecutions + Query ClickHouse: SHOW CREATE TABLE route_executions + Assert result contains "TTL start_time + toIntervalDay(30)" + - Test method: ttlConfiguredOnAgentMetrics + Query ClickHouse: SHOW CREATE TABLE agent_metrics + Assert result contains TTL clause Note: All tests that POST to data endpoints must include X-Cameleer-Protocol-Version:1 header. mvn test -pl cameleer3-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q 2>&1 | tail -15 - Health returns 200. OpenAPI docs are available and list endpoints. Protocol version header enforced on data paths, not on health/docs. Unknown JSON fields accepted. TTL confirmed in ClickHouse DDL. + Health returns 200. OpenAPI docs are available and list endpoints. Protocol version header enforced on data paths, not on health/docs. Unknown JSON fields accepted. TTL confirmed in ClickHouse DDL via HealthControllerIT test methods. @@ -180,7 +194,7 @@ Output: Working health, Swagger UI, protocol header enforcement, and verified TT -All four integration test classes green. Health endpoint accessible. OpenAPI docs list all ingestion endpoints. Protocol version header enforced on data/agent paths but not health/docs. Forward compatibility confirmed. TTL verified in ClickHouse schema. +All four integration test classes green. AbstractClickHouseIT base class works with Testcontainers. Health endpoint accessible. OpenAPI docs list all ingestion endpoints. Protocol version header enforced on data/agent paths but not health/docs. Forward compatibility confirmed. TTL verified in ClickHouse schema via HealthControllerIT. diff --git a/.planning/phases/01-ingestion-pipeline-api-foundation/01-VALIDATION.md b/.planning/phases/01-ingestion-pipeline-api-foundation/01-VALIDATION.md index b559dc6f..d83fb467 100644 --- a/.planning/phases/01-ingestion-pipeline-api-foundation/01-VALIDATION.md +++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-VALIDATION.md @@ -38,19 +38,19 @@ created: 2026-03-11 | Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status | |---------|------|------|-------------|-----------|-------------------|-------------|--------| -| 1-01-01 | 01 | 1 | INGST-04 | unit | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` | ❌ W0 | ⬜ pending | -| 1-01-02 | 01 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | ❌ W0 | ⬜ pending | -| 1-01-03 | 01 | 1 | INGST-05 | unit+integration | `mvn test -pl cameleer3-server-app -Dtest=BackpressureIT -q` | ❌ W0 | ⬜ pending | -| 1-01-04 | 01 | 1 | INGST-06 | integration | `mvn test -pl cameleer3-server-app -Dtest=ClickHouseTtlIT -q` | ❌ W0 | ⬜ pending | -| 1-02-01 | 02 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | ❌ W0 | ⬜ pending | -| 1-02-02 | 02 | 1 | INGST-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=DiagramControllerIT -q` | ❌ W0 | ⬜ pending | -| 1-02-03 | 02 | 1 | INGST-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=MetricsControllerIT -q` | ❌ W0 | ⬜ pending | -| 1-02-04 | 02 | 1 | API-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=OpenApiIT -q` | ❌ W0 | ⬜ pending | -| 1-02-05 | 02 | 1 | API-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT -q` | ❌ W0 | ⬜ pending | -| 1-02-06 | 02 | 1 | API-04 | integration | `mvn test -pl cameleer3-server-app -Dtest=ProtocolVersionIT -q` | ❌ W0 | ⬜ pending | -| 1-02-07 | 02 | 1 | API-05 | unit | `mvn test -pl cameleer3-server-app -Dtest=ForwardCompatIT -q` | ❌ W0 | ⬜ pending | +| 1-01-01 | 01 | 1 | INGST-04 | unit | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` | no W0 | pending | +| 1-01-02 | 01 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending | +| 1-01-03 | 01 | 1 | INGST-05 | integration | `mvn test -pl cameleer3-server-app -Dtest=BackpressureIT -q` | no W0 | pending | +| 1-01-04 | 01 | 1 | INGST-06 | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT#ttlConfigured* -q` | no W0 | pending | +| 1-02-01 | 02 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending | +| 1-02-02 | 02 | 1 | INGST-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=DiagramControllerIT -q` | no W0 | pending | +| 1-02-03 | 02 | 1 | INGST-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=MetricsControllerIT -q` | no W0 | pending | +| 1-02-04 | 02 | 1 | API-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=OpenApiIT -q` | no W0 | pending | +| 1-02-05 | 02 | 1 | API-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT -q` | no W0 | pending | +| 1-02-06 | 02 | 1 | API-04 | integration | `mvn test -pl cameleer3-server-app -Dtest=ProtocolVersionIT -q` | no W0 | pending | +| 1-02-07 | 02 | 1 | API-05 | unit | `mvn test -pl cameleer3-server-app -Dtest=ForwardCompatIT -q` | no W0 | pending | -*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* +*Status: pending / green / red / flaky* --- @@ -70,7 +70,7 @@ created: 2026-03-11 | Behavior | Requirement | Why Manual | Test Instructions | |----------|-------------|------------|-------------------| -| ClickHouse TTL removes data after 30 days | INGST-06 | Cannot fast-forward time in ClickHouse | Verify TTL clause in CREATE TABLE DDL, integration test uses SYSTEM TTL MERGES | +| ClickHouse TTL removes data after 30 days | INGST-06 | Cannot fast-forward time in ClickHouse | Verify TTL clause in CREATE TABLE DDL; automated test in HealthControllerIT asserts DDL contains TTL | ---