cameleer-server/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
01-ingestion-pipeline-api-foundation	03	execute	2	01-01	cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java cameleer-server-app/src/main/java/com/cameleer/server/app/CameleerServerApplication.java cameleer-server-app/src/test/resources/application-test.yml cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java cameleer-server-app/src/test/java/com/cameleer/server/app/interceptor/ProtocolVersionIT.java cameleer-server-app/src/test/java/com/cameleer/server/app/controller/HealthControllerIT.java cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenApiIT.java cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ForwardCompatIT.java	true	API-01 API-02 API-03 API-04 API-05 INGST-06	truths artifacts key_links GET /api/v1/health returns 200 with health status GET /api/v1/swagger-ui returns Swagger UI page GET /api/v1/api-docs returns OpenAPI JSON spec listing all endpoints Requests without X-Cameleer-Protocol-Version header to /api/v1/data/* return 400 Requests with wrong protocol version return 400 Health and swagger endpoints work without protocol version header Unknown JSON fields in request body do not cause deserialization errors ClickHouse tables have TTL clause for 30-day retention path provides min_lines cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java Validates X-Cameleer-Protocol-Version:1 header on data endpoints 20 path provides min_lines cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java Registers interceptor with path patterns 10 path provides min_lines cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java Shared Testcontainers base class for integration tests 20 from to via pattern WebConfig.java ProtocolVersionInterceptor.java addInterceptors registration addInterceptors.*ProtocolVersionInterceptor from to via pattern application.yml Actuator health endpoint management.endpoints config management.endpoints from to via pattern application.yml springdoc springdoc.api-docs.path and swagger-ui.path springdoc

Implement the API foundation: test infrastructure, health endpoint, OpenAPI documentation, protocol version header validation, forward compatibility, and TTL verification.

Purpose: Establishes the Testcontainers base class used by all integration tests across plans. Completes the API scaffolding so all endpoints follow the protocol v1 contract. Health endpoint enables monitoring. OpenAPI enables discoverability. Protocol version interceptor enforces compatibility. TTL verification confirms data retention. Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol header enforcement, and verified TTL retention.

<execution_context> @C:/Users/Hendrik/.claude/get-shit-done/workflows/execute-plan.md @C:/Users/Hendrik/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/01-ingestion-pipeline-api-foundation/01-RESEARCH.md @.planning/phases/01-ingestion-pipeline-api-foundation/01-01-SUMMARY.md Task 1: Test infrastructure, protocol version interceptor, WebConfig, and Spring Boot application class cameleer-server-app/src/test/resources/application-test.yml, cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java, cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java, cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java, cameleer-server-app/src/main/java/com/cameleer/server/app/CameleerServerApplication.java 1. Create application-test.yml for test profile: - Placeholder datasource config (overridden by Testcontainers in AbstractClickHouseIT) - ingestion: small buffer for tests (capacity=100, batch-size=10, flush-interval-ms=100)

2. Create AbstractClickHouseIT base class:
   - @Testcontainers + @Container with ClickHouseContainer("clickhouse/clickhouse-server:25.3")
   - @DynamicPropertySource to override spring.datasource.url/username/password
   - @SpringBootTest
   - @ActiveProfiles("test")
   - @BeforeAll: read clickhouse/init/01-schema.sql and execute it against the container via JDBC
   - Expose protected JdbcTemplate for subclasses

3. Create ProtocolVersionInterceptor implementing HandlerInterceptor:
   - preHandle: read X-Cameleer-Protocol-Version header
   - If null or not "1": set response status 400, write JSON error body {"error": "Missing or unsupported X-Cameleer-Protocol-Version header"}, return false
   - If "1": return true

4. Create WebConfig implementing WebMvcConfigurer:
   - @Configuration
   - Inject ProtocolVersionInterceptor (declare it as @Component or @Bean)
   - Override addInterceptors: register interceptor with pathPatterns "/api/v1/data/**" and "/api/v1/agents/**"
   - Explicitly EXCLUDE: "/api/v1/health", "/api/v1/api-docs/**", "/api/v1/swagger-ui/**", "/api/v1/swagger-ui.html"

5. Create or update CameleerServerApplication:
   - @SpringBootApplication in package com.cameleer.server.app
   - @EnableScheduling (needed for ClickHouseFlushScheduler from Plan 02)
   - @EnableConfigurationProperties(IngestionConfig.class)
   - Main method with SpringApplication.run()
   - Ensure package scanning covers com.cameleer.server.app and com.cameleer.server.core

mvn clean compile -pl cameleer-server-app -q 2>&1 | tail -5 AbstractClickHouseIT base class ready for integration tests. ProtocolVersionInterceptor validates header on data/agent paths. Health, swagger, and api-docs paths excluded. Application class enables scheduling and config properties. Task 2: Health, OpenAPI, protocol version, forward compat, and TTL integration tests cameleer-server-app/src/test/java/com/cameleer/server/app/controller/HealthControllerIT.java, cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenApiIT.java, cameleer-server-app/src/test/java/com/cameleer/server/app/interceptor/ProtocolVersionIT.java, cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ForwardCompatIT.java - GET /api/v1/health returns 200 with JSON containing status field - GET /api/v1/api-docs returns 200 with OpenAPI JSON containing paths - GET /api/v1/swagger-ui/index.html returns 200 (Swagger UI page) - POST /api/v1/data/executions without protocol header returns 400 - POST /api/v1/data/executions with X-Cameleer-Protocol-Version:1 does NOT return 400 (may return 202 or other based on body) - POST /api/v1/data/executions with X-Cameleer-Protocol-Version:2 returns 400 - GET /api/v1/health without protocol header returns 200 (not intercepted) - POST /api/v1/data/executions with extra unknown JSON fields returns 202 (not 400/422) - ClickHouse route_executions table SHOW CREATE TABLE includes TTL - ClickHouse agent_metrics table SHOW CREATE TABLE includes TTL 1. Create HealthControllerIT (extends AbstractClickHouseIT): - Test: GET /api/v1/health -> 200 with body containing "status" - Test: No X-Cameleer-Protocol-Version header required for health

2. Create OpenApiIT (extends AbstractClickHouseIT):
   - Test: GET /api/v1/api-docs -> 200, body contains "openapi" and "paths"
   - Test: GET /api/v1/swagger-ui/index.html -> 200 or 302 (redirect to UI)
   - Test: api-docs lists /api/v1/data/executions path

3. Create ProtocolVersionIT (extends AbstractClickHouseIT):
   - Test: POST /api/v1/data/executions without header -> 400 with error message
   - Test: POST /api/v1/data/executions with header value "2" -> 400
   - Test: POST /api/v1/data/executions with header value "1" and valid body -> 202 (not 400)
   - Test: GET /api/v1/health without header -> 200 (excluded from interceptor)
   - Test: GET /api/v1/api-docs without header -> 200 (excluded from interceptor)

4. Create ForwardCompatIT (extends AbstractClickHouseIT):
   - Test: POST /api/v1/data/executions with valid RouteExecution JSON plus extra unknown fields (e.g., "futureField": "value") -> 202 (Jackson does not fail on unknown properties)
   - This validates API-05 requirement explicitly.

5. TTL verification (add to HealthControllerIT as dedicated test methods):
   - Test method: ttlConfiguredOnRouteExecutions
     Query ClickHouse: SHOW CREATE TABLE route_executions
     Assert result contains "TTL start_time + toIntervalDay(30)"
   - Test method: ttlConfiguredOnAgentMetrics
     Query ClickHouse: SHOW CREATE TABLE agent_metrics
     Assert result contains TTL clause

Note: All tests that POST to data endpoints must include X-Cameleer-Protocol-Version:1 header.

mvn test -pl cameleer-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q 2>&1 | tail -15 Health returns 200. OpenAPI docs are available and list endpoints. Protocol version header enforced on data paths, not on health/docs. Unknown JSON fields accepted. TTL confirmed in ClickHouse DDL via HealthControllerIT test methods. - `mvn test -pl cameleer-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q` -- all tests pass - GET /api/v1/health returns 200 - GET /api/v1/api-docs returns OpenAPI spec - Missing protocol header returns 400 on data endpoints - Health works without protocol header

<success_criteria> All four integration test classes green. AbstractClickHouseIT base class works with Testcontainers. Health endpoint accessible. OpenAPI docs list all ingestion endpoints. Protocol version header enforced on data/agent paths but not health/docs. Forward compatibility confirmed. TTL verified in ClickHouse schema via HealthControllerIT. </success_criteria>

After completion, create `.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md`