docs(01-03): complete API foundation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 12:05:02 +01:00
parent 2d3fde3766
commit ff0af0ef2f
4 changed files with 182 additions and 17 deletions
--- a/.planning/REQUIREMENTS.md
+++ b/.planning/REQUIREMENTS.md
@@ -51,11 +51,11 @@ Requirements for initial release. Each maps to roadmap phases. Tracked as Gitea

 ### REST API

- [ ] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28)
- [ ] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29)
- [ ] **API-03**: Server includes `GET /api/v1/health` endpoint (#30)
- [ ] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31)
- [ ] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32)
+- [x] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28)
+- [x] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29)
+- [x] **API-03**: Server includes `GET /api/v1/health` endpoint (#30)
+- [x] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31)
+- [x] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32)

 ## v2 Requirements

--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@@ -29,7 +29,7 @@ Decimal phases appear between their surrounding integers in numeric order.
  3. When the write buffer is full, the server returns 503 and does not lose already-buffered data
  4. Data older than the configured TTL (default 30 days) is automatically removed by ClickHouse
  5. The health endpoint responds at `/api/v1/health`, OpenAPI docs are available, protocol version header is validated, and unknown JSON fields are accepted
-**Plans:** 3 plans
+**Plans:** 2/3 plans executed

 Plans:
 - [ ] 01-01-PLAN.md -- ClickHouse infrastructure, schema, WriteBuffer, repository interfaces, test infrastructure
@@ -87,7 +87,7 @@ Note: Phases 2 and 3 both depend only on Phase 1 and could execute in parallel.

 | Phase | Plans Complete | Status | Completed |
 |-------|----------------|--------|-----------|
-| 1. Ingestion Pipeline + API Foundation | 0/3 | Not started | - |
+| 1. Ingestion Pipeline + API Foundation | 2/3 | In Progress|  |
 | 2. Transaction Search + Diagrams | 0/2 | Not started | - |
 | 3. Agent Registry + SSE Push | 0/2 | Not started | - |
 | 4. Security | 0/1 | Not started | - |
--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@@ -3,15 +3,15 @@ gsd_state_version: 1.0
 milestone: v1.0
 milestone_name: milestone
 status: executing
-stopped_at: Completed 01-01-PLAN.md
-last_updated: "2026-03-11T10:50:59.918Z"
-last_activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer)
+stopped_at: Completed 01-03-PLAN.md
+last_updated: "2026-03-11T11:03:08.000Z"
+last_activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests)
 progress:
  total_phases: 4
  completed_phases: 0
  total_plans: 3
-  completed_plans: 1
-  percent: 33
+  completed_plans: 2
+  percent: 66
 ---

 # Project State
@@ -26,11 +26,11 @@ See: .planning/PROJECT.md (updated 2026-03-11)
 ## Current Position

 Phase: 1 of 4 (Ingestion Pipeline + API Foundation)
-Plan: 1 of 3 in current phase
+Plan: 2 of 3 in current phase
 Status: Executing
-Last activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer)
+Last activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests)

-Progress: [███░░░░░░░] 33%
+Progress: [██████░░░░] 66%

 ## Performance Metrics

@@ -51,6 +51,7 @@ Progress: [███░░░░░░░] 33%

 *Updated after each plan completion*
 | Phase 01 P01 | 3min | 2 tasks | 13 files |
+| Phase 01 P03 | 10min | 2 tasks | 12 files |

 ## Accumulated Context

@@ -65,6 +66,9 @@ Recent decisions affecting current work:
 - [Roadmap]: Web UI deferred to v2
 - [Phase 01]: Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config
 - [Phase 01]: Created MetricsSnapshot record in core module (cameleer3-common has no metrics model)
+- [Phase 01]: Upgraded testcontainers to 2.0.3 for Docker Desktop 29.x compatibility
+- [Phase 01]: Changed error_message/error_stacktrace to non-nullable String for tokenbf_v1 index compat
+- [Phase 01]: TTL expressions require toDateTime() cast for DateTime64 columns in ClickHouse 25.3

 ### Pending Todos

@@ -79,6 +83,6 @@ None yet.

 ## Session Continuity

-Last session: 2026-03-11T10:50:59.915Z
-Stopped at: Completed 01-01-PLAN.md
+Last session: 2026-03-11T11:03:08.000Z
+Stopped at: Completed 01-03-PLAN.md
 Resume file: None
--- a/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md
+++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md
@@ -0,0 +1,161 @@
+---
+phase: 01-ingestion-pipeline-api-foundation
+plan: 03
+subsystem: api
+tags: [spring-boot, testcontainers, openapi, interceptor, clickhouse, integration-tests]
+
+requires:
+  - phase: 01-01
+    provides: ClickHouse schema, application.yml, dependencies
+
+provides:
+  - AbstractClickHouseIT base class for all integration tests
+  - ProtocolVersionInterceptor enforcing X-Cameleer-Protocol-Version:1 on data/agent paths
+  - WebConfig with interceptor registration and path exclusions
+  - Cameleer3ServerApplication with @EnableScheduling and component scanning
+  - 12 passing integration tests (health, OpenAPI, protocol version, forward compat, TTL)
+
+affects: [01-02, 02-search, 03-agent-registry]
+
+tech-stack:
+  added: [testcontainers 2.0.3, docker-java 3.7.0]
+  patterns: [shared static Testcontainers ClickHouse container, HandlerInterceptor for protocol validation, WebMvcConfigurer for path-based interceptor registration]
+
+key-files:
+  created:
+    - cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
+    - cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java
+    - cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java
+    - cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java
+    - cameleer3-server-app/src/test/resources/application-test.yml
+    - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java
+    - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java
+    - cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java
+    - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
+  modified:
+    - cameleer3-server-app/pom.xml
+    - pom.xml
+    - clickhouse/init/01-schema.sql
+
+key-decisions:
+  - "Upgraded testcontainers from 1.20.5 to 2.0.3 for Docker Desktop 29.x compatibility (docker-java 3.7.0)"
+  - "Removed junit-jupiter dependency; manual container lifecycle via static initializer instead"
+  - "Changed error_message/error_stacktrace from Nullable(String) to String DEFAULT '' for tokenbf_v1 skip index compatibility"
+  - "TTL expressions use toDateTime() cast for DateTime64 columns in ClickHouse 25.3"
+
+patterns-established:
+  - "AbstractClickHouseIT: static container shared across test classes, @DynamicPropertySource for datasource, @BeforeAll for schema init"
+  - "ProtocolVersionInterceptor: all data/agent endpoints require X-Cameleer-Protocol-Version:1 header"
+  - "Path exclusions: health, api-docs, swagger-ui bypass protocol version check"
+
+requirements-completed: [API-01, API-02, API-03, API-04, API-05, INGST-06]
+
+duration: 10min
+completed: 2026-03-11
+---
+
+# Phase 1 Plan 03: API Foundation Summary
+
+**Protocol version interceptor, health/OpenAPI endpoints, Testcontainers IT base class, and 12 green integration tests covering TTL, forward compat, and interceptor exclusions**
+
+## Performance
+
+- **Duration:** 10 min
+- **Started:** 2026-03-11T10:52:21Z
+- **Completed:** 2026-03-11T11:03:08Z
+- **Tasks:** 2
+- **Files modified:** 12
+
+## Accomplishments
+- ProtocolVersionInterceptor validates X-Cameleer-Protocol-Version:1 on /api/v1/data/** and /api/v1/agents/** paths, returning 400 JSON error for missing or wrong version
+- AbstractClickHouseIT base class with Testcontainers ClickHouse 25.3, shared static container, schema init from 01-schema.sql
+- 12 integration tests: health endpoint (2), OpenAPI docs (2), protocol version enforcement (5), forward compatibility (1), TTL verification (2)
+- Cameleer3ServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Test infrastructure, protocol version interceptor, WebConfig, and app bootstrap** - `b8a4739` (feat)
+2. **Task 2: Integration tests for health, OpenAPI, protocol version, forward compat, and TTL** - `2d3fde3` (test)
+
+## Files Created/Modified
+- `cameleer3-server-app/src/main/java/.../Cameleer3ServerApplication.java` - Spring Boot entry point with scheduling and config properties
+- `cameleer3-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java` - Validates protocol version header on data/agent paths
+- `cameleer3-server-app/src/main/java/.../config/WebConfig.java` - Registers interceptor with path patterns and exclusions
+- `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` - Shared Testcontainers base class for ITs
+- `cameleer3-server-app/src/test/resources/application-test.yml` - Test profile with small buffer config
+- `cameleer3-server-app/src/test/java/.../controller/HealthControllerIT.java` - Health endpoint and TTL tests
+- `cameleer3-server-app/src/test/java/.../controller/OpenApiIT.java` - OpenAPI and Swagger UI tests
+- `cameleer3-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java` - Protocol header enforcement tests
+- `cameleer3-server-app/src/test/java/.../controller/ForwardCompatIT.java` - Unknown JSON fields test
+- `pom.xml` - Override testcontainers.version to 2.0.3
+- `cameleer3-server-app/pom.xml` - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3
+- `clickhouse/init/01-schema.sql` - Fix TTL expressions and error column types
+
+## Decisions Made
+- Upgraded Testcontainers to 2.0.3 because docker-java 3.4.1 (from TC 1.20.5) is incompatible with Docker Desktop 29.x API 1.52
+- Removed junit-jupiter module (doesn't exist in TC 2.x); manage container lifecycle via static initializer block instead
+- Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT '' because ClickHouse tokenbf_v1 skip indexes require non-nullable String columns
+- Added toDateTime() cast in TTL expressions because ClickHouse 25.3 requires DateTime (not DateTime64) in TTL column references
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] Fixed testcontainers junit-jupiter dependency not available in 2.0.x**
+- **Found during:** Task 2 (compilation)
+- **Issue:** org.testcontainers:junit-jupiter:2.0.2 does not exist in Maven Central
+- **Fix:** Removed junit-jupiter dependency, upgraded to TC 2.0.3, managed container lifecycle manually via static initializer
+- **Files modified:** cameleer3-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java
+- **Verification:** All tests compile and pass
+- **Committed in:** 2d3fde3
+
+**2. [Rule 3 - Blocking] Fixed Docker Desktop 29.x incompatibility with Testcontainers**
+- **Found during:** Task 2 (test execution)
+- **Issue:** docker-java 3.4.1 (from TC 1.20.5) sends unversioned API calls to Docker API 1.52, receiving 400 errors
+- **Fix:** Override testcontainers.version to 2.0.3 in parent POM (brings docker-java 3.7.0 with proper API version negotiation)
+- **Files modified:** pom.xml
+- **Verification:** ClickHouse container starts successfully
+- **Committed in:** 2d3fde3
+
+**3. [Rule 1 - Bug] Fixed ClickHouse TTL expression for DateTime64 columns**
+- **Found during:** Task 2 (schema init in tests)
+- **Issue:** ClickHouse 25.3 requires TTL expressions to resolve to DateTime, not DateTime64(3, 'UTC')
+- **Fix:** Changed `TTL start_time + INTERVAL 30 DAY` to `TTL toDateTime(start_time) + toIntervalDay(30)`
+- **Files modified:** clickhouse/init/01-schema.sql
+- **Verification:** Schema creates without errors in ClickHouse 25.3 container
+- **Committed in:** 2d3fde3
+
+**4. [Rule 1 - Bug] Fixed tokenbf_v1 index on Nullable column**
+- **Found during:** Task 2 (schema init in tests)
+- **Issue:** ClickHouse 25.3 does not allow tokenbf_v1 skip indexes on Nullable(String) columns
+- **Fix:** Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT ''
+- **Files modified:** clickhouse/init/01-schema.sql
+- **Verification:** Schema creates without errors, all 12 tests pass
+- **Committed in:** 2d3fde3
+
+---
+
+**Total deviations:** 4 auto-fixed (2 blocking, 2 bugs)
+**Impact on plan:** All fixes necessary for test execution on current Docker Desktop and ClickHouse versions. No scope creep.
+
+## Issues Encountered
+- Surefire runs from module directory, not project root; fixed schema path lookup in AbstractClickHouseIT to check both locations
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- AbstractClickHouseIT base class ready for all future integration tests
+- Protocol version interceptor active for data/agent endpoints
+- API foundation complete, ready for Plan 02 (REST controllers, ClickHouse repositories, flush scheduler)
+- Health endpoint at /api/v1/health, OpenAPI at /api/v1/api-docs, Swagger UI at /api/v1/swagger-ui
+
+## Self-Check: PASSED
+
+All 9 created files verified present. Both task commits verified in git log.
+
+---
+*Phase: 01-ingestion-pipeline-api-foundation*
+*Completed: 2026-03-11*