diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index f8ed0592..48e1f07c 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -51,11 +51,11 @@ Requirements for initial release. Each maps to roadmap phases. Tracked as Gitea ### REST API -- [ ] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28) -- [ ] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29) -- [ ] **API-03**: Server includes `GET /api/v1/health` endpoint (#30) -- [ ] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31) -- [ ] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32) +- [x] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28) +- [x] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29) +- [x] **API-03**: Server includes `GET /api/v1/health` endpoint (#30) +- [x] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31) +- [x] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32) ## v2 Requirements diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index c4e2fd9a..0fff8ce8 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -29,7 +29,7 @@ Decimal phases appear between their surrounding integers in numeric order. 3. When the write buffer is full, the server returns 503 and does not lose already-buffered data 4. Data older than the configured TTL (default 30 days) is automatically removed by ClickHouse 5. The health endpoint responds at `/api/v1/health`, OpenAPI docs are available, protocol version header is validated, and unknown JSON fields are accepted -**Plans:** 3 plans +**Plans:** 2/3 plans executed Plans: - [ ] 01-01-PLAN.md -- ClickHouse infrastructure, schema, WriteBuffer, repository interfaces, test infrastructure @@ -87,7 +87,7 @@ Note: Phases 2 and 3 both depend only on Phase 1 and could execute in parallel. | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| -| 1. Ingestion Pipeline + API Foundation | 0/3 | Not started | - | +| 1. Ingestion Pipeline + API Foundation | 2/3 | In Progress| | | 2. Transaction Search + Diagrams | 0/2 | Not started | - | | 3. Agent Registry + SSE Push | 0/2 | Not started | - | | 4. Security | 0/1 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index e466ca61..456e6679 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,15 +3,15 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: executing -stopped_at: Completed 01-01-PLAN.md -last_updated: "2026-03-11T10:50:59.918Z" -last_activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer) +stopped_at: Completed 01-03-PLAN.md +last_updated: "2026-03-11T11:03:08.000Z" +last_activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests) progress: total_phases: 4 completed_phases: 0 total_plans: 3 - completed_plans: 1 - percent: 33 + completed_plans: 2 + percent: 66 --- # Project State @@ -26,11 +26,11 @@ See: .planning/PROJECT.md (updated 2026-03-11) ## Current Position Phase: 1 of 4 (Ingestion Pipeline + API Foundation) -Plan: 1 of 3 in current phase +Plan: 2 of 3 in current phase Status: Executing -Last activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer) +Last activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests) -Progress: [███░░░░░░░] 33% +Progress: [██████░░░░] 66% ## Performance Metrics @@ -51,6 +51,7 @@ Progress: [███░░░░░░░] 33% *Updated after each plan completion* | Phase 01 P01 | 3min | 2 tasks | 13 files | +| Phase 01 P03 | 10min | 2 tasks | 12 files | ## Accumulated Context @@ -65,6 +66,9 @@ Recent decisions affecting current work: - [Roadmap]: Web UI deferred to v2 - [Phase 01]: Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config - [Phase 01]: Created MetricsSnapshot record in core module (cameleer3-common has no metrics model) +- [Phase 01]: Upgraded testcontainers to 2.0.3 for Docker Desktop 29.x compatibility +- [Phase 01]: Changed error_message/error_stacktrace to non-nullable String for tokenbf_v1 index compat +- [Phase 01]: TTL expressions require toDateTime() cast for DateTime64 columns in ClickHouse 25.3 ### Pending Todos @@ -79,6 +83,6 @@ None yet. ## Session Continuity -Last session: 2026-03-11T10:50:59.915Z -Stopped at: Completed 01-01-PLAN.md +Last session: 2026-03-11T11:03:08.000Z +Stopped at: Completed 01-03-PLAN.md Resume file: None diff --git a/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md b/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md new file mode 100644 index 00000000..8e663ecc --- /dev/null +++ b/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md @@ -0,0 +1,161 @@ +--- +phase: 01-ingestion-pipeline-api-foundation +plan: 03 +subsystem: api +tags: [spring-boot, testcontainers, openapi, interceptor, clickhouse, integration-tests] + +requires: + - phase: 01-01 + provides: ClickHouse schema, application.yml, dependencies + +provides: + - AbstractClickHouseIT base class for all integration tests + - ProtocolVersionInterceptor enforcing X-Cameleer-Protocol-Version:1 on data/agent paths + - WebConfig with interceptor registration and path exclusions + - Cameleer3ServerApplication with @EnableScheduling and component scanning + - 12 passing integration tests (health, OpenAPI, protocol version, forward compat, TTL) + +affects: [01-02, 02-search, 03-agent-registry] + +tech-stack: + added: [testcontainers 2.0.3, docker-java 3.7.0] + patterns: [shared static Testcontainers ClickHouse container, HandlerInterceptor for protocol validation, WebMvcConfigurer for path-based interceptor registration] + +key-files: + created: + - cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java + - cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java + - cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java + - cameleer3-server-app/src/test/resources/application-test.yml + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java + - cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java + modified: + - cameleer3-server-app/pom.xml + - pom.xml + - clickhouse/init/01-schema.sql + +key-decisions: + - "Upgraded testcontainers from 1.20.5 to 2.0.3 for Docker Desktop 29.x compatibility (docker-java 3.7.0)" + - "Removed junit-jupiter dependency; manual container lifecycle via static initializer instead" + - "Changed error_message/error_stacktrace from Nullable(String) to String DEFAULT '' for tokenbf_v1 skip index compatibility" + - "TTL expressions use toDateTime() cast for DateTime64 columns in ClickHouse 25.3" + +patterns-established: + - "AbstractClickHouseIT: static container shared across test classes, @DynamicPropertySource for datasource, @BeforeAll for schema init" + - "ProtocolVersionInterceptor: all data/agent endpoints require X-Cameleer-Protocol-Version:1 header" + - "Path exclusions: health, api-docs, swagger-ui bypass protocol version check" + +requirements-completed: [API-01, API-02, API-03, API-04, API-05, INGST-06] + +duration: 10min +completed: 2026-03-11 +--- + +# Phase 1 Plan 03: API Foundation Summary + +**Protocol version interceptor, health/OpenAPI endpoints, Testcontainers IT base class, and 12 green integration tests covering TTL, forward compat, and interceptor exclusions** + +## Performance + +- **Duration:** 10 min +- **Started:** 2026-03-11T10:52:21Z +- **Completed:** 2026-03-11T11:03:08Z +- **Tasks:** 2 +- **Files modified:** 12 + +## Accomplishments +- ProtocolVersionInterceptor validates X-Cameleer-Protocol-Version:1 on /api/v1/data/** and /api/v1/agents/** paths, returning 400 JSON error for missing or wrong version +- AbstractClickHouseIT base class with Testcontainers ClickHouse 25.3, shared static container, schema init from 01-schema.sql +- 12 integration tests: health endpoint (2), OpenAPI docs (2), protocol version enforcement (5), forward compatibility (1), TTL verification (2) +- Cameleer3ServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Test infrastructure, protocol version interceptor, WebConfig, and app bootstrap** - `b8a4739` (feat) +2. **Task 2: Integration tests for health, OpenAPI, protocol version, forward compat, and TTL** - `2d3fde3` (test) + +## Files Created/Modified +- `cameleer3-server-app/src/main/java/.../Cameleer3ServerApplication.java` - Spring Boot entry point with scheduling and config properties +- `cameleer3-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java` - Validates protocol version header on data/agent paths +- `cameleer3-server-app/src/main/java/.../config/WebConfig.java` - Registers interceptor with path patterns and exclusions +- `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` - Shared Testcontainers base class for ITs +- `cameleer3-server-app/src/test/resources/application-test.yml` - Test profile with small buffer config +- `cameleer3-server-app/src/test/java/.../controller/HealthControllerIT.java` - Health endpoint and TTL tests +- `cameleer3-server-app/src/test/java/.../controller/OpenApiIT.java` - OpenAPI and Swagger UI tests +- `cameleer3-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java` - Protocol header enforcement tests +- `cameleer3-server-app/src/test/java/.../controller/ForwardCompatIT.java` - Unknown JSON fields test +- `pom.xml` - Override testcontainers.version to 2.0.3 +- `cameleer3-server-app/pom.xml` - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3 +- `clickhouse/init/01-schema.sql` - Fix TTL expressions and error column types + +## Decisions Made +- Upgraded Testcontainers to 2.0.3 because docker-java 3.4.1 (from TC 1.20.5) is incompatible with Docker Desktop 29.x API 1.52 +- Removed junit-jupiter module (doesn't exist in TC 2.x); manage container lifecycle via static initializer block instead +- Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT '' because ClickHouse tokenbf_v1 skip indexes require non-nullable String columns +- Added toDateTime() cast in TTL expressions because ClickHouse 25.3 requires DateTime (not DateTime64) in TTL column references + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Fixed testcontainers junit-jupiter dependency not available in 2.0.x** +- **Found during:** Task 2 (compilation) +- **Issue:** org.testcontainers:junit-jupiter:2.0.2 does not exist in Maven Central +- **Fix:** Removed junit-jupiter dependency, upgraded to TC 2.0.3, managed container lifecycle manually via static initializer +- **Files modified:** cameleer3-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java +- **Verification:** All tests compile and pass +- **Committed in:** 2d3fde3 + +**2. [Rule 3 - Blocking] Fixed Docker Desktop 29.x incompatibility with Testcontainers** +- **Found during:** Task 2 (test execution) +- **Issue:** docker-java 3.4.1 (from TC 1.20.5) sends unversioned API calls to Docker API 1.52, receiving 400 errors +- **Fix:** Override testcontainers.version to 2.0.3 in parent POM (brings docker-java 3.7.0 with proper API version negotiation) +- **Files modified:** pom.xml +- **Verification:** ClickHouse container starts successfully +- **Committed in:** 2d3fde3 + +**3. [Rule 1 - Bug] Fixed ClickHouse TTL expression for DateTime64 columns** +- **Found during:** Task 2 (schema init in tests) +- **Issue:** ClickHouse 25.3 requires TTL expressions to resolve to DateTime, not DateTime64(3, 'UTC') +- **Fix:** Changed `TTL start_time + INTERVAL 30 DAY` to `TTL toDateTime(start_time) + toIntervalDay(30)` +- **Files modified:** clickhouse/init/01-schema.sql +- **Verification:** Schema creates without errors in ClickHouse 25.3 container +- **Committed in:** 2d3fde3 + +**4. [Rule 1 - Bug] Fixed tokenbf_v1 index on Nullable column** +- **Found during:** Task 2 (schema init in tests) +- **Issue:** ClickHouse 25.3 does not allow tokenbf_v1 skip indexes on Nullable(String) columns +- **Fix:** Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT '' +- **Files modified:** clickhouse/init/01-schema.sql +- **Verification:** Schema creates without errors, all 12 tests pass +- **Committed in:** 2d3fde3 + +--- + +**Total deviations:** 4 auto-fixed (2 blocking, 2 bugs) +**Impact on plan:** All fixes necessary for test execution on current Docker Desktop and ClickHouse versions. No scope creep. + +## Issues Encountered +- Surefire runs from module directory, not project root; fixed schema path lookup in AbstractClickHouseIT to check both locations + +## User Setup Required +None - no external service configuration required. + +## Next Phase Readiness +- AbstractClickHouseIT base class ready for all future integration tests +- Protocol version interceptor active for data/agent endpoints +- API foundation complete, ready for Plan 02 (REST controllers, ClickHouse repositories, flush scheduler) +- Health endpoint at /api/v1/health, OpenAPI at /api/v1/api-docs, Swagger UI at /api/v1/swagger-ui + +## Self-Check: PASSED + +All 9 created files verified present. Both task commits verified in git log. + +--- +*Phase: 01-ingestion-pipeline-api-foundation* +*Completed: 2026-03-11*