cameleer/cameleer-server
1
0
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(01-03): complete API foundation plan

Browse Source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
hsiegeln
2026-03-11 12:05:02 +01:00
parent 2d3fde3766
commit ff0af0ef2f
4 changed files with 182 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.planning/REQUIREMENTS.md
View File

@@ -51,11 +51,11 @@ Requirements for initial release. Each maps to roadmap phases. Tracked as Gitea
### REST API ### REST API
- [ ] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28) - [x] **API-01**: All endpoints follow the protocol v1 path structure (`/api/v1/...`) (#28)
- [ ] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29) - [x] **API-02**: API documented via OpenAPI/Swagger (springdoc-openapi) (#29)
- [ ] **API-03**: Server includes `GET /api/v1/health` endpoint (#30) - [x] **API-03**: Server includes `GET /api/v1/health` endpoint (#30)
- [ ] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31) - [x] **API-04**: All requests validated for `X-Cameleer-Protocol-Version: 1` header (#31)
- [ ] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32) - [x] **API-05**: Server accepts unknown JSON fields for forward compatibility (#32)
## v2 Requirements ## v2 Requirements

4
.planning/ROADMAP.md
View File

@@ -29,7 +29,7 @@ Decimal phases appear between their surrounding integers in numeric order.
3. When the write buffer is full, the server returns 503 and does not lose already-buffered data 3. When the write buffer is full, the server returns 503 and does not lose already-buffered data
4. Data older than the configured TTL (default 30 days) is automatically removed by ClickHouse 4. Data older than the configured TTL (default 30 days) is automatically removed by ClickHouse
5. The health endpoint responds at `/api/v1/health`, OpenAPI docs are available, protocol version header is validated, and unknown JSON fields are accepted 5. The health endpoint responds at `/api/v1/health`, OpenAPI docs are available, protocol version header is validated, and unknown JSON fields are accepted
**Plans:** 3 plans **Plans:** 2/3 plans executed
Plans: Plans:
- [ ] 01-01-PLAN.md -- ClickHouse infrastructure, schema, WriteBuffer, repository interfaces, test infrastructure - [ ] 01-01-PLAN.md -- ClickHouse infrastructure, schema, WriteBuffer, repository interfaces, test infrastructure
@@ -87,7 +87,7 @@ Note: Phases 2 and 3 both depend only on Phase 1 and could execute in parallel.
| Phase | Plans Complete | Status | Completed | | Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------| |-------|----------------|--------|-----------|
| 1. Ingestion Pipeline + API Foundation | 0/3 | Not started | - | | 1. Ingestion Pipeline + API Foundation | 2/3 | In Progress| |
| 2. Transaction Search + Diagrams | 0/2 | Not started | - | | 2. Transaction Search + Diagrams | 0/2 | Not started | - |
| 3. Agent Registry + SSE Push | 0/2 | Not started | - | | 3. Agent Registry + SSE Push | 0/2 | Not started | - |
| 4. Security | 0/1 | Not started | - | | 4. Security | 0/1 | Not started | - |

24
.planning/STATE.md
View File

@@ -3,15 +3,15 @@ gsd_state_version: 1.0
milestone: v1.0 milestone: v1.0
milestone_name: milestone milestone_name: milestone
status: executing status: executing
stopped_at: Completed 01-01-PLAN.md stopped_at: Completed 01-03-PLAN.md
last_updated: "2026-03-11T10:50:59.918Z" last_updated: "2026-03-11T11:03:08.000Z"
last_activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer) last_activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests)
progress: progress:
total_phases: 4 total_phases: 4
completed_phases: 0 completed_phases: 0
total_plans: 3 total_plans: 3
completed_plans: 1 completed_plans: 2
percent: 33 percent: 66
--- ---
# Project State # Project State
@@ -26,11 +26,11 @@ See: .planning/PROJECT.md (updated 2026-03-11)
## Current Position ## Current Position
Phase: 1 of 4 (Ingestion Pipeline + API Foundation) Phase: 1 of 4 (Ingestion Pipeline + API Foundation)
Plan: 1 of 3 in current phase Plan: 2 of 3 in current phase
Status: Executing Status: Executing
Last activity: 2026-03-11 -- Completed 01-01 (ClickHouse infrastructure and WriteBuffer) Last activity: 2026-03-11 -- Completed 01-03 (API foundation, protocol interceptor, integration tests)
Progress: [███░░░░░░░] 33% Progress: [██████░░░░] 66%
## Performance Metrics ## Performance Metrics
@@ -51,6 +51,7 @@ Progress: [███░░░░░░░] 33%
*Updated after each plan completion* *Updated after each plan completion*
| Phase 01 P01 | 3min | 2 tasks | 13 files | | Phase 01 P01 | 3min | 2 tasks | 13 files |
| Phase 01 P03 | 10min | 2 tasks | 12 files |
## Accumulated Context ## Accumulated Context
@@ -65,6 +66,9 @@ Recent decisions affecting current work:
- [Roadmap]: Web UI deferred to v2 - [Roadmap]: Web UI deferred to v2
- [Phase 01]: Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config - [Phase 01]: Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config
- [Phase 01]: Created MetricsSnapshot record in core module (cameleer3-common has no metrics model) - [Phase 01]: Created MetricsSnapshot record in core module (cameleer3-common has no metrics model)
- [Phase 01]: Upgraded testcontainers to 2.0.3 for Docker Desktop 29.x compatibility
- [Phase 01]: Changed error_message/error_stacktrace to non-nullable String for tokenbf_v1 index compat
- [Phase 01]: TTL expressions require toDateTime() cast for DateTime64 columns in ClickHouse 25.3
### Pending Todos ### Pending Todos
@@ -79,6 +83,6 @@ None yet.
## Session Continuity ## Session Continuity
Last session: 2026-03-11T10:50:59.915Z Last session: 2026-03-11T11:03:08.000Z
Stopped at: Completed 01-01-PLAN.md Stopped at: Completed 01-03-PLAN.md
Resume file: None Resume file: None

161
.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md Normal file
View File

@@ -0,0 +1,161 @@
---
phase: 01-ingestion-pipeline-api-foundation
plan: 03
subsystem: api
tags: [spring-boot, testcontainers, openapi, interceptor, clickhouse, integration-tests]
requires:
- phase: 01-01
provides: ClickHouse schema, application.yml, dependencies
provides:
- AbstractClickHouseIT base class for all integration tests
- ProtocolVersionInterceptor enforcing X-Cameleer-Protocol-Version:1 on data/agent paths
- WebConfig with interceptor registration and path exclusions
- Cameleer3ServerApplication with @EnableScheduling and component scanning
- 12 passing integration tests (health, OpenAPI, protocol version, forward compat, TTL)
affects: [01-02, 02-search, 03-agent-registry]
tech-stack:
added: [testcontainers 2.0.3, docker-java 3.7.0]
patterns: [shared static Testcontainers ClickHouse container, HandlerInterceptor for protocol validation, WebMvcConfigurer for path-based interceptor registration]
key-files:
created:
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java
- cameleer3-server-app/src/test/resources/application-test.yml
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
modified:
- cameleer3-server-app/pom.xml
- pom.xml
- clickhouse/init/01-schema.sql
key-decisions:
- "Upgraded testcontainers from 1.20.5 to 2.0.3 for Docker Desktop 29.x compatibility (docker-java 3.7.0)"
- "Removed junit-jupiter dependency; manual container lifecycle via static initializer instead"
- "Changed error_message/error_stacktrace from Nullable(String) to String DEFAULT '' for tokenbf_v1 skip index compatibility"
- "TTL expressions use toDateTime() cast for DateTime64 columns in ClickHouse 25.3"
patterns-established:
- "AbstractClickHouseIT: static container shared across test classes, @DynamicPropertySource for datasource, @BeforeAll for schema init"
- "ProtocolVersionInterceptor: all data/agent endpoints require X-Cameleer-Protocol-Version:1 header"
- "Path exclusions: health, api-docs, swagger-ui bypass protocol version check"
requirements-completed: [API-01, API-02, API-03, API-04, API-05, INGST-06]
duration: 10min
completed: 2026-03-11
---
# Phase 1 Plan 03: API Foundation Summary
**Protocol version interceptor, health/OpenAPI endpoints, Testcontainers IT base class, and 12 green integration tests covering TTL, forward compat, and interceptor exclusions**
## Performance
- **Duration:** 10 min
- **Started:** 2026-03-11T10:52:21Z
- **Completed:** 2026-03-11T11:03:08Z
- **Tasks:** 2
- **Files modified:** 12
## Accomplishments
- ProtocolVersionInterceptor validates X-Cameleer-Protocol-Version:1 on /api/v1/data/** and /api/v1/agents/** paths, returning 400 JSON error for missing or wrong version
- AbstractClickHouseIT base class with Testcontainers ClickHouse 25.3, shared static container, schema init from 01-schema.sql
- 12 integration tests: health endpoint (2), OpenAPI docs (2), protocol version enforcement (5), forward compatibility (1), TTL verification (2)
- Cameleer3ServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning
## Task Commits
Each task was committed atomically:
1. **Task 1: Test infrastructure, protocol version interceptor, WebConfig, and app bootstrap** - `b8a4739` (feat)
2. **Task 2: Integration tests for health, OpenAPI, protocol version, forward compat, and TTL** - `2d3fde3` (test)
## Files Created/Modified
- `cameleer3-server-app/src/main/java/.../Cameleer3ServerApplication.java` - Spring Boot entry point with scheduling and config properties
- `cameleer3-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java` - Validates protocol version header on data/agent paths
- `cameleer3-server-app/src/main/java/.../config/WebConfig.java` - Registers interceptor with path patterns and exclusions
- `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` - Shared Testcontainers base class for ITs
- `cameleer3-server-app/src/test/resources/application-test.yml` - Test profile with small buffer config
- `cameleer3-server-app/src/test/java/.../controller/HealthControllerIT.java` - Health endpoint and TTL tests
- `cameleer3-server-app/src/test/java/.../controller/OpenApiIT.java` - OpenAPI and Swagger UI tests
- `cameleer3-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java` - Protocol header enforcement tests
- `cameleer3-server-app/src/test/java/.../controller/ForwardCompatIT.java` - Unknown JSON fields test
- `pom.xml` - Override testcontainers.version to 2.0.3
- `cameleer3-server-app/pom.xml` - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3
- `clickhouse/init/01-schema.sql` - Fix TTL expressions and error column types
## Decisions Made
- Upgraded Testcontainers to 2.0.3 because docker-java 3.4.1 (from TC 1.20.5) is incompatible with Docker Desktop 29.x API 1.52
- Removed junit-jupiter module (doesn't exist in TC 2.x); manage container lifecycle via static initializer block instead
- Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT '' because ClickHouse tokenbf_v1 skip indexes require non-nullable String columns
- Added toDateTime() cast in TTL expressions because ClickHouse 25.3 requires DateTime (not DateTime64) in TTL column references
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 3 - Blocking] Fixed testcontainers junit-jupiter dependency not available in 2.0.x**
- **Found during:** Task 2 (compilation)
- **Issue:** org.testcontainers:junit-jupiter:2.0.2 does not exist in Maven Central
- **Fix:** Removed junit-jupiter dependency, upgraded to TC 2.0.3, managed container lifecycle manually via static initializer
- **Files modified:** cameleer3-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java
- **Verification:** All tests compile and pass
- **Committed in:** 2d3fde3
**2. [Rule 3 - Blocking] Fixed Docker Desktop 29.x incompatibility with Testcontainers**
- **Found during:** Task 2 (test execution)
- **Issue:** docker-java 3.4.1 (from TC 1.20.5) sends unversioned API calls to Docker API 1.52, receiving 400 errors
- **Fix:** Override testcontainers.version to 2.0.3 in parent POM (brings docker-java 3.7.0 with proper API version negotiation)
- **Files modified:** pom.xml
- **Verification:** ClickHouse container starts successfully
- **Committed in:** 2d3fde3
**3. [Rule 1 - Bug] Fixed ClickHouse TTL expression for DateTime64 columns**
- **Found during:** Task 2 (schema init in tests)
- **Issue:** ClickHouse 25.3 requires TTL expressions to resolve to DateTime, not DateTime64(3, 'UTC')
- **Fix:** Changed `TTL start_time + INTERVAL 30 DAY` to `TTL toDateTime(start_time) + toIntervalDay(30)`
- **Files modified:** clickhouse/init/01-schema.sql
- **Verification:** Schema creates without errors in ClickHouse 25.3 container
- **Committed in:** 2d3fde3
**4. [Rule 1 - Bug] Fixed tokenbf_v1 index on Nullable column**
- **Found during:** Task 2 (schema init in tests)
- **Issue:** ClickHouse 25.3 does not allow tokenbf_v1 skip indexes on Nullable(String) columns
- **Fix:** Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT ''
- **Files modified:** clickhouse/init/01-schema.sql
- **Verification:** Schema creates without errors, all 12 tests pass
- **Committed in:** 2d3fde3
---
**Total deviations:** 4 auto-fixed (2 blocking, 2 bugs)
**Impact on plan:** All fixes necessary for test execution on current Docker Desktop and ClickHouse versions. No scope creep.
## Issues Encountered
- Surefire runs from module directory, not project root; fixed schema path lookup in AbstractClickHouseIT to check both locations
## User Setup Required
None - no external service configuration required.
## Next Phase Readiness
- AbstractClickHouseIT base class ready for all future integration tests
- Protocol version interceptor active for data/agent endpoints
- API foundation complete, ready for Plan 02 (REST controllers, ClickHouse repositories, flush scheduler)
- Health endpoint at /api/v1/health, OpenAPI at /api/v1/api-docs, Swagger UI at /api/v1/swagger-ui
## Self-Check: PASSED
All 9 created files verified present. Both task commits verified in git log.
---
*Phase: 01-ingestion-pipeline-api-foundation*
*Completed: 2026-03-11*