cameleer/cameleer-server
1
0
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ff0af0ef2f2494315c9fd9e6f8562404a8733726
cameleer-server/.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md
hsiegeln ff0af0ef2f docs(01-03): complete API foundation plan 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 12:05:02 +01:00

8.7 KiB
Raw Blame History

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, duration, completed
phase plan subsystem tags requires provides affects tech-stack key-files key-decisions patterns-established requirements-completed duration completed
01-ingestion-pipeline-api-foundation 03 api
spring-boot
testcontainers
openapi
interceptor
clickhouse
integration-tests
phase provides
01-01 ClickHouse schema, application.yml, dependencies
AbstractClickHouseIT base class for all integration tests
ProtocolVersionInterceptor enforcing X-Cameleer-Protocol-Version:1 on data/agent paths
WebConfig with interceptor registration and path exclusions
Cameleer3ServerApplication with @EnableScheduling and component scanning
12 passing integration tests (health, OpenAPI, protocol version, forward compat, TTL)
01-02
02-search
03-agent-registry
added patterns
testcontainers 2.0.3
docker-java 3.7.0
shared static Testcontainers ClickHouse container
HandlerInterceptor for protocol validation
WebMvcConfigurer for path-based interceptor registration
created modified
cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java
cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java
cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java
cameleer3-server-app/src/test/resources/application-test.yml
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java
cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
cameleer3-server-app/pom.xml
pom.xml
clickhouse/init/01-schema.sql
Upgraded testcontainers from 1.20.5 to 2.0.3 for Docker Desktop 29.x compatibility (docker-java 3.7.0)
Removed junit-jupiter dependency; manual container lifecycle via static initializer instead
Changed error_message/error_stacktrace from Nullable(String) to String DEFAULT '' for tokenbf_v1 skip index compatibility
TTL expressions use toDateTime() cast for DateTime64 columns in ClickHouse 25.3
AbstractClickHouseIT: static container shared across test classes, @DynamicPropertySource for datasource, @BeforeAll for schema init
ProtocolVersionInterceptor: all data/agent endpoints require X-Cameleer-Protocol-Version:1 header
Path exclusions: health, api-docs, swagger-ui bypass protocol version check
API-01
API-02
API-03
API-04
API-05
INGST-06
10min 2026-03-11

Phase 1 Plan 03: API Foundation Summary

Protocol version interceptor, health/OpenAPI endpoints, Testcontainers IT base class, and 12 green integration tests covering TTL, forward compat, and interceptor exclusions

Performance

  • Duration: 10 min
  • Started: 2026-03-11T10:52:21Z
  • Completed: 2026-03-11T11:03:08Z
  • Tasks: 2
  • Files modified: 12

Accomplishments

  • ProtocolVersionInterceptor validates X-Cameleer-Protocol-Version:1 on /api/v1/data/** and /api/v1/agents/** paths, returning 400 JSON error for missing or wrong version
  • AbstractClickHouseIT base class with Testcontainers ClickHouse 25.3, shared static container, schema init from 01-schema.sql
  • 12 integration tests: health endpoint (2), OpenAPI docs (2), protocol version enforcement (5), forward compatibility (1), TTL verification (2)
  • Cameleer3ServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning

Task Commits

Each task was committed atomically:

  1. Task 1: Test infrastructure, protocol version interceptor, WebConfig, and app bootstrap - b8a4739 (feat)
  2. Task 2: Integration tests for health, OpenAPI, protocol version, forward compat, and TTL - 2d3fde3 (test)

Files Created/Modified

  • cameleer3-server-app/src/main/java/.../Cameleer3ServerApplication.java - Spring Boot entry point with scheduling and config properties
  • cameleer3-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java - Validates protocol version header on data/agent paths
  • cameleer3-server-app/src/main/java/.../config/WebConfig.java - Registers interceptor with path patterns and exclusions
  • cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java - Shared Testcontainers base class for ITs
  • cameleer3-server-app/src/test/resources/application-test.yml - Test profile with small buffer config
  • cameleer3-server-app/src/test/java/.../controller/HealthControllerIT.java - Health endpoint and TTL tests
  • cameleer3-server-app/src/test/java/.../controller/OpenApiIT.java - OpenAPI and Swagger UI tests
  • cameleer3-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java - Protocol header enforcement tests
  • cameleer3-server-app/src/test/java/.../controller/ForwardCompatIT.java - Unknown JSON fields test
  • pom.xml - Override testcontainers.version to 2.0.3
  • cameleer3-server-app/pom.xml - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3
  • clickhouse/init/01-schema.sql - Fix TTL expressions and error column types

Decisions Made

  • Upgraded Testcontainers to 2.0.3 because docker-java 3.4.1 (from TC 1.20.5) is incompatible with Docker Desktop 29.x API 1.52
  • Removed junit-jupiter module (doesn't exist in TC 2.x); manage container lifecycle via static initializer block instead
  • Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT '' because ClickHouse tokenbf_v1 skip indexes require non-nullable String columns
  • Added toDateTime() cast in TTL expressions because ClickHouse 25.3 requires DateTime (not DateTime64) in TTL column references

Deviations from Plan

Auto-fixed Issues

1. [Rule 3 - Blocking] Fixed testcontainers junit-jupiter dependency not available in 2.0.x

  • Found during: Task 2 (compilation)
  • Issue: org.testcontainers:junit-jupiter:2.0.2 does not exist in Maven Central
  • Fix: Removed junit-jupiter dependency, upgraded to TC 2.0.3, managed container lifecycle manually via static initializer
  • Files modified: cameleer3-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java
  • Verification: All tests compile and pass
  • Committed in: 2d3fde3

2. [Rule 3 - Blocking] Fixed Docker Desktop 29.x incompatibility with Testcontainers

  • Found during: Task 2 (test execution)
  • Issue: docker-java 3.4.1 (from TC 1.20.5) sends unversioned API calls to Docker API 1.52, receiving 400 errors
  • Fix: Override testcontainers.version to 2.0.3 in parent POM (brings docker-java 3.7.0 with proper API version negotiation)
  • Files modified: pom.xml
  • Verification: ClickHouse container starts successfully
  • Committed in: 2d3fde3

3. [Rule 1 - Bug] Fixed ClickHouse TTL expression for DateTime64 columns

  • Found during: Task 2 (schema init in tests)
  • Issue: ClickHouse 25.3 requires TTL expressions to resolve to DateTime, not DateTime64(3, 'UTC')
  • Fix: Changed TTL start_time + INTERVAL 30 DAY to TTL toDateTime(start_time) + toIntervalDay(30)
  • Files modified: clickhouse/init/01-schema.sql
  • Verification: Schema creates without errors in ClickHouse 25.3 container
  • Committed in: 2d3fde3

4. [Rule 1 - Bug] Fixed tokenbf_v1 index on Nullable column

  • Found during: Task 2 (schema init in tests)
  • Issue: ClickHouse 25.3 does not allow tokenbf_v1 skip indexes on Nullable(String) columns
  • Fix: Changed error_message and error_stacktrace from Nullable(String) to String DEFAULT ''
  • Files modified: clickhouse/init/01-schema.sql
  • Verification: Schema creates without errors, all 12 tests pass
  • Committed in: 2d3fde3

Total deviations: 4 auto-fixed (2 blocking, 2 bugs) Impact on plan: All fixes necessary for test execution on current Docker Desktop and ClickHouse versions. No scope creep.

Issues Encountered

  • Surefire runs from module directory, not project root; fixed schema path lookup in AbstractClickHouseIT to check both locations

User Setup Required

None - no external service configuration required.

Next Phase Readiness

  • AbstractClickHouseIT base class ready for all future integration tests
  • Protocol version interceptor active for data/agent endpoints
  • API foundation complete, ready for Plan 02 (REST controllers, ClickHouse repositories, flush scheduler)
  • Health endpoint at /api/v1/health, OpenAPI at /api/v1/api-docs, Swagger UI at /api/v1/swagger-ui

Self-Check: PASSED

All 9 created files verified present. Both task commits verified in git log.

Phase: 01-ingestion-pipeline-api-foundation Completed: 2026-03-11

Reference in New Issue View Git Blame Copy Permalink