Files

hsiegeln 7f0ceca8b1 docs(01-02): complete ingestion endpoints plan

- SUMMARY.md with 14 files, 3 commits, 11 integration tests
- STATE.md: Phase 1 complete (3/3 plans)
- ROADMAP.md: Phase 01 progress updated
- REQUIREMENTS.md: INGST-01, INGST-02, INGST-03 marked complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-11 12:16:46 +01:00

8.2 KiB

Raw Blame History

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, duration, completed

phase

plan

subsystem

tags

requires

provides

affects

tech-stack

key-files

key-decisions

patterns-established

requirements-completed

duration

completed

01-ingestion-pipeline-api-foundation

api

rest

ingestion

clickhouse

jdbc

backpressure

spring-boot

integration-tests

phase	provides
01-01	WriteBuffer, repository interfaces, IngestionConfig, ClickHouse schema

phase	provides
01-03	AbstractClickHouseIT base class, ProtocolVersionInterceptor, application bootstrap

IngestionService routing data to WriteBuffer instances

ClickHouseExecutionRepository with batch insert and parallel processor arrays

ClickHouseDiagramRepository with SHA-256 content-hash deduplication

ClickHouseMetricsRepository with batch insert for agent_metrics

ClickHouseFlushScheduler with SmartLifecycle shutdown flush

POST /api/v1/data/executions endpoint (single and array)

POST /api/v1/data/diagrams endpoint (single and array)

POST /api/v1/data/metrics endpoint (array)

Backpressure
503 with Retry-After when buffer full

11 integration tests verifying end-to-end ingestion pipeline

02-search

03-agent-registry

added

patterns

single/array JSON payload parsing via raw String body

SmartLifecycle for graceful shutdown flush

BatchPreparedStatementSetter for ClickHouse batch inserts

SHA-256 content-hash dedup for diagrams

created

modified

cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseDiagramRepository.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsRepository.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionBeanConfig.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/DiagramController.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/MetricsController.java

cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ExecutionControllerIT.java

cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/DiagramControllerIT.java

cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/MetricsControllerIT.java

cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/BackpressureIT.java

cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java

Controllers accept raw String body and detect single vs array JSON to support both payload formats

IngestionService is a plain class in core module, wired as a bean by IngestionBeanConfig in app module

Removed @Configuration from IngestionConfig to fix duplicate bean with @EnableConfigurationProperties

Controller pattern: accept raw String body, parse single/array JSON, delegate to IngestionService, return 202 or 503

Repository pattern: BatchPreparedStatementSetter for ClickHouse batch inserts

FlushScheduler pattern: SmartLifecycle for graceful shutdown, loop-drain until empty

Backpressure pattern: WriteBuffer.offer returns false -> controller returns 503 + Retry-After

INGST-01

INGST-02

INGST-03

INGST-05

7min

2026-03-11

Phase 1 Plan 02: Ingestion Endpoints and ClickHouse Pipeline Summary

Three REST ingestion endpoints with ClickHouse batch insert repositories, scheduled flush with graceful shutdown, and 11 green integration tests verifying end-to-end data flow and backpressure

Performance

Duration: 7 min
Started: 2026-03-11T11:06:47Z
Completed: 2026-03-11T11:14:00Z
Tasks: 2
Files modified: 14

Accomplishments

Complete ingestion pipeline: HTTP POST -> IngestionService -> WriteBuffer -> ClickHouseFlushScheduler -> ClickHouse repositories
Three REST endpoints accepting both single and array JSON payloads with 202 Accepted response
Backpressure returns 503 with Retry-After header when write buffer is full
ClickHouse repositories: batch insert for executions (with flattened processor arrays), JSON storage with SHA-256 dedup for diagrams, batch insert for metrics
Graceful shutdown via SmartLifecycle drains all remaining buffered data

Task Commits

Each task was committed atomically:

Task 1: IngestionService, ClickHouse repositories, and flush scheduler - 17a18cf (feat)
Task 2 RED: Failing integration tests for ingestion endpoints - d55ebc1 (test)
Task 2 GREEN: Ingestion REST controllers with backpressure - 8fe65f0 (feat)

Files Created/Modified

cameleer3-server-core/.../ingestion/IngestionService.java - Routes data to WriteBuffer instances
cameleer3-server-app/.../storage/ClickHouseExecutionRepository.java - Batch insert with parallel processor arrays
cameleer3-server-app/.../storage/ClickHouseDiagramRepository.java - JSON storage with SHA-256 content-hash dedup
cameleer3-server-app/.../storage/ClickHouseMetricsRepository.java - Batch insert for agent_metrics
cameleer3-server-app/.../ingestion/ClickHouseFlushScheduler.java - Scheduled drain + SmartLifecycle shutdown
cameleer3-server-app/.../config/IngestionBeanConfig.java - WriteBuffer and IngestionService bean wiring
cameleer3-server-app/.../controller/ExecutionController.java - POST /api/v1/data/executions
cameleer3-server-app/.../controller/DiagramController.java - POST /api/v1/data/diagrams
cameleer3-server-app/.../controller/MetricsController.java - POST /api/v1/data/metrics
cameleer3-server-app/.../config/IngestionConfig.java - Removed @Configuration (fix duplicate bean)
cameleer3-server-app/.../controller/ExecutionControllerIT.java - 4 tests: single, array, flush, unknown fields
cameleer3-server-app/.../controller/DiagramControllerIT.java - 3 tests: single, array, flush
cameleer3-server-app/.../controller/MetricsControllerIT.java - 2 tests: POST, flush
cameleer3-server-app/.../controller/BackpressureIT.java - 2 tests: 503 response, data not lost

Decisions Made

Controllers accept raw String body and detect single vs array JSON (starts with [), supporting both payload formats per protocol spec
IngestionService is a plain class in core module (no Spring annotations), wired as a bean by IngestionBeanConfig in app module
Removed @Configuration from IngestionConfig to fix duplicate bean conflict with @EnableConfigurationProperties

Deviations from Plan

Auto-fixed Issues

1. [Rule 1 - Bug] Fixed duplicate IngestionConfig bean

Found during: Task 2 (integration test context startup)
Issue: IngestionConfig had both @Configuration and @ConfigurationProperties, while @EnableConfigurationProperties(IngestionConfig.class) on the app class created a second bean, causing "expected single matching bean but found 2"
Fix: Removed @Configuration from IngestionConfig, relying solely on @EnableConfigurationProperties
Files modified: cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
Verification: Application context starts successfully, all tests pass
Committed in: 8fe65f0

Total deviations: 1 auto-fixed (1 bug) Impact on plan: Necessary fix for Spring context startup. No scope creep.

Issues Encountered

BackpressureIT initially failed because the scheduled flush drained the buffer before the test could fill it. Fixed by using a 60s flush interval and batch POST to fill buffer atomically.

User Setup Required

None - no external service configuration required.

Next Phase Readiness

All three ingestion endpoints operational and tested
Phase 1 complete: ClickHouse infrastructure, API foundation, and ingestion pipeline all working
Ready for Phase 2 (search) and Phase 3 (agent registry) which both depend only on Phase 1

Self-Check: PASSED

All 13 created files verified present. All 3 task commits verified in git log.

Phase: 01-ingestion-pipeline-api-foundation Completed: 2026-03-11

8.2 KiB Raw Blame History