cameleer/cameleer-server
1
0
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
cb3ebfea7c11e9ca6ae64a6f61e5bc80d7e5be6a
cameleer-server/.planning/phases/01-ingestion-pipeline-api-foundation/01-01-SUMMARY.md
hsiegeln cb3ebfea7c
Some checks failed
CI / cleanup-branch (push) Has been skipped
Details
CI / build (push) Failing after 18s
Details
CI / docker (push) Has been skipped
Details
CI / deploy (push) Has been skipped
Details
CI / deploy-feature (push) Has been skipped
Details
chore: rename cameleer3 to cameleer 
Rename Java packages from com.cameleer3 to com.cameleer, module
directories from cameleer3-* to cameleer-*, and all references
throughout workflows, Dockerfiles, docs, migrations, and pom.xml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:28:42 +02:00

6.1 KiB
Raw Blame History

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, duration, completed
phase plan subsystem tags requires provides affects tech-stack key-files key-decisions patterns-established requirements-completed duration completed
01-ingestion-pipeline-api-foundation 01 database
clickhouse
jdbc
docker-compose
write-buffer
backpressure
phase provides
none greenfield project skeleton
ClickHouse Docker Compose for local development
ClickHouse DDL with route_executions, route_diagrams, agent_metrics tables
WriteBuffer<T> generic bounded buffer with backpressure signal
ExecutionRepository, DiagramRepository, MetricsRepository interfaces
IngestionConfig and ClickHouseConfig Spring configuration
Application.yml with datasource, ingestion, springdoc, actuator config
01-02
01-03
02-search
03-agent-registry
added patterns
clickhouse-jdbc 0.9.7
springdoc-openapi 2.8.6
spring-boot-starter-actuator
spring-boot-starter-jdbc
testcontainers-clickhouse 2.0.2
awaitility
slf4j-api
ArrayBlockingQueue write buffer with offer/drain
all-or-nothing offerBatch
content-hash dedup for diagrams
daily partitioning with TTL + ttl_only_drop_parts
created modified
docker-compose.yml
clickhouse/init/01-schema.sql
cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/DiagramRepository.java
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/MetricsRepository.java
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/MetricsSnapshot.java
cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java
cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/WriteBufferTest.java
cameleer-server-core/pom.xml
cameleer-server-app/pom.xml
cameleer-server-app/src/main/resources/application.yml
Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config rather than manual DataSource
Created MetricsSnapshot record in core module since cameleer-common has no metrics model
ClickHouseConfig exposes JdbcTemplate bean; relies on Spring Boot DataSource auto-config
WriteBuffer pattern: ArrayBlockingQueue with offer()/offerBatch()/drain() for decoupling HTTP from ClickHouse
Repository interfaces in core module, implementations will go in app module
ClickHouse schema: DateTime64(3, 'UTC') for all timestamps, daily partitioning, TTL with ttl_only_drop_parts
INGST-04
INGST-05
INGST-06
3min 2026-03-11

Phase 1 Plan 01: ClickHouse Infrastructure and WriteBuffer Summary

ClickHouse schema with three tables (daily partitioned, TTL), Docker Compose, WriteBuffer with backpressure, and repository interfaces

Performance

  • Duration: 3 min
  • Started: 2026-03-11T10:45:57Z
  • Completed: 2026-03-11T10:49:47Z
  • Tasks: 2
  • Files modified: 13

Accomplishments

  • ClickHouse DDL with route_executions (MergeTree, bloom_filter + tokenbf_v1 skip indexes), route_diagrams (ReplacingMergeTree), agent_metrics (MergeTree with TTL)
  • Generic WriteBuffer with all-or-nothing batch semantics and 10 passing unit tests
  • Repository interfaces defining batch insert contracts for executions, diagrams, and metrics
  • Full application.yml with datasource, ingestion buffer config, springdoc, and actuator health endpoint

Task Commits

Each task was committed atomically:

  1. Task 1: Dependencies, Docker Compose, ClickHouse schema, and application config - 96c52b8 (feat)
  2. Task 2 RED: WriteBuffer failing tests - f37009e (test)
  3. Task 2 GREEN: WriteBuffer, repository interfaces, config classes - cc1c082 (feat)

Files Created/Modified

  • docker-compose.yml - ClickHouse service with ports 8123/9000, init volume mount
  • clickhouse/init/01-schema.sql - DDL for route_executions, route_diagrams, agent_metrics
  • cameleer-server-core/src/main/java/.../ingestion/WriteBuffer.java - Bounded queue with offer/offerBatch/drain
  • cameleer-server-core/src/main/java/.../storage/ExecutionRepository.java - Batch insert interface for RouteExecution
  • cameleer-server-core/src/main/java/.../storage/DiagramRepository.java - Store/find interface for RouteGraph
  • cameleer-server-core/src/main/java/.../storage/MetricsRepository.java - Batch insert interface for MetricsSnapshot
  • cameleer-server-core/src/main/java/.../storage/model/MetricsSnapshot.java - Metrics data record
  • cameleer-server-app/src/main/java/.../config/IngestionConfig.java - Buffer capacity, batch size, flush interval
  • cameleer-server-app/src/main/java/.../config/ClickHouseConfig.java - JdbcTemplate bean
  • cameleer-server-core/src/test/java/.../ingestion/WriteBufferTest.java - 10 unit tests for WriteBuffer
  • cameleer-server-core/pom.xml - Added slf4j-api
  • cameleer-server-app/pom.xml - Added clickhouse-jdbc, springdoc, actuator, testcontainers, awaitility
  • cameleer-server-app/src/main/resources/application.yml - Full config with datasource, ingestion, springdoc, actuator

Decisions Made

  • Used spring-boot-starter-jdbc to get JdbcTemplate and HikariCP auto-configuration rather than manually wiring a DataSource
  • Created MetricsSnapshot record in core module since cameleer-common does not include a metrics model
  • ClickHouseConfig is minimal -- relies on Spring Boot auto-configuring DataSource from spring.datasource properties

Deviations from Plan

None - plan executed exactly as written.

Issues Encountered

None

User Setup Required

None - no external service configuration required.

Next Phase Readiness

  • ClickHouse infrastructure ready for Plan 02 (REST controllers + flush scheduler)
  • WriteBuffer and repository interfaces ready for implementation wiring
  • Docker Compose available for local development: docker compose up -d

Self-Check: PASSED

All 10 created files verified present. All 3 task commits verified in git log.

Phase: 01-ingestion-pipeline-api-foundation Completed: 2026-03-11

Reference in New Issue View Git Blame Copy Permalink