cameleer/cameleer-server
1
0
Fork 0
Code Issues 45 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: rename cameleer3 to cameleer
Some checks failed
CI / cleanup-branch (push) Has been skipped
Details
CI / build (push) Failing after 18s
Details
CI / docker (push) Has been skipped
Details
CI / deploy (push) Has been skipped
Details
CI / deploy-feature (push) Has been skipped
Details

Browse Source 
Rename Java packages from com.cameleer3 to com.cameleer, module
directories from cameleer3-* to cameleer-*, and all references
throughout workflows, Dockerfiles, docs, migrations, and pom.xml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
hsiegeln
2026-04-15 15:28:42 +02:00
parent 1077293343
commit cb3ebfea7c
569 changed files with 4356 additions and 3245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
.planning/phases/01-ingestion-pipeline-api-foundation/01-01-PLAN.md
View File

@@ -6,18 +6,18 @@ wave: 1
depends_on: []
files_modified:
- pom.xml
- cameleer3-server-core/pom.xml
- cameleer3-server-app/pom.xml
- cameleer-server-core/pom.xml
- cameleer-server-app/pom.xml
- docker-compose.yml
- clickhouse/init/01-schema.sql
- cameleer3-server-app/src/main/resources/application.yml
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java
- cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java
- cameleer-server-app/src/main/resources/application.yml
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/DiagramRepository.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/MetricsRepository.java
- cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/WriteBufferTest.java
autonomous: true
requirements:
- INGST-04
@@ -32,7 +32,7 @@ must_haves:
- "TTL clause on tables removes data older than configured days"
- "Docker Compose starts ClickHouse and initializes the schema"
artifacts:
- path: "cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java"
- path: "cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java"
provides: "Generic bounded write buffer with offer/drain/isFull"
min_lines: 30
- path: "clickhouse/init/01-schema.sql"
@@ -41,15 +41,15 @@ must_haves:
- path: "docker-compose.yml"
provides: "Local ClickHouse service"
contains: "clickhouse-server"
- path: "cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java"
- path: "cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java"
provides: "Repository interface for execution batch inserts"
exports: ["insertBatch"]
key_links:
- from: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java"
- from: "cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java"
to: "application.yml"
via: "spring.datasource properties"
pattern: "spring\\.datasource"
- from: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java"
- from: "cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java"
to: "application.yml"
via: "ingestion.* properties"
pattern: "ingestion\\."
@@ -74,8 +74,8 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
@.planning/phases/01-ingestion-pipeline-api-foundation/01-RESEARCH.md
@pom.xml
@cameleer3-server-core/pom.xml
@cameleer3-server-app/pom.xml
@cameleer-server-core/pom.xml
@cameleer-server-app/pom.xml
</context>
<tasks>
@@ -84,11 +84,11 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
<name>Task 1: Dependencies, Docker Compose, ClickHouse schema, and application config</name>
<files>
pom.xml,
cameleer3-server-core/pom.xml,
cameleer3-server-app/pom.xml,
cameleer-server-core/pom.xml,
cameleer-server-app/pom.xml,
docker-compose.yml,
clickhouse/init/01-schema.sql,
cameleer3-server-app/src/main/resources/application.yml
cameleer-server-app/src/main/resources/application.yml
</files>
<behavior>
- docker compose up -d starts ClickHouse on ports 8123/9000
@@ -101,7 +101,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
- Maven compile succeeds with new dependencies
</behavior>
<action>
1. Add dependencies to cameleer3-server-app/pom.xml per research:
1. Add dependencies to cameleer-server-app/pom.xml per research:
- clickhouse-jdbc 0.9.7 (classifier: all)
- spring-boot-starter-actuator
- springdoc-openapi-starter-webmvc-ui 2.8.6
@@ -109,13 +109,13 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
- junit-jupiter from testcontainers 2.0.2 (test scope)
- awaitility (test scope)
2. Add slf4j-api dependency to cameleer3-server-core/pom.xml.
2. Add slf4j-api dependency to cameleer-server-core/pom.xml.
3. Create docker-compose.yml at project root with ClickHouse service:
- Image: clickhouse/clickhouse-server:25.3
- Ports: 8123:8123, 9000:9000
- Volume mount ./clickhouse/init to /docker-entrypoint-initdb.d
- Environment: CLICKHOUSE_USER=cameleer, CLICKHOUSE_PASSWORD=cameleer_dev, CLICKHOUSE_DB=cameleer3
- Environment: CLICKHOUSE_USER=cameleer, CLICKHOUSE_PASSWORD=cameleer_dev, CLICKHOUSE_DB=cameleer
- ulimits nofile 262144
4. Create clickhouse/init/01-schema.sql with the three tables from research:
@@ -124,9 +124,9 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
- agent_metrics: MergeTree, daily partitioning on collected_at, ORDER BY (agent_id, metric_name, collected_at), TTL collected_at + INTERVAL 30 DAY, SETTINGS ttl_only_drop_parts=1.
- All DateTime fields use DateTime64(3, 'UTC').
5. Create cameleer3-server-app/src/main/resources/application.yml with config from research:
5. Create cameleer-server-app/src/main/resources/application.yml with config from research:
- server.port: 8081
- spring.datasource: url=jdbc:ch://localhost:8123/cameleer3, username/password, driver-class-name
- spring.datasource: url=jdbc:ch://localhost:8123/cameleer, username/password, driver-class-name
- spring.jackson: write-dates-as-timestamps=false, fail-on-unknown-properties=false
- ingestion: buffer-capacity=50000, batch-size=5000, flush-interval-ms=1000
- clickhouse.ttl-days: 30
@@ -144,13 +144,13 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
<task type="auto" tdd="true">
<name>Task 2: WriteBuffer, repository interfaces, IngestionConfig, and ClickHouseConfig</name>
<files>
cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java,
cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java,
cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java,
cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java,
cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java,
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java,
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/DiagramRepository.java,
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/MetricsRepository.java,
cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/WriteBufferTest.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
</files>
<behavior>
- WriteBuffer(capacity=10): offer() returns true for first 10 items, false on 11th
@@ -181,7 +181,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
3. Create repository interfaces in core module:
- ExecutionRepository: void insertBatch(List<RouteExecution> executions)
- DiagramRepository: void store(RouteGraph graph), Optional<RouteGraph> findByContentHash(String hash), Optional<String> findContentHashForRoute(String routeId, String agentId)
- MetricsRepository: void insertBatch(List<MetricsSnapshot> metrics) -- use a generic type or the cameleer3-common metrics model if available; if not, create a simple MetricsData record in core module
- MetricsRepository: void insertBatch(List<MetricsSnapshot> metrics) -- use a generic type or the cameleer-common metrics model if available; if not, create a simple MetricsData record in core module
4. Create IngestionConfig as @ConfigurationProperties("ingestion"):
- bufferCapacity (int, default 50000)
@@ -193,7 +193,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
- No custom bean needed if relying on auto-config; only create if explicit JdbcTemplate customization required
</action>
<verify>
<automated>mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q 2>&1 | tail -10</automated>
<automated>mvn test -pl cameleer-server-core -Dtest=WriteBufferTest -q 2>&1 | tail -10</automated>
</verify>
<done>WriteBuffer passes all unit tests. Repository interfaces exist with correct method signatures. IngestionConfig reads from application.yml.</done>
</task>
@@ -201,7 +201,7 @@ Output: Working ClickHouse via Docker Compose, DDL with TTL, WriteBuffer with un
</tasks>
<verification>
- `mvn test -pl cameleer3-server-core -q` -- all WriteBuffer unit tests pass
- `mvn test -pl cameleer-server-core -q` -- all WriteBuffer unit tests pass
- `mvn clean compile -q` -- full project compiles with new dependencies
- `docker compose config` -- validates Docker Compose file
- clickhouse/init/01-schema.sql contains CREATE TABLE for all three tables with correct ENGINE, ORDER BY, PARTITION BY, and TTL

48
.planning/phases/01-ingestion-pipeline-api-foundation/01-01-SUMMARY.md
View File

@@ -26,22 +26,22 @@ key-files:
created:
- docker-compose.yml
- clickhouse/init/01-schema.sql
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/DiagramRepository.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsRepository.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/MetricsSnapshot.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java
- cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/WriteBufferTest.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/DiagramRepository.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/MetricsRepository.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/MetricsSnapshot.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/ClickHouseConfig.java
- cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/WriteBufferTest.java
modified:
- cameleer3-server-core/pom.xml
- cameleer3-server-app/pom.xml
- cameleer3-server-app/src/main/resources/application.yml
- cameleer-server-core/pom.xml
- cameleer-server-app/pom.xml
- cameleer-server-app/src/main/resources/application.yml
key-decisions:
- "Used spring-boot-starter-jdbc for JdbcTemplate + HikariCP auto-config rather than manual DataSource"
- "Created MetricsSnapshot record in core module since cameleer3-common has no metrics model"
- "Created MetricsSnapshot record in core module since cameleer-common has no metrics model"
- "ClickHouseConfig exposes JdbcTemplate bean; relies on Spring Boot DataSource auto-config"
patterns-established:
@@ -84,21 +84,21 @@ Each task was committed atomically:
## Files Created/Modified
- `docker-compose.yml` - ClickHouse service with ports 8123/9000, init volume mount
- `clickhouse/init/01-schema.sql` - DDL for route_executions, route_diagrams, agent_metrics
- `cameleer3-server-core/src/main/java/.../ingestion/WriteBuffer.java` - Bounded queue with offer/offerBatch/drain
- `cameleer3-server-core/src/main/java/.../storage/ExecutionRepository.java` - Batch insert interface for RouteExecution
- `cameleer3-server-core/src/main/java/.../storage/DiagramRepository.java` - Store/find interface for RouteGraph
- `cameleer3-server-core/src/main/java/.../storage/MetricsRepository.java` - Batch insert interface for MetricsSnapshot
- `cameleer3-server-core/src/main/java/.../storage/model/MetricsSnapshot.java` - Metrics data record
- `cameleer3-server-app/src/main/java/.../config/IngestionConfig.java` - Buffer capacity, batch size, flush interval
- `cameleer3-server-app/src/main/java/.../config/ClickHouseConfig.java` - JdbcTemplate bean
- `cameleer3-server-core/src/test/java/.../ingestion/WriteBufferTest.java` - 10 unit tests for WriteBuffer
- `cameleer3-server-core/pom.xml` - Added slf4j-api
- `cameleer3-server-app/pom.xml` - Added clickhouse-jdbc, springdoc, actuator, testcontainers, awaitility
- `cameleer3-server-app/src/main/resources/application.yml` - Full config with datasource, ingestion, springdoc, actuator
- `cameleer-server-core/src/main/java/.../ingestion/WriteBuffer.java` - Bounded queue with offer/offerBatch/drain
- `cameleer-server-core/src/main/java/.../storage/ExecutionRepository.java` - Batch insert interface for RouteExecution
- `cameleer-server-core/src/main/java/.../storage/DiagramRepository.java` - Store/find interface for RouteGraph
- `cameleer-server-core/src/main/java/.../storage/MetricsRepository.java` - Batch insert interface for MetricsSnapshot
- `cameleer-server-core/src/main/java/.../storage/model/MetricsSnapshot.java` - Metrics data record
- `cameleer-server-app/src/main/java/.../config/IngestionConfig.java` - Buffer capacity, batch size, flush interval
- `cameleer-server-app/src/main/java/.../config/ClickHouseConfig.java` - JdbcTemplate bean
- `cameleer-server-core/src/test/java/.../ingestion/WriteBufferTest.java` - 10 unit tests for WriteBuffer
- `cameleer-server-core/pom.xml` - Added slf4j-api
- `cameleer-server-app/pom.xml` - Added clickhouse-jdbc, springdoc, actuator, testcontainers, awaitility
- `cameleer-server-app/src/main/resources/application.yml` - Full config with datasource, ingestion, springdoc, actuator
## Decisions Made
- Used spring-boot-starter-jdbc to get JdbcTemplate and HikariCP auto-configuration rather than manually wiring a DataSource
- Created MetricsSnapshot record in core module since cameleer3-common does not include a metrics model
- Created MetricsSnapshot record in core module since cameleer-common does not include a metrics model
- ClickHouseConfig is minimal -- relies on Spring Boot auto-configuring DataSource from spring.datasource properties
## Deviations from Plan

64
.planning/phases/01-ingestion-pipeline-api-foundation/01-02-PLAN.md
View File

@@ -5,18 +5,18 @@ type: execute
wave: 3
depends_on: ["01-01", "01-03"]
files_modified:
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/DiagramController.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/MetricsController.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseDiagramRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ExecutionControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/DiagramControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/MetricsControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/BackpressureIT.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ExecutionController.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/DiagramController.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/MetricsController.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseDiagramRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseMetricsRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ClickHouseFlushScheduler.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/IngestionService.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ExecutionControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/DiagramControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/MetricsControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/BackpressureIT.java
autonomous: true
requirements:
- INGST-01
@@ -32,16 +32,16 @@ must_haves:
- "Data posted to endpoints appears in ClickHouse after flush interval"
- "When buffer is full, endpoints return 503 with Retry-After header"
artifacts:
- path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java"
- path: "cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ExecutionController.java"
provides: "POST /api/v1/data/executions endpoint"
min_lines: 20
- path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java"
- path: "cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionRepository.java"
provides: "Batch insert to route_executions table via JdbcTemplate"
min_lines: 30
- path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java"
- path: "cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ClickHouseFlushScheduler.java"
provides: "Scheduled drain of WriteBuffer into ClickHouse"
min_lines: 20
- path: "cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java"
- path: "cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/IngestionService.java"
provides: "Routes data to appropriate WriteBuffer instances"
min_lines: 20
key_links:
@@ -92,7 +92,7 @@ Output: Working ingestion flow verified by integration tests against Testcontain
<!-- Interfaces from Plan 01 that this plan depends on -->
<interfaces>
From cameleer3-server-core WriteBuffer.java:
From cameleer-server-core WriteBuffer.java:
```java
public class WriteBuffer<T> {
public WriteBuffer(int capacity);
@@ -106,7 +106,7 @@ public class WriteBuffer<T> {
}
```
From cameleer3-server-core repository interfaces:
From cameleer-server-core repository interfaces:
```java
public interface ExecutionRepository {
void insertBatch(List<RouteExecution> executions);
@@ -138,11 +138,11 @@ public class IngestionConfig {
<task type="auto" tdd="false">
<name>Task 1: IngestionService, ClickHouse repositories, and flush scheduler</name>
<files>
cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseDiagramRepository.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsRepository.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java
cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/IngestionService.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionRepository.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseDiagramRepository.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseMetricsRepository.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ClickHouseFlushScheduler.java
</files>
<action>
1. Create IngestionService in core module (no Spring annotations -- it's a plain class):
@@ -188,13 +188,13 @@ public class IngestionConfig {
<task type="auto" tdd="true">
<name>Task 2: Ingestion REST controllers and integration tests</name>
<files>
cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/DiagramController.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/MetricsController.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ExecutionControllerIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/DiagramControllerIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/MetricsControllerIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/BackpressureIT.java
cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ExecutionController.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/controller/DiagramController.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/controller/MetricsController.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ExecutionControllerIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/DiagramControllerIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/MetricsControllerIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/BackpressureIT.java
</files>
<behavior>
- POST /api/v1/data/executions with single RouteExecution JSON returns 202
@@ -245,7 +245,7 @@ public class IngestionConfig {
Note: All integration tests must include X-Cameleer-Protocol-Version:1 header (API-04 will be enforced by Plan 03's interceptor, but include the header now for forward compatibility).
</action>
<verify>
<automated>mvn test -pl cameleer3-server-app -Dtest="ExecutionControllerIT,DiagramControllerIT,MetricsControllerIT,BackpressureIT" -q 2>&1 | tail -15</automated>
<automated>mvn test -pl cameleer-server-app -Dtest="ExecutionControllerIT,DiagramControllerIT,MetricsControllerIT,BackpressureIT" -q 2>&1 | tail -15</automated>
</verify>
<done>All three ingestion endpoints return 202 on valid data. Data arrives in ClickHouse after flush. Buffer-full returns 503 with Retry-After. Unknown JSON fields accepted. Integration tests green.</done>
</task>
@@ -253,7 +253,7 @@ public class IngestionConfig {
</tasks>
<verification>
- `mvn test -pl cameleer3-server-app -Dtest="ExecutionControllerIT,DiagramControllerIT,MetricsControllerIT,BackpressureIT" -q` -- all integration tests pass
- `mvn test -pl cameleer-server-app -Dtest="ExecutionControllerIT,DiagramControllerIT,MetricsControllerIT,BackpressureIT" -q` -- all integration tests pass
- POST to /api/v1/data/executions returns 202
- POST to /api/v1/data/diagrams returns 202
- POST to /api/v1/data/metrics returns 202

58
.planning/phases/01-ingestion-pipeline-api-foundation/01-02-SUMMARY.md
View File

@@ -30,21 +30,21 @@ tech-stack:
key-files:
created:
- cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseDiagramRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsRepository.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionBeanConfig.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/DiagramController.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/MetricsController.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ExecutionControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/DiagramControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/MetricsControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/BackpressureIT.java
- cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/IngestionService.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseDiagramRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseMetricsRepository.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ClickHouseFlushScheduler.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionBeanConfig.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ExecutionController.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/DiagramController.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/controller/MetricsController.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ExecutionControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/DiagramControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/MetricsControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/BackpressureIT.java
modified:
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
key-decisions:
- "Controllers accept raw String body and detect single vs array JSON to support both payload formats"
@@ -91,20 +91,20 @@ Each task was committed atomically:
3. **Task 2 GREEN: Ingestion REST controllers with backpressure** - `8fe65f0` (feat)
## Files Created/Modified
- `cameleer3-server-core/.../ingestion/IngestionService.java` - Routes data to WriteBuffer instances
- `cameleer3-server-app/.../storage/ClickHouseExecutionRepository.java` - Batch insert with parallel processor arrays
- `cameleer3-server-app/.../storage/ClickHouseDiagramRepository.java` - JSON storage with SHA-256 content-hash dedup
- `cameleer3-server-app/.../storage/ClickHouseMetricsRepository.java` - Batch insert for agent_metrics
- `cameleer3-server-app/.../ingestion/ClickHouseFlushScheduler.java` - Scheduled drain + SmartLifecycle shutdown
- `cameleer3-server-app/.../config/IngestionBeanConfig.java` - WriteBuffer and IngestionService bean wiring
- `cameleer3-server-app/.../controller/ExecutionController.java` - POST /api/v1/data/executions
- `cameleer3-server-app/.../controller/DiagramController.java` - POST /api/v1/data/diagrams
- `cameleer3-server-app/.../controller/MetricsController.java` - POST /api/v1/data/metrics
- `cameleer3-server-app/.../config/IngestionConfig.java` - Removed @Configuration (fix duplicate bean)
- `cameleer3-server-app/.../controller/ExecutionControllerIT.java` - 4 tests: single, array, flush, unknown fields
- `cameleer3-server-app/.../controller/DiagramControllerIT.java` - 3 tests: single, array, flush
- `cameleer3-server-app/.../controller/MetricsControllerIT.java` - 2 tests: POST, flush
- `cameleer3-server-app/.../controller/BackpressureIT.java` - 2 tests: 503 response, data not lost
- `cameleer-server-core/.../ingestion/IngestionService.java` - Routes data to WriteBuffer instances
- `cameleer-server-app/.../storage/ClickHouseExecutionRepository.java` - Batch insert with parallel processor arrays
- `cameleer-server-app/.../storage/ClickHouseDiagramRepository.java` - JSON storage with SHA-256 content-hash dedup
- `cameleer-server-app/.../storage/ClickHouseMetricsRepository.java` - Batch insert for agent_metrics
- `cameleer-server-app/.../ingestion/ClickHouseFlushScheduler.java` - Scheduled drain + SmartLifecycle shutdown
- `cameleer-server-app/.../config/IngestionBeanConfig.java` - WriteBuffer and IngestionService bean wiring
- `cameleer-server-app/.../controller/ExecutionController.java` - POST /api/v1/data/executions
- `cameleer-server-app/.../controller/DiagramController.java` - POST /api/v1/data/diagrams
- `cameleer-server-app/.../controller/MetricsController.java` - POST /api/v1/data/metrics
- `cameleer-server-app/.../config/IngestionConfig.java` - Removed @Configuration (fix duplicate bean)
- `cameleer-server-app/.../controller/ExecutionControllerIT.java` - 4 tests: single, array, flush, unknown fields
- `cameleer-server-app/.../controller/DiagramControllerIT.java` - 3 tests: single, array, flush
- `cameleer-server-app/.../controller/MetricsControllerIT.java` - 2 tests: POST, flush
- `cameleer-server-app/.../controller/BackpressureIT.java` - 2 tests: 503 response, data not lost
## Decisions Made
- Controllers accept raw String body and detect single vs array JSON (starts with `[`), supporting both payload formats per protocol spec
@@ -119,7 +119,7 @@ Each task was committed atomically:
- **Found during:** Task 2 (integration test context startup)
- **Issue:** IngestionConfig had both `@Configuration` and `@ConfigurationProperties`, while `@EnableConfigurationProperties(IngestionConfig.class)` on the app class created a second bean, causing "expected single matching bean but found 2"
- **Fix:** Removed `@Configuration` from IngestionConfig, relying solely on `@EnableConfigurationProperties`
- **Files modified:** cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionConfig.java
- **Files modified:** cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionConfig.java
- **Verification:** Application context starts successfully, all tests pass
- **Committed in:** 8fe65f0

54
.planning/phases/01-ingestion-pipeline-api-foundation/01-03-PLAN.md
View File

@@ -5,15 +5,15 @@ type: execute
wave: 2
depends_on: ["01-01"]
files_modified:
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
- cameleer3-server-app/src/test/resources/application-test.yml
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/CameleerServerApplication.java
- cameleer-server-app/src/test/resources/application-test.yml
- cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/interceptor/ProtocolVersionIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/HealthControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenApiIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ForwardCompatIT.java
autonomous: true
requirements:
- API-01
@@ -34,13 +34,13 @@ must_haves:
- "Unknown JSON fields in request body do not cause deserialization errors"
- "ClickHouse tables have TTL clause for 30-day retention"
artifacts:
- path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java"
- path: "cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java"
provides: "Validates X-Cameleer-Protocol-Version:1 header on data endpoints"
min_lines: 20
- path: "cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java"
- path: "cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java"
provides: "Registers interceptor with path patterns"
min_lines: 10
- path: "cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java"
- path: "cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java"
provides: "Shared Testcontainers base class for integration tests"
min_lines: 20
key_links:
@@ -83,11 +83,11 @@ Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol he
<task type="auto">
<name>Task 1: Test infrastructure, protocol version interceptor, WebConfig, and Spring Boot application class</name>
<files>
cameleer3-server-app/src/test/resources/application-test.yml,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java,
cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
cameleer-server-app/src/test/resources/application-test.yml,
cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java,
cameleer-server-app/src/main/java/com/cameleer/server/app/CameleerServerApplication.java
</files>
<action>
1. Create application-test.yml for test profile:
@@ -113,15 +113,15 @@ Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol he
- Override addInterceptors: register interceptor with pathPatterns "/api/v1/data/**" and "/api/v1/agents/**"
- Explicitly EXCLUDE: "/api/v1/health", "/api/v1/api-docs/**", "/api/v1/swagger-ui/**", "/api/v1/swagger-ui.html"
5. Create or update Cameleer3ServerApplication:
- @SpringBootApplication in package com.cameleer3.server.app
5. Create or update CameleerServerApplication:
- @SpringBootApplication in package com.cameleer.server.app
- @EnableScheduling (needed for ClickHouseFlushScheduler from Plan 02)
- @EnableConfigurationProperties(IngestionConfig.class)
- Main method with SpringApplication.run()
- Ensure package scanning covers com.cameleer3.server.app and com.cameleer3.server.core
- Ensure package scanning covers com.cameleer.server.app and com.cameleer.server.core
</action>
<verify>
<automated>mvn clean compile -pl cameleer3-server-app -q 2>&1 | tail -5</automated>
<automated>mvn clean compile -pl cameleer-server-app -q 2>&1 | tail -5</automated>
</verify>
<done>AbstractClickHouseIT base class ready for integration tests. ProtocolVersionInterceptor validates header on data/agent paths. Health, swagger, and api-docs paths excluded. Application class enables scheduling and config properties.</done>
</task>
@@ -129,10 +129,10 @@ Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol he
<task type="auto" tdd="true">
<name>Task 2: Health, OpenAPI, protocol version, forward compat, and TTL integration tests</name>
<files>
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java,
cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/HealthControllerIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenApiIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/interceptor/ProtocolVersionIT.java,
cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ForwardCompatIT.java
</files>
<behavior>
- GET /api/v1/health returns 200 with JSON containing status field
@@ -178,7 +178,7 @@ Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol he
Note: All tests that POST to data endpoints must include X-Cameleer-Protocol-Version:1 header.
</action>
<verify>
<automated>mvn test -pl cameleer3-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q 2>&1 | tail -15</automated>
<automated>mvn test -pl cameleer-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q 2>&1 | tail -15</automated>
</verify>
<done>Health returns 200. OpenAPI docs are available and list endpoints. Protocol version header enforced on data paths, not on health/docs. Unknown JSON fields accepted. TTL confirmed in ClickHouse DDL via HealthControllerIT test methods.</done>
</task>
@@ -186,7 +186,7 @@ Output: AbstractClickHouseIT base class, working health, Swagger UI, protocol he
</tasks>
<verification>
- `mvn test -pl cameleer3-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q` -- all tests pass
- `mvn test -pl cameleer-server-app -Dtest="HealthControllerIT,OpenApiIT,ProtocolVersionIT,ForwardCompatIT" -q` -- all tests pass
- GET /api/v1/health returns 200
- GET /api/v1/api-docs returns OpenAPI spec
- Missing protocol header returns 400 on data endpoints

46
.planning/phases/01-ingestion-pipeline-api-foundation/01-03-SUMMARY.md
View File

@@ -12,7 +12,7 @@ provides:
- AbstractClickHouseIT base class for all integration tests
- ProtocolVersionInterceptor enforcing X-Cameleer-Protocol-Version:1 on data/agent paths
- WebConfig with interceptor registration and path exclusions
- Cameleer3ServerApplication with @EnableScheduling and component scanning
- CameleerServerApplication with @EnableScheduling and component scanning
- 12 passing integration tests (health, OpenAPI, protocol version, forward compat, TTL)
affects: [01-02, 02-search, 03-agent-registry]
@@ -23,17 +23,17 @@ tech-stack:
key-files:
created:
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/Cameleer3ServerApplication.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java
- cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java
- cameleer3-server-app/src/test/resources/application-test.yml
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/HealthControllerIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/OpenApiIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/interceptor/ProtocolVersionIT.java
- cameleer3-server-app/src/test/java/com/cameleer3/server/app/controller/ForwardCompatIT.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/CameleerServerApplication.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java
- cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java
- cameleer-server-app/src/test/resources/application-test.yml
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/HealthControllerIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenApiIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/interceptor/ProtocolVersionIT.java
- cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ForwardCompatIT.java
modified:
- cameleer3-server-app/pom.xml
- cameleer-server-app/pom.xml
- pom.xml
- clickhouse/init/01-schema.sql
@@ -70,7 +70,7 @@ completed: 2026-03-11
- ProtocolVersionInterceptor validates X-Cameleer-Protocol-Version:1 on /api/v1/data/** and /api/v1/agents/** paths, returning 400 JSON error for missing or wrong version
- AbstractClickHouseIT base class with Testcontainers ClickHouse 25.3, shared static container, schema init from 01-schema.sql
- 12 integration tests: health endpoint (2), OpenAPI docs (2), protocol version enforcement (5), forward compatibility (1), TTL verification (2)
- Cameleer3ServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning
- CameleerServerApplication with @EnableScheduling, @EnableConfigurationProperties, and dual package scanning
## Task Commits
@@ -80,17 +80,17 @@ Each task was committed atomically:
2. **Task 2: Integration tests for health, OpenAPI, protocol version, forward compat, and TTL** - `2d3fde3` (test)
## Files Created/Modified
- `cameleer3-server-app/src/main/java/.../Cameleer3ServerApplication.java` - Spring Boot entry point with scheduling and config properties
- `cameleer3-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java` - Validates protocol version header on data/agent paths
- `cameleer3-server-app/src/main/java/.../config/WebConfig.java` - Registers interceptor with path patterns and exclusions
- `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` - Shared Testcontainers base class for ITs
- `cameleer3-server-app/src/test/resources/application-test.yml` - Test profile with small buffer config
- `cameleer3-server-app/src/test/java/.../controller/HealthControllerIT.java` - Health endpoint and TTL tests
- `cameleer3-server-app/src/test/java/.../controller/OpenApiIT.java` - OpenAPI and Swagger UI tests
- `cameleer3-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java` - Protocol header enforcement tests
- `cameleer3-server-app/src/test/java/.../controller/ForwardCompatIT.java` - Unknown JSON fields test
- `cameleer-server-app/src/main/java/.../CameleerServerApplication.java` - Spring Boot entry point with scheduling and config properties
- `cameleer-server-app/src/main/java/.../interceptor/ProtocolVersionInterceptor.java` - Validates protocol version header on data/agent paths
- `cameleer-server-app/src/main/java/.../config/WebConfig.java` - Registers interceptor with path patterns and exclusions
- `cameleer-server-app/src/test/java/.../AbstractClickHouseIT.java` - Shared Testcontainers base class for ITs
- `cameleer-server-app/src/test/resources/application-test.yml` - Test profile with small buffer config
- `cameleer-server-app/src/test/java/.../controller/HealthControllerIT.java` - Health endpoint and TTL tests
- `cameleer-server-app/src/test/java/.../controller/OpenApiIT.java` - OpenAPI and Swagger UI tests
- `cameleer-server-app/src/test/java/.../interceptor/ProtocolVersionIT.java` - Protocol header enforcement tests
- `cameleer-server-app/src/test/java/.../controller/ForwardCompatIT.java` - Unknown JSON fields test
- `pom.xml` - Override testcontainers.version to 2.0.3
- `cameleer3-server-app/pom.xml` - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3
- `cameleer-server-app/pom.xml` - Remove junit-jupiter, upgrade testcontainers-clickhouse to 2.0.3
- `clickhouse/init/01-schema.sql` - Fix TTL expressions and error column types
## Decisions Made
@@ -107,7 +107,7 @@ Each task was committed atomically:
- **Found during:** Task 2 (compilation)
- **Issue:** org.testcontainers:junit-jupiter:2.0.2 does not exist in Maven Central
- **Fix:** Removed junit-jupiter dependency, upgraded to TC 2.0.3, managed container lifecycle manually via static initializer
- **Files modified:** cameleer3-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java
- **Files modified:** cameleer-server-app/pom.xml, pom.xml, AbstractClickHouseIT.java
- **Verification:** All tests compile and pass
- **Committed in:** 2d3fde3

56
.planning/phases/01-ingestion-pipeline-api-foundation/01-RESEARCH.md
View File

@@ -6,7 +6,7 @@
## Summary
Phase 1 establishes the data pipeline and API skeleton for Cameleer3 Server. Agents POST execution data, diagrams, and metrics to REST endpoints; the server buffers these in memory and batch-flushes to ClickHouse. The ClickHouse schema design is the most critical and least reversible decision in this phase -- ORDER BY and partitioning cannot be changed without table recreation.
Phase 1 establishes the data pipeline and API skeleton for Cameleer Server. Agents POST execution data, diagrams, and metrics to REST endpoints; the server buffers these in memory and batch-flushes to ClickHouse. The ClickHouse schema design is the most critical and least reversible decision in this phase -- ORDER BY and partitioning cannot be changed without table recreation.
The ClickHouse Java ecosystem has undergone significant changes. The recommended approach is **clickhouse-jdbc v0.9.7** (JDBC V2 driver) with Spring Boot's JdbcTemplate for batch inserts. An alternative is the standalone **client-v2** artifact which offers a POJO-based insert API, but JDBC integration with Spring Boot is more conventional and better documented. ClickHouse now has a native full-text index (TYPE text, GA as of March 2026) that supersedes the older tokenbf_v1 bloom filter approach -- this is relevant for Phase 2 but should be accounted for in schema design now.
@@ -17,7 +17,7 @@ The ClickHouse Java ecosystem has undergone significant changes. The recommended
| ID | Description | Research Support |
|----|-------------|-----------------|
| INGST-01 (#1) | Accept RouteExecution via POST /api/v1/data/executions, return 202 | REST controller + async write buffer pattern; Jackson deserialization of cameleer3-common models |
| INGST-01 (#1) | Accept RouteExecution via POST /api/v1/data/executions, return 202 | REST controller + async write buffer pattern; Jackson deserialization of cameleer-common models |
| INGST-02 (#2) | Accept RouteGraph via POST /api/v1/data/diagrams, return 202 | Same pattern; separate ClickHouse table for diagrams with content-hash dedup |
| INGST-03 (#3) | Accept metrics via POST /api/v1/data/metrics, return 202 | Same pattern; separate ClickHouse table for metrics |
| INGST-04 (#4) | In-memory batch buffer with configurable flush interval/size | ArrayBlockingQueue + @Scheduled flush; configurable via application.yml |
@@ -60,7 +60,7 @@ The ClickHouse Java ecosystem has undergone significant changes. The recommended
| ArrayBlockingQueue | LMAX Disruptor | Disruptor is faster under extreme contention but adds complexity; ABQ is sufficient for this throughput |
| Spring JdbcTemplate | Raw JDBC PreparedStatement | JdbcTemplate provides cleaner error handling and resource management; no meaningful overhead |
**Installation (add to cameleer3-server-app/pom.xml):**
**Installation (add to cameleer-server-app/pom.xml):**
```xml
<!-- ClickHouse JDBC V2 -->
<dependency>
@@ -103,7 +103,7 @@ The ClickHouse Java ecosystem has undergone significant changes. The recommended
</dependency>
```
**Add to cameleer3-server-core/pom.xml:**
**Add to cameleer-server-core/pom.xml:**
```xml
<!-- SLF4J for logging (no Spring dependency) -->
<dependency>
@@ -117,7 +117,7 @@ The ClickHouse Java ecosystem has undergone significant changes. The recommended
### Recommended Project Structure
```
cameleer3-server-core/src/main/java/com/cameleer3/server/core/
cameleer-server-core/src/main/java/com/cameleer/server/core/
ingestion/
WriteBuffer.java # Bounded queue + flush logic
IngestionService.java # Accepts data, routes to buffer
@@ -126,9 +126,9 @@ cameleer3-server-core/src/main/java/com/cameleer3/server/core/
DiagramRepository.java # Interface: store/retrieve diagrams
MetricsRepository.java # Interface: store metrics
model/
(extend/complement cameleer3-common models as needed)
(extend/complement cameleer-common models as needed)
cameleer3-server-app/src/main/java/com/cameleer3/server/app/
cameleer-server-app/src/main/java/com/cameleer/server/app/
config/
ClickHouseConfig.java # DataSource + JdbcTemplate bean
IngestionConfig.java # Buffer size, flush interval from YAML
@@ -424,7 +424,7 @@ services:
environment:
CLICKHOUSE_USER: cameleer
CLICKHOUSE_PASSWORD: cameleer_dev
CLICKHOUSE_DB: cameleer3
CLICKHOUSE_DB: cameleer
ulimits:
nofile:
soft: 262144
@@ -442,7 +442,7 @@ server:
spring:
datasource:
url: jdbc:ch://localhost:8123/cameleer3
url: jdbc:ch://localhost:8123/cameleer
username: cameleer
password: cameleer_dev
driver-class-name: com.clickhouse.jdbc.ClickHouseDriver
@@ -493,10 +493,10 @@ management:
## Open Questions
1. **Exact cameleer3-common model structure**
1. **Exact cameleer-common model structure**
- What we know: Models include RouteExecution, ProcessorExecution, ExchangeSnapshot, RouteGraph, RouteNode, RouteEdge
- What's unclear: Exact field names, types, nesting structure -- needed to design ClickHouse schema precisely
- Recommendation: Read cameleer3-common source code before implementing schema. Schema must match the wire format.
- Recommendation: Read cameleer-common source code before implementing schema. Schema must match the wire format.
2. **ClickHouse JDBC V2 + HikariCP compatibility**
- What we know: clickhouse-jdbc 0.9.7 implements JDBC spec; HikariCP is Spring Boot default
@@ -515,36 +515,36 @@ management:
| Property | Value |
|----------|-------|
| Framework | JUnit 5 (Spring Boot managed) + Testcontainers 2.0.2 |
| Config file | cameleer3-server-app/src/test/resources/application-test.yml (Wave 0) |
| Quick run command | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` |
| Config file | cameleer-server-app/src/test/resources/application-test.yml (Wave 0) |
| Quick run command | `mvn test -pl cameleer-server-core -Dtest=WriteBufferTest -q` |
| Full suite command | `mvn verify` |
### Phase Requirements -> Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| INGST-01 | POST /api/v1/data/executions returns 202, data in ClickHouse | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | Wave 0 |
| INGST-02 | POST /api/v1/data/diagrams returns 202 | integration | `mvn test -pl cameleer3-server-app -Dtest=DiagramControllerIT -q` | Wave 0 |
| INGST-03 | POST /api/v1/data/metrics returns 202 | integration | `mvn test -pl cameleer3-server-app -Dtest=MetricsControllerIT -q` | Wave 0 |
| INGST-04 | Buffer flushes at interval/size | unit | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` | Wave 0 |
| INGST-05 | 503 when buffer full | unit+integration | `mvn test -pl cameleer3-server-app -Dtest=BackpressureIT -q` | Wave 0 |
| INGST-06 | TTL removes old data | integration | `mvn test -pl cameleer3-server-app -Dtest=ClickHouseTtlIT -q` | Wave 0 |
| INGST-01 | POST /api/v1/data/executions returns 202, data in ClickHouse | integration | `mvn test -pl cameleer-server-app -Dtest=ExecutionControllerIT -q` | Wave 0 |
| INGST-02 | POST /api/v1/data/diagrams returns 202 | integration | `mvn test -pl cameleer-server-app -Dtest=DiagramControllerIT -q` | Wave 0 |
| INGST-03 | POST /api/v1/data/metrics returns 202 | integration | `mvn test -pl cameleer-server-app -Dtest=MetricsControllerIT -q` | Wave 0 |
| INGST-04 | Buffer flushes at interval/size | unit | `mvn test -pl cameleer-server-core -Dtest=WriteBufferTest -q` | Wave 0 |
| INGST-05 | 503 when buffer full | unit+integration | `mvn test -pl cameleer-server-app -Dtest=BackpressureIT -q` | Wave 0 |
| INGST-06 | TTL removes old data | integration | `mvn test -pl cameleer-server-app -Dtest=ClickHouseTtlIT -q` | Wave 0 |
| API-01 | Endpoints under /api/v1/ | integration | Covered by controller ITs | Wave 0 |
| API-02 | OpenAPI docs available | integration | `mvn test -pl cameleer3-server-app -Dtest=OpenApiIT -q` | Wave 0 |
| API-03 | GET /api/v1/health responds | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT -q` | Wave 0 |
| API-04 | Protocol version header validated | integration | `mvn test -pl cameleer3-server-app -Dtest=ProtocolVersionIT -q` | Wave 0 |
| API-05 | Unknown JSON fields accepted | unit | `mvn test -pl cameleer3-server-app -Dtest=ForwardCompatIT -q` | Wave 0 |
| API-02 | OpenAPI docs available | integration | `mvn test -pl cameleer-server-app -Dtest=OpenApiIT -q` | Wave 0 |
| API-03 | GET /api/v1/health responds | integration | `mvn test -pl cameleer-server-app -Dtest=HealthControllerIT -q` | Wave 0 |
| API-04 | Protocol version header validated | integration | `mvn test -pl cameleer-server-app -Dtest=ProtocolVersionIT -q` | Wave 0 |
| API-05 | Unknown JSON fields accepted | unit | `mvn test -pl cameleer-server-app -Dtest=ForwardCompatIT -q` | Wave 0 |
### Sampling Rate
- **Per task commit:** `mvn test -pl cameleer3-server-core -q` (unit tests, fast)
- **Per task commit:** `mvn test -pl cameleer-server-core -q` (unit tests, fast)
- **Per wave merge:** `mvn verify` (full suite with Testcontainers integration tests)
- **Phase gate:** Full suite green before verification
### Wave 0 Gaps
- [ ] `cameleer3-server-app/src/test/resources/application-test.yml` -- test ClickHouse config
- [ ] `cameleer3-server-core/src/test/java/.../WriteBufferTest.java` -- buffer unit tests
- [ ] `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` -- shared Testcontainers base class
- [ ] `cameleer3-server-app/src/test/java/.../ExecutionControllerIT.java` -- ingestion integration test
- [ ] `cameleer-server-app/src/test/resources/application-test.yml` -- test ClickHouse config
- [ ] `cameleer-server-core/src/test/java/.../WriteBufferTest.java` -- buffer unit tests
- [ ] `cameleer-server-app/src/test/java/.../AbstractClickHouseIT.java` -- shared Testcontainers base class
- [ ] `cameleer-server-app/src/test/java/.../ExecutionControllerIT.java` -- ingestion integration test
- [ ] Docker available on test machine for Testcontainers
## Sources

36
.planning/phases/01-ingestion-pipeline-api-foundation/01-VALIDATION.md
View File

@@ -18,8 +18,8 @@ created: 2026-03-11
| Property | Value |
|----------|-------|
| **Framework** | JUnit 5 (Spring Boot managed) + Testcontainers 2.0.2 |
| **Config file** | cameleer3-server-app/src/test/resources/application-test.yml (Wave 0) |
| **Quick run command** | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` |
| **Config file** | cameleer-server-app/src/test/resources/application-test.yml (Wave 0) |
| **Quick run command** | `mvn test -pl cameleer-server-core -Dtest=WriteBufferTest -q` |
| **Full suite command** | `mvn verify` |
| **Estimated runtime** | ~30 seconds |
@@ -27,7 +27,7 @@ created: 2026-03-11
## Sampling Rate
- **After every task commit:** Run `mvn test -pl cameleer3-server-core -q`
- **After every task commit:** Run `mvn test -pl cameleer-server-core -q`
- **After every plan wave:** Run `mvn verify`
- **Before `/gsd:verify-work`:** Full suite must be green
- **Max feedback latency:** 30 seconds
@@ -38,17 +38,17 @@ created: 2026-03-11
| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status |
|---------|------|------|-------------|-----------|-------------------|-------------|--------|
| 1-01-01 | 01 | 1 | INGST-04 | unit | `mvn test -pl cameleer3-server-core -Dtest=WriteBufferTest -q` | no W0 | pending |
| 1-01-02 | 01 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending |
| 1-01-03 | 01 | 1 | INGST-05 | integration | `mvn test -pl cameleer3-server-app -Dtest=BackpressureIT -q` | no W0 | pending |
| 1-01-04 | 01 | 1 | INGST-06 | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT#ttlConfigured* -q` | no W0 | pending |
| 1-02-01 | 02 | 1 | INGST-01 | integration | `mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending |
| 1-02-02 | 02 | 1 | INGST-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=DiagramControllerIT -q` | no W0 | pending |
| 1-02-03 | 02 | 1 | INGST-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=MetricsControllerIT -q` | no W0 | pending |
| 1-02-04 | 02 | 1 | API-02 | integration | `mvn test -pl cameleer3-server-app -Dtest=OpenApiIT -q` | no W0 | pending |
| 1-02-05 | 02 | 1 | API-03 | integration | `mvn test -pl cameleer3-server-app -Dtest=HealthControllerIT -q` | no W0 | pending |
| 1-02-06 | 02 | 1 | API-04 | integration | `mvn test -pl cameleer3-server-app -Dtest=ProtocolVersionIT -q` | no W0 | pending |
| 1-02-07 | 02 | 1 | API-05 | unit | `mvn test -pl cameleer3-server-app -Dtest=ForwardCompatIT -q` | no W0 | pending |
| 1-01-01 | 01 | 1 | INGST-04 | unit | `mvn test -pl cameleer-server-core -Dtest=WriteBufferTest -q` | no W0 | pending |
| 1-01-02 | 01 | 1 | INGST-01 | integration | `mvn test -pl cameleer-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending |
| 1-01-03 | 01 | 1 | INGST-05 | integration | `mvn test -pl cameleer-server-app -Dtest=BackpressureIT -q` | no W0 | pending |
| 1-01-04 | 01 | 1 | INGST-06 | integration | `mvn test -pl cameleer-server-app -Dtest=HealthControllerIT#ttlConfigured* -q` | no W0 | pending |
| 1-02-01 | 02 | 1 | INGST-01 | integration | `mvn test -pl cameleer-server-app -Dtest=ExecutionControllerIT -q` | no W0 | pending |
| 1-02-02 | 02 | 1 | INGST-02 | integration | `mvn test -pl cameleer-server-app -Dtest=DiagramControllerIT -q` | no W0 | pending |
| 1-02-03 | 02 | 1 | INGST-03 | integration | `mvn test -pl cameleer-server-app -Dtest=MetricsControllerIT -q` | no W0 | pending |
| 1-02-04 | 02 | 1 | API-02 | integration | `mvn test -pl cameleer-server-app -Dtest=OpenApiIT -q` | no W0 | pending |
| 1-02-05 | 02 | 1 | API-03 | integration | `mvn test -pl cameleer-server-app -Dtest=HealthControllerIT -q` | no W0 | pending |
| 1-02-06 | 02 | 1 | API-04 | integration | `mvn test -pl cameleer-server-app -Dtest=ProtocolVersionIT -q` | no W0 | pending |
| 1-02-07 | 02 | 1 | API-05 | unit | `mvn test -pl cameleer-server-app -Dtest=ForwardCompatIT -q` | no W0 | pending |
*Status: pending / green / red / flaky*
@@ -56,10 +56,10 @@ created: 2026-03-11
## Wave 0 Requirements
- [ ] `cameleer3-server-app/src/test/resources/application-test.yml` — test ClickHouse config
- [ ] `cameleer3-server-core/src/test/java/.../WriteBufferTest.java` — buffer unit tests
- [ ] `cameleer3-server-app/src/test/java/.../AbstractClickHouseIT.java` — shared Testcontainers base class
- [ ] `cameleer3-server-app/src/test/java/.../ExecutionControllerIT.java` — ingestion integration test
- [ ] `cameleer-server-app/src/test/resources/application-test.yml` — test ClickHouse config
- [ ] `cameleer-server-core/src/test/java/.../WriteBufferTest.java` — buffer unit tests
- [ ] `cameleer-server-app/src/test/java/.../AbstractClickHouseIT.java` — shared Testcontainers base class
- [ ] `cameleer-server-app/src/test/java/.../ExecutionControllerIT.java` — ingestion integration test
- [ ] Docker available on test machine for Testcontainers
*If none: "Existing infrastructure covers all phase requirements."*

20
.planning/phases/01-ingestion-pipeline-api-foundation/01-VERIFICATION.md
View File

@@ -35,27 +35,27 @@ re_verification: false
| Artifact | Expected | Status | Details |
|---|---|---|---|
| `cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/WriteBuffer.java` | Generic bounded write buffer with offer/drain/isFull | VERIFIED | 80 lines; `ArrayBlockingQueue`-backed; implements `offer`, `offerBatch` (all-or-nothing), `drain`, `isFull`, `size`, `capacity`, `remainingCapacity` |
| `cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/WriteBuffer.java` | Generic bounded write buffer with offer/drain/isFull | VERIFIED | 80 lines; `ArrayBlockingQueue`-backed; implements `offer`, `offerBatch` (all-or-nothing), `drain`, `isFull`, `size`, `capacity`, `remainingCapacity` |
| `clickhouse/init/01-schema.sql` | ClickHouse DDL for all three tables | VERIFIED | Contains `CREATE TABLE route_executions`, `route_diagrams`, `agent_metrics`; correct ENGINE, ORDER BY, PARTITION BY, TTL with `toDateTime()` cast |
| `docker-compose.yml` | Local ClickHouse service | VERIFIED | `clickhouse/clickhouse-server:25.3`; ports 8123/9000; init volume mount; credentials configured |
| `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/ExecutionRepository.java` | Repository interface for execution batch inserts | VERIFIED | Declares `void insertBatch(List<RouteExecution> executions)` |
| `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/ExecutionRepository.java` | Repository interface for execution batch inserts | VERIFIED | Declares `void insertBatch(List<RouteExecution> executions)` |
#### Plan 01-02 Artifacts
| Artifact | Expected | Status | Details |
|---|---|---|---|
| `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ExecutionController.java` | POST /api/v1/data/executions endpoint | VERIFIED | 79 lines; `@PostMapping("/executions")`; handles single/array via raw String parsing; returns 202 or 503 + Retry-After |
| `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionRepository.java` | Batch insert to route_executions via JdbcTemplate | VERIFIED | 118 lines; `@Repository`; `BatchPreparedStatementSetter`; flattens processor tree to parallel arrays |
| `cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ClickHouseFlushScheduler.java` | Scheduled drain of WriteBuffer into ClickHouse | VERIFIED | 160 lines; `@Scheduled(fixedDelayString="${ingestion.flush-interval-ms:1000}")`; implements `SmartLifecycle` for shutdown drain |
| `cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/IngestionService.java` | Routes data to appropriate WriteBuffer instances | VERIFIED | 115 lines; plain class; `acceptExecution`, `acceptExecutions`, `acceptDiagram`, `acceptDiagrams`, `acceptMetrics`; delegates to typed `WriteBuffer` instances |
| `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ExecutionController.java` | POST /api/v1/data/executions endpoint | VERIFIED | 79 lines; `@PostMapping("/executions")`; handles single/array via raw String parsing; returns 202 or 503 + Retry-After |
| `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionRepository.java` | Batch insert to route_executions via JdbcTemplate | VERIFIED | 118 lines; `@Repository`; `BatchPreparedStatementSetter`; flattens processor tree to parallel arrays |
| `cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ClickHouseFlushScheduler.java` | Scheduled drain of WriteBuffer into ClickHouse | VERIFIED | 160 lines; `@Scheduled(fixedDelayString="${ingestion.flush-interval-ms:1000}")`; implements `SmartLifecycle` for shutdown drain |
| `cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/IngestionService.java` | Routes data to appropriate WriteBuffer instances | VERIFIED | 115 lines; plain class; `acceptExecution`, `acceptExecutions`, `acceptDiagram`, `acceptDiagrams`, `acceptMetrics`; delegates to typed `WriteBuffer` instances |
#### Plan 01-03 Artifacts
| Artifact | Expected | Status | Details |
|---|---|---|---|
| `cameleer3-server-app/src/main/java/com/cameleer3/server/app/interceptor/ProtocolVersionInterceptor.java` | Validates X-Cameleer-Protocol-Version:1 header on data endpoints | VERIFIED | 47 lines; implements `HandlerInterceptor.preHandle`; returns 400 JSON on missing/wrong version |
| `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/WebConfig.java` | Registers interceptor with path patterns | VERIFIED | 35 lines; `addInterceptors` registers interceptor on `/api/v1/data/**` and `/api/v1/agents/**`; excludes health, api-docs, swagger-ui |
| `cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractClickHouseIT.java` | Shared Testcontainers base class for integration tests | VERIFIED | 73 lines; static `ClickHouseContainer`; `@DynamicPropertySource`; `@BeforeAll` schema init from SQL file; `JdbcTemplate` exposed to subclasses |
| `cameleer-server-app/src/main/java/com/cameleer/server/app/interceptor/ProtocolVersionInterceptor.java` | Validates X-Cameleer-Protocol-Version:1 header on data endpoints | VERIFIED | 47 lines; implements `HandlerInterceptor.preHandle`; returns 400 JSON on missing/wrong version |
| `cameleer-server-app/src/main/java/com/cameleer/server/app/config/WebConfig.java` | Registers interceptor with path patterns | VERIFIED | 35 lines; `addInterceptors` registers interceptor on `/api/v1/data/**` and `/api/v1/agents/**`; excludes health, api-docs, swagger-ui |
| `cameleer-server-app/src/test/java/com/cameleer/server/app/AbstractClickHouseIT.java` | Shared Testcontainers base class for integration tests | VERIFIED | 73 lines; static `ClickHouseContainer`; `@DynamicPropertySource`; `@BeforeAll` schema init from SQL file; `JdbcTemplate` exposed to subclasses |
---
@@ -113,7 +113,7 @@ No orphaned requirements — all 11 IDs declared in plan frontmatter match the R
### Anti-Patterns Found
No anti-patterns detected. Scanned all source files in `cameleer3-server-app/src/main` and `cameleer3-server-core/src/main` for TODO/FIXME/PLACEHOLDER/stub return patterns. None found.
No anti-patterns detected. Scanned all source files in `cameleer-server-app/src/main` and `cameleer-server-core/src/main` for TODO/FIXME/PLACEHOLDER/stub return patterns. None found.
One minor observation (not a blocker):