chore: rename cameleer3 to cameleer
Rename Java packages from com.cameleer3 to com.cameleer, module directories from cameleer3-* to cameleer-*, and all references throughout workflows, Dockerfiles, docs, migrations, and pom.xml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
hsiegeln
@@ -10,7 +10,7 @@
|
||||
|
||||
**Design Spec:** `docs/superpowers/specs/2026-03-31-clickhouse-migration-design.md`
|
||||
|
||||
**Note on cameleer3-common:** The agent team is refactoring `cameleer3-common` to add `ExecutionChunk` and `FlatProcessorRecord`. Until that library is published, this plan defines server-side DTOs in `cameleer3-server-core` that mirror the common models. When the common lib is ready, swap the server DTOs for the shared classes (import change only).
|
||||
**Note on cameleer-common:** The agent team is refactoring `cameleer-common` to add `ExecutionChunk` and `FlatProcessorRecord`. Until that library is published, this plan defines server-side DTOs in `cameleer-server-core` that mirror the common models. When the common lib is ready, swap the server DTOs for the shared classes (import change only).
|
||||
|
||||
---
|
||||
|
||||
@@ -18,38 +18,38 @@
|
||||
|
||||
| File | Responsibility |
|
||||
|------|----------------|
|
||||
| `cameleer3-server-app/.../resources/clickhouse/V2__executions.sql` | DDL for `executions` table (ReplacingMergeTree) |
|
||||
| `cameleer3-server-app/.../resources/clickhouse/V3__processor_executions.sql` | DDL for `processor_executions` table (MergeTree) — uses seq/parentSeq/iteration |
|
||||
| `cameleer3-server-core/.../model/ExecutionChunk.java` | Server-side DTO mirroring agent's ExecutionChunk (temporary until common lib ready) |
|
||||
| `cameleer3-server-core/.../model/FlatProcessorRecord.java` | Server-side DTO mirroring agent's FlatProcessorRecord (temporary until common lib ready) |
|
||||
| `cameleer3-server-core/.../ingestion/ChunkAccumulator.java` | Accumulates exchange envelope across chunks, pushes processor records + final execution row to WriteBuffers |
|
||||
| `cameleer3-server-core/.../ingestion/MergedExecution.java` | Record holding merged execution envelope + version + tenant |
|
||||
| `cameleer3-server-app/.../storage/ClickHouseExecutionStore.java` | Batch INSERT for executions + processor_executions to ClickHouse |
|
||||
| `cameleer3-server-app/.../search/ClickHouseSearchIndex.java` | SearchIndex impl using SQL with ngram indexes |
|
||||
| `cameleer3-server-app/.../ingestion/ExecutionFlushScheduler.java` | Drains execution + processor WriteBuffers → ClickHouseExecutionStore |
|
||||
| `cameleer3-server-app/.../controller/ChunkIngestionController.java` | REST endpoint `POST /api/v1/data/chunks` accepting ExecutionChunk |
|
||||
| `cameleer3-server-app/.../config/StorageBeanConfig.java` | Modified: add chunk accumulator + CH search beans |
|
||||
| `cameleer3-server-app/.../config/IngestionBeanConfig.java` | Modified: add execution + processor WriteBuffer beans |
|
||||
| `cameleer3-server-app/.../resources/application.yml` | Modified: add `cameleer.storage.search` flag |
|
||||
| `cameleer3-server-app/...test.../storage/ClickHouseExecutionStoreIT.java` | Integration test for CH execution writes |
|
||||
| `cameleer3-server-app/...test.../search/ClickHouseSearchIndexIT.java` | Integration test for CH search |
|
||||
| `cameleer3-server-core/...test.../ingestion/ChunkAccumulatorTest.java` | Unit test for accumulator logic |
|
||||
| `cameleer-server-app/.../resources/clickhouse/V2__executions.sql` | DDL for `executions` table (ReplacingMergeTree) |
|
||||
| `cameleer-server-app/.../resources/clickhouse/V3__processor_executions.sql` | DDL for `processor_executions` table (MergeTree) — uses seq/parentSeq/iteration |
|
||||
| `cameleer-server-core/.../model/ExecutionChunk.java` | Server-side DTO mirroring agent's ExecutionChunk (temporary until common lib ready) |
|
||||
| `cameleer-server-core/.../model/FlatProcessorRecord.java` | Server-side DTO mirroring agent's FlatProcessorRecord (temporary until common lib ready) |
|
||||
| `cameleer-server-core/.../ingestion/ChunkAccumulator.java` | Accumulates exchange envelope across chunks, pushes processor records + final execution row to WriteBuffers |
|
||||
| `cameleer-server-core/.../ingestion/MergedExecution.java` | Record holding merged execution envelope + version + tenant |
|
||||
| `cameleer-server-app/.../storage/ClickHouseExecutionStore.java` | Batch INSERT for executions + processor_executions to ClickHouse |
|
||||
| `cameleer-server-app/.../search/ClickHouseSearchIndex.java` | SearchIndex impl using SQL with ngram indexes |
|
||||
| `cameleer-server-app/.../ingestion/ExecutionFlushScheduler.java` | Drains execution + processor WriteBuffers → ClickHouseExecutionStore |
|
||||
| `cameleer-server-app/.../controller/ChunkIngestionController.java` | REST endpoint `POST /api/v1/data/chunks` accepting ExecutionChunk |
|
||||
| `cameleer-server-app/.../config/StorageBeanConfig.java` | Modified: add chunk accumulator + CH search beans |
|
||||
| `cameleer-server-app/.../config/IngestionBeanConfig.java` | Modified: add execution + processor WriteBuffer beans |
|
||||
| `cameleer-server-app/.../resources/application.yml` | Modified: add `cameleer.storage.search` flag |
|
||||
| `cameleer-server-app/...test.../storage/ClickHouseExecutionStoreIT.java` | Integration test for CH execution writes |
|
||||
| `cameleer-server-app/...test.../search/ClickHouseSearchIndexIT.java` | Integration test for CH search |
|
||||
| `cameleer-server-core/...test.../ingestion/ChunkAccumulatorTest.java` | Unit test for accumulator logic |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Server-Side DTOs (ExecutionChunk + FlatProcessorRecord)
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/FlatProcessorRecord.java`
|
||||
- Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/ExecutionChunk.java`
|
||||
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/FlatProcessorRecord.java`
|
||||
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/ExecutionChunk.java`
|
||||
|
||||
These mirror the agent's models exactly. When `cameleer3-common` is published with these classes, delete these files and update imports.
|
||||
These mirror the agent's models exactly. When `cameleer-common` is published with these classes, delete these files and update imports.
|
||||
|
||||
- [ ] **Step 1: Create FlatProcessorRecord**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/FlatProcessorRecord.java
|
||||
package com.cameleer3.server.core.storage.model;
|
||||
// cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/FlatProcessorRecord.java
|
||||
package com.cameleer.server.core.storage.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
@@ -59,7 +59,7 @@ import java.util.Map;
|
||||
|
||||
/**
|
||||
* Flat processor execution record with seq/parentSeq for tree reconstruction.
|
||||
* Mirrors cameleer3-common FlatProcessorRecord — replace with common lib import when available.
|
||||
* Mirrors cameleer-common FlatProcessorRecord — replace with common lib import when available.
|
||||
*/
|
||||
@JsonIgnoreProperties(ignoreUnknown = true)
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
@@ -96,8 +96,8 @@ public record FlatProcessorRecord(
|
||||
- [ ] **Step 2: Create ExecutionChunk**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/ExecutionChunk.java
|
||||
package com.cameleer3.server.core.storage.model;
|
||||
// cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/ExecutionChunk.java
|
||||
package com.cameleer.server.core.storage.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
@@ -109,7 +109,7 @@ import java.util.Map;
|
||||
|
||||
/**
|
||||
* Chunk document: exchange envelope + list of FlatProcessorRecords.
|
||||
* Mirrors cameleer3-common ExecutionChunk — replace with common lib import when available.
|
||||
* Mirrors cameleer-common ExecutionChunk — replace with common lib import when available.
|
||||
*/
|
||||
@JsonIgnoreProperties(ignoreUnknown = true)
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
@@ -144,8 +144,8 @@ public record ExecutionChunk(
|
||||
- [ ] **Step 3: Write deserialization test**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/test/java/com/cameleer3/server/core/storage/model/ExecutionChunkDeserializationTest.java
|
||||
package com.cameleer3.server.core.storage.model;
|
||||
// cameleer-server-core/src/test/java/com/cameleer/server/core/storage/model/ExecutionChunkDeserializationTest.java
|
||||
package com.cameleer.server.core.storage.model;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
|
||||
@@ -259,7 +259,7 @@ class ExecutionChunkDeserializationTest {
|
||||
- [ ] **Step 4: Run tests**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-core -Dtest=ExecutionChunkDeserializationTest
|
||||
mvn test -pl cameleer-server-core -Dtest=ExecutionChunkDeserializationTest
|
||||
```
|
||||
|
||||
Expected: PASS (3 tests).
|
||||
@@ -267,9 +267,9 @@ Expected: PASS (3 tests).
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/FlatProcessorRecord.java \
|
||||
cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/ExecutionChunk.java \
|
||||
cameleer3-server-core/src/test/java/com/cameleer3/server/core/storage/model/ExecutionChunkDeserializationTest.java
|
||||
git add cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/FlatProcessorRecord.java \
|
||||
cameleer-server-core/src/main/java/com/cameleer/server/core/storage/model/ExecutionChunk.java \
|
||||
cameleer-server-core/src/test/java/com/cameleer/server/core/storage/model/ExecutionChunkDeserializationTest.java
|
||||
git commit -m "feat: add server-side ExecutionChunk and FlatProcessorRecord DTOs"
|
||||
```
|
||||
|
||||
@@ -278,8 +278,8 @@ git commit -m "feat: add server-side ExecutionChunk and FlatProcessorRecord DTOs
|
||||
### Task 2: DDL Scripts for executions and processor_executions
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-app/src/main/resources/clickhouse/V2__executions.sql`
|
||||
- Create: `cameleer3-server-app/src/main/resources/clickhouse/V3__processor_executions.sql`
|
||||
- Create: `cameleer-server-app/src/main/resources/clickhouse/V2__executions.sql`
|
||||
- Create: `cameleer-server-app/src/main/resources/clickhouse/V3__processor_executions.sql`
|
||||
|
||||
- [ ] **Step 1: Create executions DDL**
|
||||
|
||||
@@ -395,7 +395,7 @@ SETTINGS index_granularity = 8192;
|
||||
- [ ] **Step 3: Verify DDL loads**
|
||||
|
||||
```bash
|
||||
mvn clean compile -pl cameleer3-server-app
|
||||
mvn clean compile -pl cameleer-server-app
|
||||
```
|
||||
|
||||
`ClickHouseSchemaInitializer` scans `classpath:clickhouse/*.sql` automatically.
|
||||
@@ -403,8 +403,8 @@ mvn clean compile -pl cameleer3-server-app
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-app/src/main/resources/clickhouse/V2__executions.sql \
|
||||
cameleer3-server-app/src/main/resources/clickhouse/V3__processor_executions.sql
|
||||
git add cameleer-server-app/src/main/resources/clickhouse/V2__executions.sql \
|
||||
cameleer-server-app/src/main/resources/clickhouse/V3__processor_executions.sql
|
||||
git commit -m "feat(clickhouse): add executions and processor_executions DDL for chunked transport"
|
||||
```
|
||||
|
||||
@@ -413,17 +413,17 @@ git commit -m "feat(clickhouse): add executions and processor_executions DDL for
|
||||
### Task 3: MergedExecution + ClickHouseExecutionStore
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/MergedExecution.java`
|
||||
- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionStore.java`
|
||||
- Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseExecutionStoreIT.java`
|
||||
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/MergedExecution.java`
|
||||
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionStore.java`
|
||||
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseExecutionStoreIT.java`
|
||||
|
||||
The store handles batch INSERT for both `executions` (from MergedExecution) and `processor_executions` (from FlatProcessorRecord). It does NOT implement the `ExecutionStore` interface — it has its own batch API consumed by the flush scheduler.
|
||||
|
||||
- [ ] **Step 1: Create MergedExecution record**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/MergedExecution.java
|
||||
package com.cameleer3.server.core.ingestion;
|
||||
// cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/MergedExecution.java
|
||||
package com.cameleer.server.core.ingestion;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
@@ -468,11 +468,11 @@ public record MergedExecution(
|
||||
- [ ] **Step 2: Write the failing integration test**
|
||||
|
||||
```java
|
||||
// cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseExecutionStoreIT.java
|
||||
package com.cameleer3.server.app.storage;
|
||||
// cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseExecutionStoreIT.java
|
||||
package com.cameleer.server.app.storage;
|
||||
|
||||
import com.cameleer3.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer3.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.cameleer.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -654,7 +654,7 @@ class ClickHouseExecutionStoreIT {
|
||||
- [ ] **Step 3: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-app -Dtest=ClickHouseExecutionStoreIT -Dfailsafe.provider=surefire -DfailIfNoTests=false
|
||||
mvn test -pl cameleer-server-app -Dtest=ClickHouseExecutionStoreIT -Dfailsafe.provider=surefire -DfailIfNoTests=false
|
||||
```
|
||||
|
||||
Expected: compilation error — `ClickHouseExecutionStore` does not exist.
|
||||
@@ -662,11 +662,11 @@ Expected: compilation error — `ClickHouseExecutionStore` does not exist.
|
||||
- [ ] **Step 4: Implement ClickHouseExecutionStore**
|
||||
|
||||
```java
|
||||
// cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionStore.java
|
||||
package com.cameleer3.server.app.storage;
|
||||
// cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionStore.java
|
||||
package com.cameleer.server.app.storage;
|
||||
|
||||
import com.cameleer3.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer3.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.cameleer.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
@@ -840,7 +840,7 @@ public class ClickHouseExecutionStore {
|
||||
- [ ] **Step 5: Run test to verify it passes**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-app -Dtest=ClickHouseExecutionStoreIT -Dfailsafe.provider=surefire
|
||||
mvn test -pl cameleer-server-app -Dtest=ClickHouseExecutionStoreIT -Dfailsafe.provider=surefire
|
||||
```
|
||||
|
||||
Expected: all 5 tests PASS.
|
||||
@@ -848,9 +848,9 @@ Expected: all 5 tests PASS.
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/MergedExecution.java \
|
||||
cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseExecutionStore.java \
|
||||
cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseExecutionStoreIT.java
|
||||
git add cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/MergedExecution.java \
|
||||
cameleer-server-app/src/main/java/com/cameleer/server/app/storage/ClickHouseExecutionStore.java \
|
||||
cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseExecutionStoreIT.java
|
||||
git commit -m "feat(clickhouse): add ClickHouseExecutionStore with batch insert for chunked format"
|
||||
```
|
||||
|
||||
@@ -859,8 +859,8 @@ git commit -m "feat(clickhouse): add ClickHouseExecutionStore with batch insert
|
||||
### Task 4: ChunkAccumulator
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/ChunkAccumulatorTest.java`
|
||||
- Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/ChunkAccumulator.java`
|
||||
- Create: `cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/ChunkAccumulatorTest.java`
|
||||
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/ChunkAccumulator.java`
|
||||
|
||||
The ChunkAccumulator receives `ExecutionChunk` documents. For each chunk:
|
||||
- Processor records are pushed to a sink immediately (they're append-only)
|
||||
@@ -872,11 +872,11 @@ A scheduled sweep flushes stale exchanges (no final chunk received within 5 minu
|
||||
- [ ] **Step 1: Write the failing unit test**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/ChunkAccumulatorTest.java
|
||||
package com.cameleer3.server.core.ingestion;
|
||||
// cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/ChunkAccumulatorTest.java
|
||||
package com.cameleer.server.core.ingestion;
|
||||
|
||||
import com.cameleer3.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer3.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.cameleer.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer.server.core.storage.model.FlatProcessorRecord;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -1060,7 +1060,7 @@ class ChunkAccumulatorTest {
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-core -Dtest=ChunkAccumulatorTest -DfailIfNoTests=false
|
||||
mvn test -pl cameleer-server-core -Dtest=ChunkAccumulatorTest -DfailIfNoTests=false
|
||||
```
|
||||
|
||||
Expected: compilation error — `ChunkAccumulator` does not exist.
|
||||
@@ -1068,11 +1068,11 @@ Expected: compilation error — `ChunkAccumulator` does not exist.
|
||||
- [ ] **Step 3: Implement ChunkAccumulator**
|
||||
|
||||
```java
|
||||
// cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/ChunkAccumulator.java
|
||||
package com.cameleer3.server.core.ingestion;
|
||||
// cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/ChunkAccumulator.java
|
||||
package com.cameleer.server.core.ingestion;
|
||||
|
||||
import com.cameleer3.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer3.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.cameleer.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.slf4j.Logger;
|
||||
@@ -1267,7 +1267,7 @@ public class ChunkAccumulator {
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-core -Dtest=ChunkAccumulatorTest
|
||||
mvn test -pl cameleer-server-core -Dtest=ChunkAccumulatorTest
|
||||
```
|
||||
|
||||
Expected: all 5 tests PASS.
|
||||
@@ -1275,8 +1275,8 @@ Expected: all 5 tests PASS.
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/ingestion/ChunkAccumulator.java \
|
||||
cameleer3-server-core/src/test/java/com/cameleer3/server/core/ingestion/ChunkAccumulatorTest.java
|
||||
git add cameleer-server-core/src/main/java/com/cameleer/server/core/ingestion/ChunkAccumulator.java \
|
||||
cameleer-server-core/src/test/java/com/cameleer/server/core/ingestion/ChunkAccumulatorTest.java
|
||||
git commit -m "feat(clickhouse): add ChunkAccumulator for chunked execution ingestion"
|
||||
```
|
||||
|
||||
@@ -1285,22 +1285,22 @@ git commit -m "feat(clickhouse): add ChunkAccumulator for chunked execution inge
|
||||
### Task 5: ExecutionFlushScheduler + ChunkIngestionController
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ExecutionFlushScheduler.java`
|
||||
- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ChunkIngestionController.java`
|
||||
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ExecutionFlushScheduler.java`
|
||||
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ChunkIngestionController.java`
|
||||
|
||||
- [ ] **Step 1: Implement ExecutionFlushScheduler**
|
||||
|
||||
Follows `MetricsFlushScheduler` pattern. Drains two WriteBuffers (executions + processor batches) and calls ClickHouseExecutionStore. Also runs the stale sweep.
|
||||
|
||||
```java
|
||||
// cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ExecutionFlushScheduler.java
|
||||
package com.cameleer3.server.app.ingestion;
|
||||
// cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ExecutionFlushScheduler.java
|
||||
package com.cameleer.server.app.ingestion;
|
||||
|
||||
import com.cameleer3.server.app.config.IngestionConfig;
|
||||
import com.cameleer3.server.app.storage.ClickHouseExecutionStore;
|
||||
import com.cameleer3.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer3.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer3.server.core.ingestion.WriteBuffer;
|
||||
import com.cameleer.server.app.config.IngestionConfig;
|
||||
import com.cameleer.server.app.storage.ClickHouseExecutionStore;
|
||||
import com.cameleer.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer.server.core.ingestion.WriteBuffer;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.context.SmartLifecycle;
|
||||
@@ -1395,11 +1395,11 @@ public class ExecutionFlushScheduler implements SmartLifecycle {
|
||||
- [ ] **Step 2: Implement ChunkIngestionController**
|
||||
|
||||
```java
|
||||
// cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ChunkIngestionController.java
|
||||
package com.cameleer3.server.app.controller;
|
||||
// cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ChunkIngestionController.java
|
||||
package com.cameleer.server.app.controller;
|
||||
|
||||
import com.cameleer3.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer3.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer.server.core.storage.model.ExecutionChunk;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
@@ -1450,14 +1450,14 @@ public class ChunkIngestionController {
|
||||
- [ ] **Step 3: Compile**
|
||||
|
||||
```bash
|
||||
mvn clean compile -pl cameleer3-server-app
|
||||
mvn clean compile -pl cameleer-server-app
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/ingestion/ExecutionFlushScheduler.java \
|
||||
cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/ChunkIngestionController.java
|
||||
git add cameleer-server-app/src/main/java/com/cameleer/server/app/ingestion/ExecutionFlushScheduler.java \
|
||||
cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ChunkIngestionController.java
|
||||
git commit -m "feat(clickhouse): add ExecutionFlushScheduler and ChunkIngestionController"
|
||||
```
|
||||
|
||||
@@ -1466,8 +1466,8 @@ git commit -m "feat(clickhouse): add ExecutionFlushScheduler and ChunkIngestionC
|
||||
### Task 6: ClickHouseSearchIndex
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/search/ClickHouseSearchIndexIT.java`
|
||||
- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/search/ClickHouseSearchIndex.java`
|
||||
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/search/ClickHouseSearchIndexIT.java`
|
||||
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/search/ClickHouseSearchIndex.java`
|
||||
|
||||
Same as the original plan — implements `SearchIndex` using SQL against ClickHouse. The search query patterns are unchanged: `_search_text LIKE '%term%'` on executions, subquery join on processor_executions for body/header/error scoped searches.
|
||||
|
||||
@@ -1480,7 +1480,7 @@ Use the same test class from the original plan's Task 5, Step 1. The test seeds
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-app -Dtest=ClickHouseSearchIndexIT -Dfailsafe.provider=surefire -DfailIfNoTests=false
|
||||
mvn test -pl cameleer-server-app -Dtest=ClickHouseSearchIndexIT -Dfailsafe.provider=surefire -DfailIfNoTests=false
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement ClickHouseSearchIndex**
|
||||
@@ -1490,14 +1490,14 @@ Use the same implementation from the original plan's Task 5, Step 3. The SQL que
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-app -Dtest=ClickHouseSearchIndexIT -Dfailsafe.provider=surefire
|
||||
mvn test -pl cameleer-server-app -Dtest=ClickHouseSearchIndexIT -Dfailsafe.provider=surefire
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/search/ClickHouseSearchIndex.java \
|
||||
cameleer3-server-app/src/test/java/com/cameleer3/server/app/search/ClickHouseSearchIndexIT.java
|
||||
git add cameleer-server-app/src/main/java/com/cameleer/server/app/search/ClickHouseSearchIndex.java \
|
||||
cameleer-server-app/src/test/java/com/cameleer/server/app/search/ClickHouseSearchIndexIT.java
|
||||
git commit -m "feat(clickhouse): add ClickHouseSearchIndex with ngram-accelerated SQL search"
|
||||
```
|
||||
|
||||
@@ -1506,10 +1506,10 @@ git commit -m "feat(clickhouse): add ClickHouseSearchIndex with ngram-accelerate
|
||||
### Task 7: Feature Flag Wiring
|
||||
|
||||
**Files:**
|
||||
- Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java`
|
||||
- Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionBeanConfig.java`
|
||||
- Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/search/OpenSearchIndex.java`
|
||||
- Modify: `cameleer3-server-app/src/main/resources/application.yml`
|
||||
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/config/StorageBeanConfig.java`
|
||||
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionBeanConfig.java`
|
||||
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/search/OpenSearchIndex.java`
|
||||
- Modify: `cameleer-server-app/src/main/resources/application.yml`
|
||||
- Modify: `deploy/base/server.yaml`
|
||||
|
||||
Wire up the ChunkAccumulator, WriteBuffers, flush scheduler, and search switching.
|
||||
@@ -1608,10 +1608,10 @@ mvn clean verify -DskipITs
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java \
|
||||
cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/IngestionBeanConfig.java \
|
||||
cameleer3-server-app/src/main/java/com/cameleer3/server/app/search/OpenSearchIndex.java \
|
||||
cameleer3-server-app/src/main/resources/application.yml \
|
||||
git add cameleer-server-app/src/main/java/com/cameleer/server/app/config/StorageBeanConfig.java \
|
||||
cameleer-server-app/src/main/java/com/cameleer/server/app/config/IngestionBeanConfig.java \
|
||||
cameleer-server-app/src/main/java/com/cameleer/server/app/search/OpenSearchIndex.java \
|
||||
cameleer-server-app/src/main/resources/application.yml \
|
||||
deploy/base/server.yaml
|
||||
git commit -m "feat(clickhouse): wire ChunkAccumulator, flush scheduler, and search feature flag"
|
||||
```
|
||||
@@ -1621,24 +1621,24 @@ git commit -m "feat(clickhouse): wire ChunkAccumulator, flush scheduler, and sea
|
||||
### Task 8: End-to-End Integration Test
|
||||
|
||||
**Files:**
|
||||
- Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseChunkPipelineIT.java`
|
||||
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseChunkPipelineIT.java`
|
||||
|
||||
Validates the full pipeline: ChunkAccumulator → WriteBuffers → ClickHouseExecutionStore → ClickHouseSearchIndex.
|
||||
|
||||
- [ ] **Step 1: Write the integration test**
|
||||
|
||||
```java
|
||||
// cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseChunkPipelineIT.java
|
||||
package com.cameleer3.server.app.storage;
|
||||
// cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseChunkPipelineIT.java
|
||||
package com.cameleer.server.app.storage;
|
||||
|
||||
import com.cameleer3.server.app.search.ClickHouseSearchIndex;
|
||||
import com.cameleer3.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer3.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer3.server.core.search.ExecutionSummary;
|
||||
import com.cameleer3.server.core.search.SearchRequest;
|
||||
import com.cameleer3.server.core.search.SearchResult;
|
||||
import com.cameleer3.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer3.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.cameleer.server.app.search.ClickHouseSearchIndex;
|
||||
import com.cameleer.server.core.ingestion.ChunkAccumulator;
|
||||
import com.cameleer.server.core.ingestion.MergedExecution;
|
||||
import com.cameleer.server.core.search.ExecutionSummary;
|
||||
import com.cameleer.server.core.search.SearchRequest;
|
||||
import com.cameleer.server.core.search.SearchResult;
|
||||
import com.cameleer.server.core.storage.model.ExecutionChunk;
|
||||
import com.cameleer.server.core.storage.model.FlatProcessorRecord;
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -1797,7 +1797,7 @@ class ClickHouseChunkPipelineIT {
|
||||
- [ ] **Step 2: Run the integration test**
|
||||
|
||||
```bash
|
||||
mvn test -pl cameleer3-server-app -Dtest=ClickHouseChunkPipelineIT -Dfailsafe.provider=surefire
|
||||
mvn test -pl cameleer-server-app -Dtest=ClickHouseChunkPipelineIT -Dfailsafe.provider=surefire
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
@@ -1805,7 +1805,7 @@ Expected: PASS.
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseChunkPipelineIT.java
|
||||
git add cameleer-server-app/src/test/java/com/cameleer/server/app/storage/ClickHouseChunkPipelineIT.java
|
||||
git commit -m "test(clickhouse): add end-to-end chunk pipeline integration test"
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user