diff --git a/docs/superpowers/plans/2026-03-31-clickhouse-phase1-foundation.md b/docs/superpowers/plans/2026-03-31-clickhouse-phase1-foundation.md new file mode 100644 index 00000000..6e84d575 --- /dev/null +++ b/docs/superpowers/plans/2026-03-31-clickhouse-phase1-foundation.md @@ -0,0 +1,1193 @@ +# ClickHouse Migration Phase 1: Foundation + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add ClickHouse as a second data store alongside PostgreSQL, validated with the metrics pipeline end-to-end (write + read). Feature-flagged so either PG or CH serves metrics. + +**Architecture:** Add `clickhouse-jdbc` dependency, a second `DataSource`/`JdbcTemplate` bean qualified as `@ClickHouse`, an idempotent schema initializer that creates ClickHouse tables on startup, and a feature flag (`cameleer.storage.metrics=clickhouse|postgres`) that selects the active `MetricsStore` implementation. The metrics read path (`AgentMetricsController`) is refactored behind a new `MetricsQueryStore` interface so it too can be swapped. + +**Tech Stack:** ClickHouse JDBC 0.7.x, Spring Boot 3.4.3, JdbcTemplate, Testcontainers (clickhouse module) + +**Design Spec:** `docs/superpowers/specs/2026-03-31-clickhouse-migration-design.md` + +--- + +## File Map + +| Action | File | Responsibility | +|--------|------|----------------| +| Modify | `cameleer3-server-app/pom.xml` | Add clickhouse-jdbc + testcontainers-clickhouse dependencies | +| Modify | `cameleer3-server-app/src/main/resources/application.yml` | Add `clickhouse.*` and `cameleer.storage.metrics` config keys | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java` | ClickHouse DataSource + JdbcTemplate beans | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java` | Config properties class for `clickhouse.*` prefix | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java` | Runs idempotent DDL on startup | +| Create | `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` | DDL for `agent_metrics` table | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java` | `MetricsStore` impl writing to ClickHouse | +| Create | `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java` | Interface: query metrics time-series | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java` | PG impl of MetricsQueryStore (extracted from controller) | +| Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java` | CH impl of MetricsQueryStore | +| Modify | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java` | Use MetricsQueryStore instead of raw JdbcTemplate | +| Modify | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java` | Conditional beans for PG vs CH metrics stores | +| Create | `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java` | Unit test | +| Create | `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java` | Unit test | +| Create | `deploy/clickhouse.yaml` | K8s StatefulSet + Service for ClickHouse | +| Modify | `deploy/base/server.yaml` | Add CLICKHOUSE_URL env var | + +--- + +### Task 1: Add ClickHouse Dependencies + +**Files:** +- Modify: `cameleer3-server-app/pom.xml` + +- [ ] **Step 1: Add clickhouse-jdbc and testcontainers-clickhouse to app POM** + +In `cameleer3-server-app/pom.xml`, add these two dependencies. Place the runtime dependency after the opensearch dependencies (around line 59), and the test dependency after the opensearch-testcontainers dependency (around line 128): + +```xml + + + com.clickhouse + clickhouse-jdbc + 0.7.1-patch5 + all + + + + + org.testcontainers + clickhouse + test + +``` + +Note: The `all` classifier bundles the HTTP client, avoiding transitive dependency conflicts. The testcontainers `clickhouse` module version is managed by the `testcontainers.version` property in the parent POM. + +- [ ] **Step 2: Verify compilation** + +Run: `mvn clean compile -pl cameleer3-server-app -am` +Expected: BUILD SUCCESS + +- [ ] **Step 3: Commit** + +```bash +git add cameleer3-server-app/pom.xml +git commit -m "build: add clickhouse-jdbc and testcontainers-clickhouse dependencies" +``` + +--- + +### Task 2: ClickHouse Configuration + +**Files:** +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java` +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java` +- Modify: `cameleer3-server-app/src/main/resources/application.yml` + +- [ ] **Step 1: Create ClickHouseProperties** + +```java +package com.cameleer3.server.app.config; + +import org.springframework.boot.context.properties.ConfigurationProperties; + +@ConfigurationProperties(prefix = "clickhouse") +public class ClickHouseProperties { + + private String url = "jdbc:clickhouse://localhost:8123/cameleer"; + private String username = "default"; + private String password = ""; + + public String getUrl() { return url; } + public void setUrl(String url) { this.url = url; } + + public String getUsername() { return username; } + public void setUsername(String username) { this.username = username; } + + public String getPassword() { return password; } + public void setPassword(String password) { this.password = password; } +} +``` + +- [ ] **Step 2: Create ClickHouseConfig** + +```java +package com.cameleer3.server.app.config; + +import com.zaxxer.hikari.HikariDataSource; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.JdbcTemplate; + +import javax.sql.DataSource; + +@Configuration +@EnableConfigurationProperties(ClickHouseProperties.class) +@ConditionalOnProperty(name = "clickhouse.enabled", havingValue = "true") +public class ClickHouseConfig { + + @Bean(name = "clickHouseDataSource") + public DataSource clickHouseDataSource(ClickHouseProperties props) { + HikariDataSource ds = new HikariDataSource(); + ds.setJdbcUrl(props.getUrl()); + ds.setUsername(props.getUsername()); + ds.setPassword(props.getPassword()); + ds.setMaximumPoolSize(10); + ds.setPoolName("clickhouse-pool"); + return ds; + } + + @Bean(name = "clickHouseJdbcTemplate") + public JdbcTemplate clickHouseJdbcTemplate( + @Qualifier("clickHouseDataSource") DataSource ds) { + return new JdbcTemplate(ds); + } +} +``` + +- [ ] **Step 3: Add configuration to application.yml** + +Add at the end of `application.yml`: + +```yaml +clickhouse: + enabled: ${CLICKHOUSE_ENABLED:false} + url: ${CLICKHOUSE_URL:jdbc:clickhouse://localhost:8123/cameleer?async_insert=1&wait_for_async_insert=0} + username: ${CLICKHOUSE_USERNAME:default} + password: ${CLICKHOUSE_PASSWORD:} + +cameleer: + storage: + metrics: ${CAMELEER_STORAGE_METRICS:postgres} +``` + +Note: The existing `cameleer:` block already has `body-size-limit` and `retention-days`. Add the `storage.metrics` key under it. + +- [ ] **Step 4: Verify compilation** + +Run: `mvn clean compile -pl cameleer3-server-app -am` +Expected: BUILD SUCCESS + +- [ ] **Step 5: Commit** + +```bash +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java +git add cameleer3-server-app/src/main/resources/application.yml +git commit -m "feat: add ClickHouse DataSource and JdbcTemplate configuration" +``` + +--- + +### Task 3: Schema Initializer + DDL Script + +**Files:** +- Create: `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java` + +- [ ] **Step 1: Create DDL script** + +File: `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` + +```sql +CREATE TABLE IF NOT EXISTS agent_metrics ( + tenant_id LowCardinality(String) DEFAULT 'default', + collected_at DateTime64(3), + agent_id LowCardinality(String), + metric_name LowCardinality(String), + metric_value Float64, + tags Map(String, String) DEFAULT map(), + server_received_at DateTime64(3) DEFAULT now64(3) +) +ENGINE = MergeTree() +PARTITION BY (tenant_id, toYYYYMM(collected_at)) +ORDER BY (tenant_id, agent_id, metric_name, collected_at) +TTL collected_at + INTERVAL 365 DAY DELETE +SETTINGS index_granularity = 8192; +``` + +- [ ] **Step 2: Create ClickHouseSchemaInitializer** + +```java +package com.cameleer3.server.app.config; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.event.ApplicationReadyEvent; +import org.springframework.context.event.EventListener; +import org.springframework.core.io.Resource; +import org.springframework.core.io.support.PathMatchingResourcePatternResolver; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Component; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Comparator; + +@Component +@ConditionalOnProperty(name = "clickhouse.enabled", havingValue = "true") +public class ClickHouseSchemaInitializer { + + private static final Logger log = LoggerFactory.getLogger(ClickHouseSchemaInitializer.class); + + private final JdbcTemplate clickHouseJdbc; + + public ClickHouseSchemaInitializer( + @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { + this.clickHouseJdbc = clickHouseJdbc; + } + + @EventListener(ApplicationReadyEvent.class) + public void initializeSchema() throws IOException { + PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(); + Resource[] scripts = resolver.getResources("classpath:clickhouse/*.sql"); + + Arrays.sort(scripts, Comparator.comparing(Resource::getFilename)); + + for (Resource script : scripts) { + String sql = script.getContentAsString(StandardCharsets.UTF_8); + log.info("Executing ClickHouse schema script: {}", script.getFilename()); + for (String statement : sql.split(";")) { + String trimmed = statement.trim(); + if (!trimmed.isEmpty()) { + clickHouseJdbc.execute(trimmed); + } + } + } + + log.info("ClickHouse schema initialization complete ({} scripts)", scripts.length); + } +} +``` + +- [ ] **Step 3: Verify compilation** + +Run: `mvn clean compile -pl cameleer3-server-app -am` +Expected: BUILD SUCCESS + +- [ ] **Step 4: Commit** + +```bash +git add cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java +git commit -m "feat: add ClickHouse schema initializer with agent_metrics DDL" +``` + +--- + +### Task 4: ClickHouseMetricsStore (TDD) + +**Files:** +- Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java` +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java` + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer3.server.app.storage; + +import com.cameleer3.server.core.storage.model.MetricsSnapshot; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.testcontainers.clickhouse.ClickHouseContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; + +import com.zaxxer.hikari.HikariDataSource; + +import java.time.Instant; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +@Testcontainers +class ClickHouseMetricsStoreTest { + + @Container + static final ClickHouseContainer clickhouse = + new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); + + private JdbcTemplate jdbc; + private ClickHouseMetricsStore store; + + @BeforeEach + void setUp() { + HikariDataSource ds = new HikariDataSource(); + ds.setJdbcUrl(clickhouse.getJdbcUrl()); + ds.setUsername(clickhouse.getUsername()); + ds.setPassword(clickhouse.getPassword()); + + jdbc = new JdbcTemplate(ds); + + jdbc.execute(""" + CREATE TABLE IF NOT EXISTS agent_metrics ( + tenant_id LowCardinality(String) DEFAULT 'default', + collected_at DateTime64(3), + agent_id LowCardinality(String), + metric_name LowCardinality(String), + metric_value Float64, + tags Map(String, String) DEFAULT map(), + server_received_at DateTime64(3) DEFAULT now64(3) + ) + ENGINE = MergeTree() + ORDER BY (tenant_id, agent_id, metric_name, collected_at) + """); + + jdbc.execute("TRUNCATE TABLE agent_metrics"); + + store = new ClickHouseMetricsStore(jdbc); + } + + @Test + void insertBatch_writesMetricsToClickHouse() { + List batch = List.of( + new MetricsSnapshot("agent-1", Instant.parse("2026-03-31T10:00:00Z"), + "cpu.usage", 75.5, Map.of("host", "server-1")), + new MetricsSnapshot("agent-1", Instant.parse("2026-03-31T10:00:01Z"), + "memory.free", 1024.0, null) + ); + + store.insertBatch(batch); + + Integer count = jdbc.queryForObject( + "SELECT count() FROM agent_metrics WHERE agent_id = 'agent-1'", + Integer.class); + assertThat(count).isEqualTo(2); + } + + @Test + void insertBatch_storesTags() { + store.insertBatch(List.of( + new MetricsSnapshot("agent-2", Instant.parse("2026-03-31T10:00:00Z"), + "disk.used", 500.0, Map.of("mount", "/data", "fs", "ext4")) + )); + + Map tags = jdbc.queryForObject( + "SELECT tags FROM agent_metrics WHERE agent_id = 'agent-2'", + (rs, n) -> { + // ClickHouse returns Map as a map-like structure via JDBC + @SuppressWarnings("unchecked") + Map m = (Map) rs.getObject("tags"); + return m; + }); + assertThat(tags).containsEntry("mount", "/data").containsEntry("fs", "ext4"); + } + + @Test + void insertBatch_emptyList_doesNothing() { + store.insertBatch(List.of()); + + Integer count = jdbc.queryForObject("SELECT count() FROM agent_metrics", Integer.class); + assertThat(count).isEqualTo(0); + } + + @Test + void insertBatch_nullTags_defaultsToEmptyMap() { + store.insertBatch(List.of( + new MetricsSnapshot("agent-3", Instant.parse("2026-03-31T10:00:00Z"), + "cpu.usage", 50.0, null) + )); + + Integer count = jdbc.queryForObject( + "SELECT count() FROM agent_metrics WHERE agent_id = 'agent-3'", + Integer.class); + assertThat(count).isEqualTo(1); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsStoreTest -DfailIfNoTests=false` +Expected: FAIL — `ClickHouseMetricsStore` class does not exist + +- [ ] **Step 3: Write ClickHouseMetricsStore** + +```java +package com.cameleer3.server.app.storage; + +import com.cameleer3.server.core.storage.MetricsStore; +import com.cameleer3.server.core.storage.model.MetricsSnapshot; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.sql.Timestamp; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class ClickHouseMetricsStore implements MetricsStore { + + private final JdbcTemplate jdbc; + + public ClickHouseMetricsStore(JdbcTemplate jdbc) { + this.jdbc = jdbc; + } + + @Override + public void insertBatch(List snapshots) { + if (snapshots.isEmpty()) return; + + jdbc.batchUpdate(""" + INSERT INTO agent_metrics (agent_id, metric_name, metric_value, tags, collected_at) + VALUES (?, ?, ?, ?, ?) + """, + snapshots.stream().map(s -> new Object[]{ + s.agentId(), + s.metricName(), + s.metricValue(), + tagsToClickHouseMap(s.tags()), + Timestamp.from(s.collectedAt()) + }).toList()); + } + + private Map tagsToClickHouseMap(Map tags) { + if (tags == null || tags.isEmpty()) return new HashMap<>(); + return tags; + } +} +``` + +Note: ClickHouse JDBC driver handles `Map` natively for `Map(String, String)` columns. No JSON serialization needed (unlike the PG implementation that converts to JSONB string). + +- [ ] **Step 4: Run test to verify it passes** + +Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsStoreTest` +Expected: PASS — all 4 tests green + +If the tags test fails due to JDBC Map handling, adjust `tagsToClickHouseMap` to return a `java.util.HashMap` (ClickHouse JDBC requires a mutable map). If needed, wrap: `return new HashMap<>(tags)`. + +- [ ] **Step 5: Commit** + +```bash +git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java +git commit -m "feat: add ClickHouseMetricsStore with batch insert" +``` + +--- + +### Task 5: MetricsQueryStore Interface + PostgreSQL Implementation (TDD) + +Extract the query logic from `AgentMetricsController` into an interface. + +**Files:** +- Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java` +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java` +- Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java` + +- [ ] **Step 1: Create MetricsQueryStore interface** + +```java +package com.cameleer3.server.core.storage; + +import com.cameleer3.server.core.storage.model.MetricTimeSeries; + +import java.time.Instant; +import java.util.List; +import java.util.Map; + +public interface MetricsQueryStore { + + /** + * Query time-bucketed metrics for an agent. + * + * @param agentId the agent identifier + * @param metricNames list of metric names to query + * @param from start of time range (inclusive) + * @param to end of time range (exclusive) + * @param buckets number of time buckets to divide the range into + * @return map of metric name to list of (time, value) pairs + */ + Map> queryTimeSeries( + String agentId, List metricNames, + Instant from, Instant to, int buckets); +} +``` + +- [ ] **Step 2: Create MetricTimeSeries.Bucket record** + +```java +package com.cameleer3.server.core.storage.model; + +import java.time.Instant; +import java.util.List; + +public record MetricTimeSeries(String metricName, List buckets) { + + public record Bucket(Instant time, double value) {} +} +``` + +- [ ] **Step 3: Create PostgresMetricsQueryStore (extract from controller)** + +```java +package com.cameleer3.server.app.storage; + +import com.cameleer3.server.core.storage.MetricsQueryStore; +import com.cameleer3.server.core.storage.model.MetricTimeSeries; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.sql.Timestamp; +import java.time.Instant; +import java.util.*; + +public class PostgresMetricsQueryStore implements MetricsQueryStore { + + private final JdbcTemplate jdbc; + + public PostgresMetricsQueryStore(JdbcTemplate jdbc) { + this.jdbc = jdbc; + } + + @Override + public Map> queryTimeSeries( + String agentId, List metricNames, + Instant from, Instant to, int buckets) { + + long intervalMs = (to.toEpochMilli() - from.toEpochMilli()) / Math.max(buckets, 1); + String intervalStr = intervalMs + " milliseconds"; + + Map> result = new LinkedHashMap<>(); + for (String name : metricNames) { + result.put(name.trim(), new ArrayList<>()); + } + + String sql = """ + SELECT time_bucket(CAST(? AS interval), collected_at) AS bucket, + metric_name, + AVG(metric_value) AS avg_value + FROM agent_metrics + WHERE agent_id = ? + AND collected_at >= ? AND collected_at < ? + AND metric_name = ANY(?) + GROUP BY bucket, metric_name + ORDER BY bucket + """; + + String[] namesArray = metricNames.stream().map(String::trim).toArray(String[]::new); + jdbc.query(sql, rs -> { + String metricName = rs.getString("metric_name"); + Instant bucket = rs.getTimestamp("bucket").toInstant(); + double value = rs.getDouble("avg_value"); + result.computeIfAbsent(metricName, k -> new ArrayList<>()) + .add(new MetricTimeSeries.Bucket(bucket, value)); + }, intervalStr, agentId, Timestamp.from(from), Timestamp.from(to), namesArray); + + return result; + } +} +``` + +- [ ] **Step 4: Refactor AgentMetricsController to use MetricsQueryStore** + +Replace the entire `AgentMetricsController.java` content: + +```java +package com.cameleer3.server.app.controller; + +import com.cameleer3.server.app.dto.AgentMetricsResponse; +import com.cameleer3.server.app.dto.MetricBucket; +import com.cameleer3.server.core.storage.MetricsQueryStore; +import com.cameleer3.server.core.storage.model.MetricTimeSeries; +import org.springframework.web.bind.annotation.*; + +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.*; +import java.util.stream.Collectors; + +@RestController +@RequestMapping("/api/v1/agents/{agentId}/metrics") +public class AgentMetricsController { + + private final MetricsQueryStore metricsQueryStore; + + public AgentMetricsController(MetricsQueryStore metricsQueryStore) { + this.metricsQueryStore = metricsQueryStore; + } + + @GetMapping + public AgentMetricsResponse getMetrics( + @PathVariable String agentId, + @RequestParam String names, + @RequestParam(required = false) Instant from, + @RequestParam(required = false) Instant to, + @RequestParam(defaultValue = "60") int buckets) { + + if (from == null) from = Instant.now().minus(1, ChronoUnit.HOURS); + if (to == null) to = Instant.now(); + + List metricNames = Arrays.asList(names.split(",")); + + Map> raw = + metricsQueryStore.queryTimeSeries(agentId, metricNames, from, to, buckets); + + // Convert to existing DTO format + Map> result = raw.entrySet().stream() + .collect(Collectors.toMap( + Map.Entry::getKey, + e -> e.getValue().stream() + .map(b -> new MetricBucket(b.time(), b.value())) + .toList(), + (a, b) -> a, + LinkedHashMap::new)); + + return new AgentMetricsResponse(result); + } +} +``` + +- [ ] **Step 5: Add PostgresMetricsQueryStore bean to StorageBeanConfig** + +The refactored `AgentMetricsController` needs a `MetricsQueryStore` bean. Add this to `StorageBeanConfig.java`: + +```java +import com.cameleer3.server.app.storage.PostgresMetricsQueryStore; +import com.cameleer3.server.core.storage.MetricsQueryStore; + +// Add this bean method: +@Bean +public MetricsQueryStore metricsQueryStore(JdbcTemplate jdbc) { + return new PostgresMetricsQueryStore(jdbc); +} +``` + +This creates the PG implementation as the default. Task 7 will replace this with conditional beans for PG vs CH. + +- [ ] **Step 6: Verify compilation** + +Run: `mvn clean compile -pl cameleer3-server-app -am` +Expected: BUILD SUCCESS + +- [ ] **Step 7: Commit** + +```bash +git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java +git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/MetricTimeSeries.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java +git commit -m "refactor: extract MetricsQueryStore interface from AgentMetricsController" +``` + +--- + +### Task 6: ClickHouseMetricsQueryStore (TDD) + +**Files:** +- Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java` +- Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java` + +- [ ] **Step 1: Write the failing test** + +```java +package com.cameleer3.server.app.storage; + +import com.cameleer3.server.core.storage.model.MetricTimeSeries; +import com.zaxxer.hikari.HikariDataSource; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.jdbc.core.JdbcTemplate; +import org.testcontainers.clickhouse.ClickHouseContainer; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; + +import java.time.Instant; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +@Testcontainers +class ClickHouseMetricsQueryStoreTest { + + @Container + static final ClickHouseContainer clickhouse = + new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); + + private JdbcTemplate jdbc; + private ClickHouseMetricsQueryStore queryStore; + + @BeforeEach + void setUp() { + HikariDataSource ds = new HikariDataSource(); + ds.setJdbcUrl(clickhouse.getJdbcUrl()); + ds.setUsername(clickhouse.getUsername()); + ds.setPassword(clickhouse.getPassword()); + + jdbc = new JdbcTemplate(ds); + + jdbc.execute(""" + CREATE TABLE IF NOT EXISTS agent_metrics ( + tenant_id LowCardinality(String) DEFAULT 'default', + collected_at DateTime64(3), + agent_id LowCardinality(String), + metric_name LowCardinality(String), + metric_value Float64, + tags Map(String, String) DEFAULT map(), + server_received_at DateTime64(3) DEFAULT now64(3) + ) + ENGINE = MergeTree() + ORDER BY (tenant_id, agent_id, metric_name, collected_at) + """); + + jdbc.execute("TRUNCATE TABLE agent_metrics"); + + // Seed test data: 6 data points across 1 hour for two metrics + Instant base = Instant.parse("2026-03-31T10:00:00Z"); + for (int i = 0; i < 6; i++) { + Instant ts = base.plusSeconds(i * 600); // every 10 minutes + jdbc.update("INSERT INTO agent_metrics (agent_id, metric_name, metric_value, collected_at) VALUES (?, ?, ?, ?)", + "agent-1", "cpu.usage", 50.0 + i * 5, java.sql.Timestamp.from(ts)); + jdbc.update("INSERT INTO agent_metrics (agent_id, metric_name, metric_value, collected_at) VALUES (?, ?, ?, ?)", + "agent-1", "memory.free", 1000.0 - i * 100, java.sql.Timestamp.from(ts)); + } + + queryStore = new ClickHouseMetricsQueryStore(jdbc); + } + + @Test + void queryTimeSeries_returnsDataGroupedByMetric() { + Instant from = Instant.parse("2026-03-31T10:00:00Z"); + Instant to = Instant.parse("2026-03-31T11:00:00Z"); + + Map> result = + queryStore.queryTimeSeries("agent-1", List.of("cpu.usage", "memory.free"), from, to, 6); + + assertThat(result).containsKeys("cpu.usage", "memory.free"); + assertThat(result.get("cpu.usage")).isNotEmpty(); + assertThat(result.get("memory.free")).isNotEmpty(); + } + + @Test + void queryTimeSeries_bucketsAverageCorrectly() { + Instant from = Instant.parse("2026-03-31T10:00:00Z"); + Instant to = Instant.parse("2026-03-31T11:00:00Z"); + + // 1 bucket for the entire hour = average of all 6 values + Map> result = + queryStore.queryTimeSeries("agent-1", List.of("cpu.usage"), from, to, 1); + + assertThat(result.get("cpu.usage")).hasSize(1); + // Values: 50, 55, 60, 65, 70, 75 → avg = 62.5 + assertThat(result.get("cpu.usage").get(0).value()).isCloseTo(62.5, org.assertj.core.data.Offset.offset(0.1)); + } + + @Test + void queryTimeSeries_noData_returnsEmptyLists() { + Instant from = Instant.parse("2025-01-01T00:00:00Z"); + Instant to = Instant.parse("2025-01-01T01:00:00Z"); + + Map> result = + queryStore.queryTimeSeries("agent-1", List.of("cpu.usage"), from, to, 6); + + assertThat(result.get("cpu.usage")).isEmpty(); + } + + @Test + void queryTimeSeries_unknownAgent_returnsEmpty() { + Instant from = Instant.parse("2026-03-31T10:00:00Z"); + Instant to = Instant.parse("2026-03-31T11:00:00Z"); + + Map> result = + queryStore.queryTimeSeries("nonexistent", List.of("cpu.usage"), from, to, 6); + + assertThat(result.get("cpu.usage")).isEmpty(); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsQueryStoreTest -DfailIfNoTests=false` +Expected: FAIL — `ClickHouseMetricsQueryStore` class does not exist + +- [ ] **Step 3: Write ClickHouseMetricsQueryStore** + +```java +package com.cameleer3.server.app.storage; + +import com.cameleer3.server.core.storage.MetricsQueryStore; +import com.cameleer3.server.core.storage.model.MetricTimeSeries; +import org.springframework.jdbc.core.JdbcTemplate; + +import java.time.Instant; +import java.util.*; + +public class ClickHouseMetricsQueryStore implements MetricsQueryStore { + + private final JdbcTemplate jdbc; + + public ClickHouseMetricsQueryStore(JdbcTemplate jdbc) { + this.jdbc = jdbc; + } + + @Override + public Map> queryTimeSeries( + String agentId, List metricNames, + Instant from, Instant to, int buckets) { + + long intervalSeconds = Math.max(60, + (to.getEpochSecond() - from.getEpochSecond()) / Math.max(buckets, 1)); + + Map> result = new LinkedHashMap<>(); + for (String name : metricNames) { + result.put(name.trim(), new ArrayList<>()); + } + + String[] namesArray = metricNames.stream().map(String::trim).toArray(String[]::new); + + String sql = """ + SELECT toStartOfInterval(collected_at, INTERVAL ? SECOND) AS bucket, + metric_name, + avg(metric_value) AS avg_value + FROM agent_metrics + WHERE agent_id = ? + AND collected_at >= ? + AND collected_at < ? + AND metric_name IN (?) + GROUP BY bucket, metric_name + ORDER BY bucket + """; + + // ClickHouse JDBC doesn't support array params with IN (?). + // Build the IN clause manually with safe values (metric names are validated by controller). + StringBuilder inClause = new StringBuilder(); + for (int i = 0; i < namesArray.length; i++) { + if (i > 0) inClause.append(", "); + inClause.append("'").append(namesArray[i].replace("'", "\\'")).append("'"); + } + + String finalSql = """ + SELECT toStartOfInterval(collected_at, INTERVAL %d SECOND) AS bucket, + metric_name, + avg(metric_value) AS avg_value + FROM agent_metrics + WHERE agent_id = ? + AND collected_at >= ? + AND collected_at < ? + AND metric_name IN (%s) + GROUP BY bucket, metric_name + ORDER BY bucket + """.formatted(intervalSeconds, inClause); + + jdbc.query(finalSql, rs -> { + String metricName = rs.getString("metric_name"); + Instant bucket = rs.getTimestamp("bucket").toInstant(); + double value = rs.getDouble("avg_value"); + result.computeIfAbsent(metricName, k -> new ArrayList<>()) + .add(new MetricTimeSeries.Bucket(bucket, value)); + }, agentId, + java.sql.Timestamp.from(from), + java.sql.Timestamp.from(to)); + + return result; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsQueryStoreTest` +Expected: PASS — all 4 tests green + +- [ ] **Step 5: Commit** + +```bash +git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java +git commit -m "feat: add ClickHouseMetricsQueryStore with time-bucketed queries" +``` + +--- + +### Task 7: Wire Feature Flag in StorageBeanConfig + +**Files:** +- Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java` + +- [ ] **Step 1: Replace simple beans with conditional MetricsStore and MetricsQueryStore beans** + +In `StorageBeanConfig.java`, **remove** the unconditional `metricsQueryStore` bean added in Task 5. Replace with these conditional bean definitions: + +```java +import com.cameleer3.server.app.storage.ClickHouseMetricsStore; +import com.cameleer3.server.app.storage.ClickHouseMetricsQueryStore; +import com.cameleer3.server.app.storage.PostgresMetricsQueryStore; +import com.cameleer3.server.core.storage.MetricsQueryStore; + +// ... existing beans ... + +@Bean +@ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "clickhouse") +public MetricsStore clickHouseMetricsStore( + @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { + return new ClickHouseMetricsStore(clickHouseJdbc); +} + +@Bean +@ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "postgres", matchIfMissing = true) +public MetricsStore postgresMetricsStore(JdbcTemplate jdbc) { + return new PostgresMetricsStore(jdbc); +} + +@Bean +@ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "clickhouse") +public MetricsQueryStore clickHouseMetricsQueryStore( + @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { + return new ClickHouseMetricsQueryStore(clickHouseJdbc); +} + +@Bean +@ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "postgres", matchIfMissing = true) +public MetricsQueryStore postgresMetricsQueryStore(JdbcTemplate jdbc) { + return new PostgresMetricsQueryStore(jdbc); +} +``` + +Also remove the `@Repository` annotation from `PostgresMetricsStore.java` since the bean is now created explicitly in config. If `PostgresMetricsStore` has `@Repository`, remove it. Check the class — it does have `@Repository`, so remove it. + +Add the necessary imports to `StorageBeanConfig.java`: +```java +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.beans.factory.annotation.Qualifier; +import com.cameleer3.server.core.storage.MetricsStore; +``` + +- [ ] **Step 2: Remove @Repository from PostgresMetricsStore** + +In `PostgresMetricsStore.java`, remove the `@Repository` annotation from the class declaration. The bean is now created in `StorageBeanConfig`. + +- [ ] **Step 3: Verify compilation** + +Run: `mvn clean compile -pl cameleer3-server-app -am` +Expected: BUILD SUCCESS + +- [ ] **Step 4: Run existing tests to verify no regression** + +Run: `mvn test -pl cameleer3-server-app` +Expected: All existing tests pass. The default config (`cameleer.storage.metrics=postgres`) means the PG beans are created, ClickHouse beans are not (ClickHouse DataSource not needed). + +- [ ] **Step 5: Commit** + +```bash +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java +git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsStore.java +git commit -m "feat: wire MetricsStore and MetricsQueryStore with feature flag" +``` + +--- + +### Task 8: ClickHouse Kubernetes Deployment + +**Files:** +- Create: `deploy/clickhouse.yaml` +- Modify: `deploy/base/server.yaml` + +- [ ] **Step 1: Create ClickHouse StatefulSet + Service manifest** + +File: `deploy/clickhouse.yaml` + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: clickhouse + namespace: cameleer +spec: + serviceName: clickhouse + replicas: 1 + selector: + matchLabels: + app: clickhouse + template: + metadata: + labels: + app: clickhouse + spec: + containers: + - name: clickhouse + image: clickhouse/clickhouse-server:24.12 + ports: + - containerPort: 8123 + name: http + - containerPort: 9000 + name: native + volumeMounts: + - name: data + mountPath: /var/lib/clickhouse + resources: + requests: + memory: "2Gi" + cpu: "500m" + limits: + memory: "4Gi" + cpu: "2000m" + livenessProbe: + httpGet: + path: /ping + port: 8123 + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 3 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /ping + port: 8123 + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 50Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: clickhouse + namespace: cameleer +spec: + clusterIP: None + selector: + app: clickhouse + ports: + - port: 8123 + targetPort: 8123 + name: http + - port: 9000 + targetPort: 9000 + name: native +``` + +- [ ] **Step 2: Add ClickHouse env vars to server.yaml** + +Add these env vars to the `cameleer3-server` container in `deploy/base/server.yaml`, after the existing env vars (before `resources:`): + +```yaml + - name: CLICKHOUSE_ENABLED + value: "true" + - name: CLICKHOUSE_URL + value: "jdbc:clickhouse://clickhouse.cameleer.svc.cluster.local:8123/cameleer?async_insert=1&wait_for_async_insert=0" + - name: CAMELEER_STORAGE_METRICS + value: "postgres" +``` + +Note: `CAMELEER_STORAGE_METRICS` defaults to `postgres`. Set to `clickhouse` when ready to switch. + +- [ ] **Step 3: Commit** + +```bash +git add deploy/clickhouse.yaml +git add deploy/base/server.yaml +git commit -m "deploy: add ClickHouse StatefulSet and server env vars" +``` + +--- + +### Task 9: Integration Smoke Test + +**Files:** +- Modify: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractPostgresIT.java` (add ClickHouse container) + +- [ ] **Step 1: Update AbstractPostgresIT to optionally start ClickHouse** + +This is a non-breaking addition. The existing integration tests continue to use PG. A future ClickHouse-specific IT can extend this base. + +Add the ClickHouse container to `AbstractPostgresIT`: + +```java +import org.testcontainers.clickhouse.ClickHouseContainer; + +// Add after opensearch static field: +static final ClickHouseContainer clickhouse; + +// In the static initializer, add after opensearch.start(): +clickhouse = new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); +clickhouse.start(); + +// In configureProperties, add: +registry.add("clickhouse.enabled", () -> "true"); +registry.add("clickhouse.url", clickhouse::getJdbcUrl); +registry.add("clickhouse.username", clickhouse::getUsername); +registry.add("clickhouse.password", clickhouse::getPassword); +``` + +- [ ] **Step 2: Run all integration tests** + +Run: `mvn verify -pl cameleer3-server-app` +Expected: All ITs pass. ClickHouse schema initializer runs and creates the `agent_metrics` table. The `MetricsControllerIT` still uses PG (default storage flag is `postgres`). + +- [ ] **Step 3: Commit** + +```bash +git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractPostgresIT.java +git commit -m "test: add ClickHouse testcontainer to integration test base" +``` + +--- + +### Task 10: Final Verification + +- [ ] **Step 1: Run full build** + +Run: `mvn clean verify` +Expected: BUILD SUCCESS (unit tests + integration tests all pass) + +- [ ] **Step 2: Verify ClickHouse switch works locally (manual)** + +Start ClickHouse locally: +```bash +docker run -d --name clickhouse-test -p 8123:8123 clickhouse/clickhouse-server:24.12 +``` + +Create the database: +```bash +curl "http://localhost:8123/" --data "CREATE DATABASE IF NOT EXISTS cameleer" +``` + +Start the server with ClickHouse enabled: +```bash +CLICKHOUSE_ENABLED=true CAMELEER_STORAGE_METRICS=clickhouse java -jar cameleer3-server-app/target/cameleer3-server-app-1.0-SNAPSHOT.jar +``` + +Verify schema initialization in the logs: `Executing ClickHouse schema script: V1__agent_metrics.sql` + +POST a metric: +```bash +curl -X POST http://localhost:8081/api/v1/data/metrics \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '[{"agentId":"test","collectedAt":"2026-03-31T10:00:00Z","metricName":"cpu","metricValue":75.0,"tags":{}}]' +``` + +Verify data in ClickHouse: +```bash +curl "http://localhost:8123/" --data "SELECT * FROM cameleer.agent_metrics FORMAT Pretty" +``` + +Stop and clean up: +```bash +docker stop clickhouse-test && docker rm clickhouse-test +``` + +- [ ] **Step 3: Final commit (if any fixes were needed)** + +```bash +git add -A +git commit -m "fix: address issues found during manual verification" +```