# ClickHouse Migration Phase 1: Foundation > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Add ClickHouse as a second data store alongside PostgreSQL, validated with the metrics pipeline end-to-end (write + read). Feature-flagged so either PG or CH serves metrics. **Architecture:** Add `clickhouse-jdbc` dependency, a second `DataSource`/`JdbcTemplate` bean qualified as `@ClickHouse`, an idempotent schema initializer that creates ClickHouse tables on startup, and a feature flag (`cameleer.storage.metrics=clickhouse|postgres`) that selects the active `MetricsStore` implementation. The metrics read path (`AgentMetricsController`) is refactored behind a new `MetricsQueryStore` interface so it too can be swapped. **Tech Stack:** ClickHouse JDBC 0.7.x, Spring Boot 3.4.3, JdbcTemplate, Testcontainers (clickhouse module) **Design Spec:** `docs/superpowers/specs/2026-03-31-clickhouse-migration-design.md` --- ## File Map | Action | File | Responsibility | |--------|------|----------------| | Modify | `cameleer3-server-app/pom.xml` | Add clickhouse-jdbc + testcontainers-clickhouse dependencies | | Modify | `cameleer3-server-app/src/main/resources/application.yml` | Add `clickhouse.*` and `cameleer.storage.metrics` config keys | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java` | ClickHouse DataSource + JdbcTemplate beans | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java` | Config properties class for `clickhouse.*` prefix | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java` | Runs idempotent DDL on startup | | Create | `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` | DDL for `agent_metrics` table | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java` | `MetricsStore` impl writing to ClickHouse | | Create | `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java` | Interface: query metrics time-series | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java` | PG impl of MetricsQueryStore (extracted from controller) | | Create | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java` | CH impl of MetricsQueryStore | | Modify | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java` | Use MetricsQueryStore instead of raw JdbcTemplate | | Modify | `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java` | Conditional beans for PG vs CH metrics stores | | Create | `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java` | Unit test | | Create | `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java` | Unit test | | Create | `deploy/clickhouse.yaml` | K8s StatefulSet + Service for ClickHouse | | Modify | `deploy/base/server.yaml` | Add CLICKHOUSE_URL env var | --- ### Task 1: Add ClickHouse Dependencies **Files:** - Modify: `cameleer3-server-app/pom.xml` - [ ] **Step 1: Add clickhouse-jdbc and testcontainers-clickhouse to app POM** In `cameleer3-server-app/pom.xml`, add these two dependencies. Place the runtime dependency after the opensearch dependencies (around line 59), and the test dependency after the opensearch-testcontainers dependency (around line 128): ```xml com.clickhouse clickhouse-jdbc 0.7.1-patch5 all org.testcontainers clickhouse test ``` Note: The `all` classifier bundles the HTTP client, avoiding transitive dependency conflicts. The testcontainers `clickhouse` module version is managed by the `testcontainers.version` property in the parent POM. - [ ] **Step 2: Verify compilation** Run: `mvn clean compile -pl cameleer3-server-app -am` Expected: BUILD SUCCESS - [ ] **Step 3: Commit** ```bash git add cameleer3-server-app/pom.xml git commit -m "build: add clickhouse-jdbc and testcontainers-clickhouse dependencies" ``` --- ### Task 2: ClickHouse Configuration **Files:** - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java` - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java` - Modify: `cameleer3-server-app/src/main/resources/application.yml` - [ ] **Step 1: Create ClickHouseProperties** ```java package com.cameleer3.server.app.config; import org.springframework.boot.context.properties.ConfigurationProperties; @ConfigurationProperties(prefix = "clickhouse") public class ClickHouseProperties { private String url = "jdbc:clickhouse://localhost:8123/cameleer"; private String username = "default"; private String password = ""; public String getUrl() { return url; } public void setUrl(String url) { this.url = url; } public String getUsername() { return username; } public void setUsername(String username) { this.username = username; } public String getPassword() { return password; } public void setPassword(String password) { this.password = password; } } ``` - [ ] **Step 2: Create ClickHouseConfig** ```java package com.cameleer3.server.app.config; import com.zaxxer.hikari.HikariDataSource; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.boot.context.properties.EnableConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.jdbc.core.JdbcTemplate; import javax.sql.DataSource; @Configuration @EnableConfigurationProperties(ClickHouseProperties.class) @ConditionalOnProperty(name = "clickhouse.enabled", havingValue = "true") public class ClickHouseConfig { @Bean(name = "clickHouseDataSource") public DataSource clickHouseDataSource(ClickHouseProperties props) { HikariDataSource ds = new HikariDataSource(); ds.setJdbcUrl(props.getUrl()); ds.setUsername(props.getUsername()); ds.setPassword(props.getPassword()); ds.setMaximumPoolSize(10); ds.setPoolName("clickhouse-pool"); return ds; } @Bean(name = "clickHouseJdbcTemplate") public JdbcTemplate clickHouseJdbcTemplate( @Qualifier("clickHouseDataSource") DataSource ds) { return new JdbcTemplate(ds); } } ``` - [ ] **Step 3: Add configuration to application.yml** Add at the end of `application.yml`: ```yaml clickhouse: enabled: ${CLICKHOUSE_ENABLED:false} url: ${CLICKHOUSE_URL:jdbc:clickhouse://localhost:8123/cameleer?async_insert=1&wait_for_async_insert=0} username: ${CLICKHOUSE_USERNAME:default} password: ${CLICKHOUSE_PASSWORD:} cameleer: storage: metrics: ${CAMELEER_STORAGE_METRICS:postgres} ``` Note: The existing `cameleer:` block already has `body-size-limit` and `retention-days`. Add the `storage.metrics` key under it. - [ ] **Step 4: Verify compilation** Run: `mvn clean compile -pl cameleer3-server-app -am` Expected: BUILD SUCCESS - [ ] **Step 5: Commit** ```bash git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseProperties.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseConfig.java git add cameleer3-server-app/src/main/resources/application.yml git commit -m "feat: add ClickHouse DataSource and JdbcTemplate configuration" ``` --- ### Task 3: Schema Initializer + DDL Script **Files:** - Create: `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java` - [ ] **Step 1: Create DDL script** File: `cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql` ```sql CREATE TABLE IF NOT EXISTS agent_metrics ( tenant_id LowCardinality(String) DEFAULT 'default', collected_at DateTime64(3), agent_id LowCardinality(String), metric_name LowCardinality(String), metric_value Float64, tags Map(String, String) DEFAULT map(), server_received_at DateTime64(3) DEFAULT now64(3) ) ENGINE = MergeTree() PARTITION BY (tenant_id, toYYYYMM(collected_at)) ORDER BY (tenant_id, agent_id, metric_name, collected_at) TTL collected_at + INTERVAL 365 DAY DELETE SETTINGS index_granularity = 8192; ``` - [ ] **Step 2: Create ClickHouseSchemaInitializer** ```java package com.cameleer3.server.app.config; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.core.io.Resource; import org.springframework.core.io.support.PathMatchingResourcePatternResolver; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Component; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.Arrays; import java.util.Comparator; @Component @ConditionalOnProperty(name = "clickhouse.enabled", havingValue = "true") public class ClickHouseSchemaInitializer { private static final Logger log = LoggerFactory.getLogger(ClickHouseSchemaInitializer.class); private final JdbcTemplate clickHouseJdbc; public ClickHouseSchemaInitializer( @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { this.clickHouseJdbc = clickHouseJdbc; } @EventListener(ApplicationReadyEvent.class) public void initializeSchema() throws IOException { PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(); Resource[] scripts = resolver.getResources("classpath:clickhouse/*.sql"); Arrays.sort(scripts, Comparator.comparing(Resource::getFilename)); for (Resource script : scripts) { String sql = script.getContentAsString(StandardCharsets.UTF_8); log.info("Executing ClickHouse schema script: {}", script.getFilename()); for (String statement : sql.split(";")) { String trimmed = statement.trim(); if (!trimmed.isEmpty()) { clickHouseJdbc.execute(trimmed); } } } log.info("ClickHouse schema initialization complete ({} scripts)", scripts.length); } } ``` - [ ] **Step 3: Verify compilation** Run: `mvn clean compile -pl cameleer3-server-app -am` Expected: BUILD SUCCESS - [ ] **Step 4: Commit** ```bash git add cameleer3-server-app/src/main/resources/clickhouse/V1__agent_metrics.sql git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/ClickHouseSchemaInitializer.java git commit -m "feat: add ClickHouse schema initializer with agent_metrics DDL" ``` --- ### Task 4: ClickHouseMetricsStore (TDD) **Files:** - Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java` - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java` - [ ] **Step 1: Write the failing test** ```java package com.cameleer3.server.app.storage; import com.cameleer3.server.core.storage.model.MetricsSnapshot; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.springframework.jdbc.core.JdbcTemplate; import org.testcontainers.clickhouse.ClickHouseContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import com.zaxxer.hikari.HikariDataSource; import java.time.Instant; import java.util.List; import java.util.Map; import static org.assertj.core.api.Assertions.assertThat; @Testcontainers class ClickHouseMetricsStoreTest { @Container static final ClickHouseContainer clickhouse = new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); private JdbcTemplate jdbc; private ClickHouseMetricsStore store; @BeforeEach void setUp() { HikariDataSource ds = new HikariDataSource(); ds.setJdbcUrl(clickhouse.getJdbcUrl()); ds.setUsername(clickhouse.getUsername()); ds.setPassword(clickhouse.getPassword()); jdbc = new JdbcTemplate(ds); jdbc.execute(""" CREATE TABLE IF NOT EXISTS agent_metrics ( tenant_id LowCardinality(String) DEFAULT 'default', collected_at DateTime64(3), agent_id LowCardinality(String), metric_name LowCardinality(String), metric_value Float64, tags Map(String, String) DEFAULT map(), server_received_at DateTime64(3) DEFAULT now64(3) ) ENGINE = MergeTree() ORDER BY (tenant_id, agent_id, metric_name, collected_at) """); jdbc.execute("TRUNCATE TABLE agent_metrics"); store = new ClickHouseMetricsStore(jdbc); } @Test void insertBatch_writesMetricsToClickHouse() { List batch = List.of( new MetricsSnapshot("agent-1", Instant.parse("2026-03-31T10:00:00Z"), "cpu.usage", 75.5, Map.of("host", "server-1")), new MetricsSnapshot("agent-1", Instant.parse("2026-03-31T10:00:01Z"), "memory.free", 1024.0, null) ); store.insertBatch(batch); Integer count = jdbc.queryForObject( "SELECT count() FROM agent_metrics WHERE agent_id = 'agent-1'", Integer.class); assertThat(count).isEqualTo(2); } @Test void insertBatch_storesTags() { store.insertBatch(List.of( new MetricsSnapshot("agent-2", Instant.parse("2026-03-31T10:00:00Z"), "disk.used", 500.0, Map.of("mount", "/data", "fs", "ext4")) )); Map tags = jdbc.queryForObject( "SELECT tags FROM agent_metrics WHERE agent_id = 'agent-2'", (rs, n) -> { // ClickHouse returns Map as a map-like structure via JDBC @SuppressWarnings("unchecked") Map m = (Map) rs.getObject("tags"); return m; }); assertThat(tags).containsEntry("mount", "/data").containsEntry("fs", "ext4"); } @Test void insertBatch_emptyList_doesNothing() { store.insertBatch(List.of()); Integer count = jdbc.queryForObject("SELECT count() FROM agent_metrics", Integer.class); assertThat(count).isEqualTo(0); } @Test void insertBatch_nullTags_defaultsToEmptyMap() { store.insertBatch(List.of( new MetricsSnapshot("agent-3", Instant.parse("2026-03-31T10:00:00Z"), "cpu.usage", 50.0, null) )); Integer count = jdbc.queryForObject( "SELECT count() FROM agent_metrics WHERE agent_id = 'agent-3'", Integer.class); assertThat(count).isEqualTo(1); } } ``` - [ ] **Step 2: Run test to verify it fails** Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsStoreTest -DfailIfNoTests=false` Expected: FAIL — `ClickHouseMetricsStore` class does not exist - [ ] **Step 3: Write ClickHouseMetricsStore** ```java package com.cameleer3.server.app.storage; import com.cameleer3.server.core.storage.MetricsStore; import com.cameleer3.server.core.storage.model.MetricsSnapshot; import org.springframework.jdbc.core.JdbcTemplate; import java.sql.Timestamp; import java.util.HashMap; import java.util.List; import java.util.Map; public class ClickHouseMetricsStore implements MetricsStore { private final JdbcTemplate jdbc; public ClickHouseMetricsStore(JdbcTemplate jdbc) { this.jdbc = jdbc; } @Override public void insertBatch(List snapshots) { if (snapshots.isEmpty()) return; jdbc.batchUpdate(""" INSERT INTO agent_metrics (agent_id, metric_name, metric_value, tags, collected_at) VALUES (?, ?, ?, ?, ?) """, snapshots.stream().map(s -> new Object[]{ s.agentId(), s.metricName(), s.metricValue(), tagsToClickHouseMap(s.tags()), Timestamp.from(s.collectedAt()) }).toList()); } private Map tagsToClickHouseMap(Map tags) { if (tags == null || tags.isEmpty()) return new HashMap<>(); return tags; } } ``` Note: ClickHouse JDBC driver handles `Map` natively for `Map(String, String)` columns. No JSON serialization needed (unlike the PG implementation that converts to JSONB string). - [ ] **Step 4: Run test to verify it passes** Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsStoreTest` Expected: PASS — all 4 tests green If the tags test fails due to JDBC Map handling, adjust `tagsToClickHouseMap` to return a `java.util.HashMap` (ClickHouse JDBC requires a mutable map). If needed, wrap: `return new HashMap<>(tags)`. - [ ] **Step 5: Commit** ```bash git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsStoreTest.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsStore.java git commit -m "feat: add ClickHouseMetricsStore with batch insert" ``` --- ### Task 5: MetricsQueryStore Interface + PostgreSQL Implementation (TDD) Extract the query logic from `AgentMetricsController` into an interface. **Files:** - Create: `cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java` - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java` - Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java` - [ ] **Step 1: Create MetricsQueryStore interface** ```java package com.cameleer3.server.core.storage; import com.cameleer3.server.core.storage.model.MetricTimeSeries; import java.time.Instant; import java.util.List; import java.util.Map; public interface MetricsQueryStore { /** * Query time-bucketed metrics for an agent. * * @param agentId the agent identifier * @param metricNames list of metric names to query * @param from start of time range (inclusive) * @param to end of time range (exclusive) * @param buckets number of time buckets to divide the range into * @return map of metric name to list of (time, value) pairs */ Map> queryTimeSeries( String agentId, List metricNames, Instant from, Instant to, int buckets); } ``` - [ ] **Step 2: Create MetricTimeSeries.Bucket record** ```java package com.cameleer3.server.core.storage.model; import java.time.Instant; import java.util.List; public record MetricTimeSeries(String metricName, List buckets) { public record Bucket(Instant time, double value) {} } ``` - [ ] **Step 3: Create PostgresMetricsQueryStore (extract from controller)** ```java package com.cameleer3.server.app.storage; import com.cameleer3.server.core.storage.MetricsQueryStore; import com.cameleer3.server.core.storage.model.MetricTimeSeries; import org.springframework.jdbc.core.JdbcTemplate; import java.sql.Timestamp; import java.time.Instant; import java.util.*; public class PostgresMetricsQueryStore implements MetricsQueryStore { private final JdbcTemplate jdbc; public PostgresMetricsQueryStore(JdbcTemplate jdbc) { this.jdbc = jdbc; } @Override public Map> queryTimeSeries( String agentId, List metricNames, Instant from, Instant to, int buckets) { long intervalMs = (to.toEpochMilli() - from.toEpochMilli()) / Math.max(buckets, 1); String intervalStr = intervalMs + " milliseconds"; Map> result = new LinkedHashMap<>(); for (String name : metricNames) { result.put(name.trim(), new ArrayList<>()); } String sql = """ SELECT time_bucket(CAST(? AS interval), collected_at) AS bucket, metric_name, AVG(metric_value) AS avg_value FROM agent_metrics WHERE agent_id = ? AND collected_at >= ? AND collected_at < ? AND metric_name = ANY(?) GROUP BY bucket, metric_name ORDER BY bucket """; String[] namesArray = metricNames.stream().map(String::trim).toArray(String[]::new); jdbc.query(sql, rs -> { String metricName = rs.getString("metric_name"); Instant bucket = rs.getTimestamp("bucket").toInstant(); double value = rs.getDouble("avg_value"); result.computeIfAbsent(metricName, k -> new ArrayList<>()) .add(new MetricTimeSeries.Bucket(bucket, value)); }, intervalStr, agentId, Timestamp.from(from), Timestamp.from(to), namesArray); return result; } } ``` - [ ] **Step 4: Refactor AgentMetricsController to use MetricsQueryStore** Replace the entire `AgentMetricsController.java` content: ```java package com.cameleer3.server.app.controller; import com.cameleer3.server.app.dto.AgentMetricsResponse; import com.cameleer3.server.app.dto.MetricBucket; import com.cameleer3.server.core.storage.MetricsQueryStore; import com.cameleer3.server.core.storage.model.MetricTimeSeries; import org.springframework.web.bind.annotation.*; import java.time.Instant; import java.time.temporal.ChronoUnit; import java.util.*; import java.util.stream.Collectors; @RestController @RequestMapping("/api/v1/agents/{agentId}/metrics") public class AgentMetricsController { private final MetricsQueryStore metricsQueryStore; public AgentMetricsController(MetricsQueryStore metricsQueryStore) { this.metricsQueryStore = metricsQueryStore; } @GetMapping public AgentMetricsResponse getMetrics( @PathVariable String agentId, @RequestParam String names, @RequestParam(required = false) Instant from, @RequestParam(required = false) Instant to, @RequestParam(defaultValue = "60") int buckets) { if (from == null) from = Instant.now().minus(1, ChronoUnit.HOURS); if (to == null) to = Instant.now(); List metricNames = Arrays.asList(names.split(",")); Map> raw = metricsQueryStore.queryTimeSeries(agentId, metricNames, from, to, buckets); // Convert to existing DTO format Map> result = raw.entrySet().stream() .collect(Collectors.toMap( Map.Entry::getKey, e -> e.getValue().stream() .map(b -> new MetricBucket(b.time(), b.value())) .toList(), (a, b) -> a, LinkedHashMap::new)); return new AgentMetricsResponse(result); } } ``` - [ ] **Step 5: Add PostgresMetricsQueryStore bean to StorageBeanConfig** The refactored `AgentMetricsController` needs a `MetricsQueryStore` bean. Add this to `StorageBeanConfig.java`: ```java import com.cameleer3.server.app.storage.PostgresMetricsQueryStore; import com.cameleer3.server.core.storage.MetricsQueryStore; // Add this bean method: @Bean public MetricsQueryStore metricsQueryStore(JdbcTemplate jdbc) { return new PostgresMetricsQueryStore(jdbc); } ``` This creates the PG implementation as the default. Task 7 will replace this with conditional beans for PG vs CH. - [ ] **Step 6: Verify compilation** Run: `mvn clean compile -pl cameleer3-server-app -am` Expected: BUILD SUCCESS - [ ] **Step 7: Commit** ```bash git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/MetricsQueryStore.java git add cameleer3-server-core/src/main/java/com/cameleer3/server/core/storage/model/MetricTimeSeries.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsQueryStore.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/controller/AgentMetricsController.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java git commit -m "refactor: extract MetricsQueryStore interface from AgentMetricsController" ``` --- ### Task 6: ClickHouseMetricsQueryStore (TDD) **Files:** - Create: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java` - Create: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java` - [ ] **Step 1: Write the failing test** ```java package com.cameleer3.server.app.storage; import com.cameleer3.server.core.storage.model.MetricTimeSeries; import com.zaxxer.hikari.HikariDataSource; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.springframework.jdbc.core.JdbcTemplate; import org.testcontainers.clickhouse.ClickHouseContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import java.time.Instant; import java.util.List; import java.util.Map; import static org.assertj.core.api.Assertions.assertThat; @Testcontainers class ClickHouseMetricsQueryStoreTest { @Container static final ClickHouseContainer clickhouse = new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); private JdbcTemplate jdbc; private ClickHouseMetricsQueryStore queryStore; @BeforeEach void setUp() { HikariDataSource ds = new HikariDataSource(); ds.setJdbcUrl(clickhouse.getJdbcUrl()); ds.setUsername(clickhouse.getUsername()); ds.setPassword(clickhouse.getPassword()); jdbc = new JdbcTemplate(ds); jdbc.execute(""" CREATE TABLE IF NOT EXISTS agent_metrics ( tenant_id LowCardinality(String) DEFAULT 'default', collected_at DateTime64(3), agent_id LowCardinality(String), metric_name LowCardinality(String), metric_value Float64, tags Map(String, String) DEFAULT map(), server_received_at DateTime64(3) DEFAULT now64(3) ) ENGINE = MergeTree() ORDER BY (tenant_id, agent_id, metric_name, collected_at) """); jdbc.execute("TRUNCATE TABLE agent_metrics"); // Seed test data: 6 data points across 1 hour for two metrics Instant base = Instant.parse("2026-03-31T10:00:00Z"); for (int i = 0; i < 6; i++) { Instant ts = base.plusSeconds(i * 600); // every 10 minutes jdbc.update("INSERT INTO agent_metrics (agent_id, metric_name, metric_value, collected_at) VALUES (?, ?, ?, ?)", "agent-1", "cpu.usage", 50.0 + i * 5, java.sql.Timestamp.from(ts)); jdbc.update("INSERT INTO agent_metrics (agent_id, metric_name, metric_value, collected_at) VALUES (?, ?, ?, ?)", "agent-1", "memory.free", 1000.0 - i * 100, java.sql.Timestamp.from(ts)); } queryStore = new ClickHouseMetricsQueryStore(jdbc); } @Test void queryTimeSeries_returnsDataGroupedByMetric() { Instant from = Instant.parse("2026-03-31T10:00:00Z"); Instant to = Instant.parse("2026-03-31T11:00:00Z"); Map> result = queryStore.queryTimeSeries("agent-1", List.of("cpu.usage", "memory.free"), from, to, 6); assertThat(result).containsKeys("cpu.usage", "memory.free"); assertThat(result.get("cpu.usage")).isNotEmpty(); assertThat(result.get("memory.free")).isNotEmpty(); } @Test void queryTimeSeries_bucketsAverageCorrectly() { Instant from = Instant.parse("2026-03-31T10:00:00Z"); Instant to = Instant.parse("2026-03-31T11:00:00Z"); // 1 bucket for the entire hour = average of all 6 values Map> result = queryStore.queryTimeSeries("agent-1", List.of("cpu.usage"), from, to, 1); assertThat(result.get("cpu.usage")).hasSize(1); // Values: 50, 55, 60, 65, 70, 75 → avg = 62.5 assertThat(result.get("cpu.usage").get(0).value()).isCloseTo(62.5, org.assertj.core.data.Offset.offset(0.1)); } @Test void queryTimeSeries_noData_returnsEmptyLists() { Instant from = Instant.parse("2025-01-01T00:00:00Z"); Instant to = Instant.parse("2025-01-01T01:00:00Z"); Map> result = queryStore.queryTimeSeries("agent-1", List.of("cpu.usage"), from, to, 6); assertThat(result.get("cpu.usage")).isEmpty(); } @Test void queryTimeSeries_unknownAgent_returnsEmpty() { Instant from = Instant.parse("2026-03-31T10:00:00Z"); Instant to = Instant.parse("2026-03-31T11:00:00Z"); Map> result = queryStore.queryTimeSeries("nonexistent", List.of("cpu.usage"), from, to, 6); assertThat(result.get("cpu.usage")).isEmpty(); } } ``` - [ ] **Step 2: Run test to verify it fails** Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsQueryStoreTest -DfailIfNoTests=false` Expected: FAIL — `ClickHouseMetricsQueryStore` class does not exist - [ ] **Step 3: Write ClickHouseMetricsQueryStore** ```java package com.cameleer3.server.app.storage; import com.cameleer3.server.core.storage.MetricsQueryStore; import com.cameleer3.server.core.storage.model.MetricTimeSeries; import org.springframework.jdbc.core.JdbcTemplate; import java.time.Instant; import java.util.*; public class ClickHouseMetricsQueryStore implements MetricsQueryStore { private final JdbcTemplate jdbc; public ClickHouseMetricsQueryStore(JdbcTemplate jdbc) { this.jdbc = jdbc; } @Override public Map> queryTimeSeries( String agentId, List metricNames, Instant from, Instant to, int buckets) { long intervalSeconds = Math.max(60, (to.getEpochSecond() - from.getEpochSecond()) / Math.max(buckets, 1)); Map> result = new LinkedHashMap<>(); for (String name : metricNames) { result.put(name.trim(), new ArrayList<>()); } String[] namesArray = metricNames.stream().map(String::trim).toArray(String[]::new); String sql = """ SELECT toStartOfInterval(collected_at, INTERVAL ? SECOND) AS bucket, metric_name, avg(metric_value) AS avg_value FROM agent_metrics WHERE agent_id = ? AND collected_at >= ? AND collected_at < ? AND metric_name IN (?) GROUP BY bucket, metric_name ORDER BY bucket """; // ClickHouse JDBC doesn't support array params with IN (?). // Build the IN clause manually with safe values (metric names are validated by controller). StringBuilder inClause = new StringBuilder(); for (int i = 0; i < namesArray.length; i++) { if (i > 0) inClause.append(", "); inClause.append("'").append(namesArray[i].replace("'", "\\'")).append("'"); } String finalSql = """ SELECT toStartOfInterval(collected_at, INTERVAL %d SECOND) AS bucket, metric_name, avg(metric_value) AS avg_value FROM agent_metrics WHERE agent_id = ? AND collected_at >= ? AND collected_at < ? AND metric_name IN (%s) GROUP BY bucket, metric_name ORDER BY bucket """.formatted(intervalSeconds, inClause); jdbc.query(finalSql, rs -> { String metricName = rs.getString("metric_name"); Instant bucket = rs.getTimestamp("bucket").toInstant(); double value = rs.getDouble("avg_value"); result.computeIfAbsent(metricName, k -> new ArrayList<>()) .add(new MetricTimeSeries.Bucket(bucket, value)); }, agentId, java.sql.Timestamp.from(from), java.sql.Timestamp.from(to)); return result; } } ``` - [ ] **Step 4: Run test to verify it passes** Run: `mvn test -pl cameleer3-server-app -Dtest=ClickHouseMetricsQueryStoreTest` Expected: PASS — all 4 tests green - [ ] **Step 5: Commit** ```bash git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStoreTest.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/ClickHouseMetricsQueryStore.java git commit -m "feat: add ClickHouseMetricsQueryStore with time-bucketed queries" ``` --- ### Task 7: Wire Feature Flag in StorageBeanConfig **Files:** - Modify: `cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java` - [ ] **Step 1: Replace simple beans with conditional MetricsStore and MetricsQueryStore beans** In `StorageBeanConfig.java`, **remove** the unconditional `metricsQueryStore` bean added in Task 5. Replace with these conditional bean definitions: ```java import com.cameleer3.server.app.storage.ClickHouseMetricsStore; import com.cameleer3.server.app.storage.ClickHouseMetricsQueryStore; import com.cameleer3.server.app.storage.PostgresMetricsQueryStore; import com.cameleer3.server.core.storage.MetricsQueryStore; // ... existing beans ... @Bean @ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "clickhouse") public MetricsStore clickHouseMetricsStore( @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { return new ClickHouseMetricsStore(clickHouseJdbc); } @Bean @ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "postgres", matchIfMissing = true) public MetricsStore postgresMetricsStore(JdbcTemplate jdbc) { return new PostgresMetricsStore(jdbc); } @Bean @ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "clickhouse") public MetricsQueryStore clickHouseMetricsQueryStore( @Qualifier("clickHouseJdbcTemplate") JdbcTemplate clickHouseJdbc) { return new ClickHouseMetricsQueryStore(clickHouseJdbc); } @Bean @ConditionalOnProperty(name = "cameleer.storage.metrics", havingValue = "postgres", matchIfMissing = true) public MetricsQueryStore postgresMetricsQueryStore(JdbcTemplate jdbc) { return new PostgresMetricsQueryStore(jdbc); } ``` Also remove the `@Repository` annotation from `PostgresMetricsStore.java` since the bean is now created explicitly in config. If `PostgresMetricsStore` has `@Repository`, remove it. Check the class — it does have `@Repository`, so remove it. Add the necessary imports to `StorageBeanConfig.java`: ```java import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.beans.factory.annotation.Qualifier; import com.cameleer3.server.core.storage.MetricsStore; ``` - [ ] **Step 2: Remove @Repository from PostgresMetricsStore** In `PostgresMetricsStore.java`, remove the `@Repository` annotation from the class declaration. The bean is now created in `StorageBeanConfig`. - [ ] **Step 3: Verify compilation** Run: `mvn clean compile -pl cameleer3-server-app -am` Expected: BUILD SUCCESS - [ ] **Step 4: Run existing tests to verify no regression** Run: `mvn test -pl cameleer3-server-app` Expected: All existing tests pass. The default config (`cameleer.storage.metrics=postgres`) means the PG beans are created, ClickHouse beans are not (ClickHouse DataSource not needed). - [ ] **Step 5: Commit** ```bash git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/config/StorageBeanConfig.java git add cameleer3-server-app/src/main/java/com/cameleer3/server/app/storage/PostgresMetricsStore.java git commit -m "feat: wire MetricsStore and MetricsQueryStore with feature flag" ``` --- ### Task 8: ClickHouse Kubernetes Deployment **Files:** - Create: `deploy/clickhouse.yaml` - Modify: `deploy/base/server.yaml` - [ ] **Step 1: Create ClickHouse StatefulSet + Service manifest** File: `deploy/clickhouse.yaml` ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: clickhouse namespace: cameleer spec: serviceName: clickhouse replicas: 1 selector: matchLabels: app: clickhouse template: metadata: labels: app: clickhouse spec: containers: - name: clickhouse image: clickhouse/clickhouse-server:24.12 ports: - containerPort: 8123 name: http - containerPort: 9000 name: native volumeMounts: - name: data mountPath: /var/lib/clickhouse resources: requests: memory: "2Gi" cpu: "500m" limits: memory: "4Gi" cpu: "2000m" livenessProbe: httpGet: path: /ping port: 8123 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 3 failureThreshold: 3 readinessProbe: httpGet: path: /ping port: 8123 initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 3 volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi --- apiVersion: v1 kind: Service metadata: name: clickhouse namespace: cameleer spec: clusterIP: None selector: app: clickhouse ports: - port: 8123 targetPort: 8123 name: http - port: 9000 targetPort: 9000 name: native ``` - [ ] **Step 2: Add ClickHouse env vars to server.yaml** Add these env vars to the `cameleer3-server` container in `deploy/base/server.yaml`, after the existing env vars (before `resources:`): ```yaml - name: CLICKHOUSE_ENABLED value: "true" - name: CLICKHOUSE_URL value: "jdbc:clickhouse://clickhouse.cameleer.svc.cluster.local:8123/cameleer?async_insert=1&wait_for_async_insert=0" - name: CAMELEER_STORAGE_METRICS value: "postgres" ``` Note: `CAMELEER_STORAGE_METRICS` defaults to `postgres`. Set to `clickhouse` when ready to switch. - [ ] **Step 3: Commit** ```bash git add deploy/clickhouse.yaml git add deploy/base/server.yaml git commit -m "deploy: add ClickHouse StatefulSet and server env vars" ``` --- ### Task 9: Integration Smoke Test **Files:** - Modify: `cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractPostgresIT.java` (add ClickHouse container) - [ ] **Step 1: Update AbstractPostgresIT to optionally start ClickHouse** This is a non-breaking addition. The existing integration tests continue to use PG. A future ClickHouse-specific IT can extend this base. Add the ClickHouse container to `AbstractPostgresIT`: ```java import org.testcontainers.clickhouse.ClickHouseContainer; // Add after opensearch static field: static final ClickHouseContainer clickhouse; // In the static initializer, add after opensearch.start(): clickhouse = new ClickHouseContainer("clickhouse/clickhouse-server:24.12"); clickhouse.start(); // In configureProperties, add: registry.add("clickhouse.enabled", () -> "true"); registry.add("clickhouse.url", clickhouse::getJdbcUrl); registry.add("clickhouse.username", clickhouse::getUsername); registry.add("clickhouse.password", clickhouse::getPassword); ``` - [ ] **Step 2: Run all integration tests** Run: `mvn verify -pl cameleer3-server-app` Expected: All ITs pass. ClickHouse schema initializer runs and creates the `agent_metrics` table. The `MetricsControllerIT` still uses PG (default storage flag is `postgres`). - [ ] **Step 3: Commit** ```bash git add cameleer3-server-app/src/test/java/com/cameleer3/server/app/AbstractPostgresIT.java git commit -m "test: add ClickHouse testcontainer to integration test base" ``` --- ### Task 10: Final Verification - [ ] **Step 1: Run full build** Run: `mvn clean verify` Expected: BUILD SUCCESS (unit tests + integration tests all pass) - [ ] **Step 2: Verify ClickHouse switch works locally (manual)** Start ClickHouse locally: ```bash docker run -d --name clickhouse-test -p 8123:8123 clickhouse/clickhouse-server:24.12 ``` Create the database: ```bash curl "http://localhost:8123/" --data "CREATE DATABASE IF NOT EXISTS cameleer" ``` Start the server with ClickHouse enabled: ```bash CLICKHOUSE_ENABLED=true CAMELEER_STORAGE_METRICS=clickhouse java -jar cameleer3-server-app/target/cameleer3-server-app-1.0-SNAPSHOT.jar ``` Verify schema initialization in the logs: `Executing ClickHouse schema script: V1__agent_metrics.sql` POST a metric: ```bash curl -X POST http://localhost:8081/api/v1/data/metrics \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '[{"agentId":"test","collectedAt":"2026-03-31T10:00:00Z","metricName":"cpu","metricValue":75.0,"tags":{}}]' ``` Verify data in ClickHouse: ```bash curl "http://localhost:8123/" --data "SELECT * FROM cameleer.agent_metrics FORMAT Pretty" ``` Stop and clean up: ```bash docker stop clickhouse-test && docker rm clickhouse-test ``` - [ ] **Step 3: Final commit (if any fixes were needed)** ```bash git add -A git commit -m "fix: address issues found during manual verification" ```