chore: update docs and config for PostgreSQL/OpenSearch storage layer
- Set failsafe reuseForks=true to reuse JVM across IT classes (faster test suite) - Replace ClickHouse with PostgreSQL+OpenSearch in docker-compose.yml - Remove redundant docker-compose.dev.yml - Update CLAUDE.md and HOWTO.md to reflect new storage stack Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
40
HOWTO.md
40
HOWTO.md
@@ -21,20 +21,20 @@ mvn clean verify # compile + run all tests (needs Docker for integrati
|
||||
|
||||
## Infrastructure Setup
|
||||
|
||||
Start ClickHouse:
|
||||
Start PostgreSQL and OpenSearch:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This starts ClickHouse 25.3 and automatically runs the schema init scripts (`clickhouse/init/01-schema.sql`, `clickhouse/init/02-search-columns.sql`, `clickhouse/init/03-users.sql`).
|
||||
This starts TimescaleDB (PostgreSQL 16) and OpenSearch 2.19. The database schema is applied automatically via Flyway migrations on server startup.
|
||||
|
||||
| Service | Port | Purpose |
|
||||
|------------|------|------------------|
|
||||
| ClickHouse | 8123 | HTTP API (JDBC) |
|
||||
| ClickHouse | 9000 | Native protocol |
|
||||
| Service | Port | Purpose |
|
||||
|------------|------|----------------------|
|
||||
| PostgreSQL | 5432 | JDBC (Spring JDBC) |
|
||||
| OpenSearch | 9200 | REST API (full-text) |
|
||||
|
||||
ClickHouse credentials: `cameleer` / `cameleer_dev`, database `cameleer3`.
|
||||
PostgreSQL credentials: `cameleer` / `cameleer_dev`, database `cameleer3`.
|
||||
|
||||
## Run the Server
|
||||
|
||||
@@ -109,7 +109,7 @@ The env-var local user gets `ADMIN` role. Agents get `AGENT` role at registratio
|
||||
|
||||
### OIDC Login (Optional)
|
||||
|
||||
OIDC configuration is stored in ClickHouse and managed via the admin API or UI. The SPA checks if OIDC is available:
|
||||
OIDC configuration is stored in PostgreSQL and managed via the admin API or UI. The SPA checks if OIDC is available:
|
||||
|
||||
```bash
|
||||
# 1. SPA checks if OIDC is available (returns 404 if not configured)
|
||||
@@ -340,9 +340,8 @@ Key settings in `cameleer3-server-app/src/main/resources/application.yml`:
|
||||
|---------|---------|-------------|
|
||||
| `server.port` | 8081 | Server port |
|
||||
| `ingestion.buffer-capacity` | 50000 | Max items in write buffer |
|
||||
| `ingestion.batch-size` | 5000 | Items per ClickHouse batch insert |
|
||||
| `ingestion.batch-size` | 5000 | Items per batch insert |
|
||||
| `ingestion.flush-interval-ms` | 1000 | Buffer flush interval (ms) |
|
||||
| `ingestion.data-ttl-days` | 30 | ClickHouse TTL for auto-deletion |
|
||||
| `agent-registry.heartbeat-interval-seconds` | 30 | Expected heartbeat interval |
|
||||
| `agent-registry.stale-threshold-seconds` | 90 | Time before agent marked STALE |
|
||||
| `agent-registry.dead-threshold-seconds` | 300 | Time after STALE before DEAD |
|
||||
@@ -386,7 +385,7 @@ npm run generate-api # Requires backend running on :8081
|
||||
|
||||
## Running Tests
|
||||
|
||||
Integration tests use Testcontainers (starts ClickHouse automatically — requires Docker):
|
||||
Integration tests use Testcontainers (starts PostgreSQL and OpenSearch automatically — requires Docker):
|
||||
|
||||
```bash
|
||||
# All tests
|
||||
@@ -399,14 +398,13 @@ mvn test -pl cameleer3-server-core
|
||||
mvn test -pl cameleer3-server-app -Dtest=ExecutionControllerIT
|
||||
```
|
||||
|
||||
## Verify ClickHouse Data
|
||||
## Verify Database Data
|
||||
|
||||
After posting data and waiting for the flush interval (1s default):
|
||||
|
||||
```bash
|
||||
docker exec -it cameleer3-server-clickhouse-1 clickhouse-client \
|
||||
--user cameleer --password cameleer_dev -d cameleer3 \
|
||||
-q "SELECT count() FROM route_executions"
|
||||
docker exec -it cameleer3-server-postgres-1 psql -U cameleer -d cameleer3 \
|
||||
-c "SELECT count(*) FROM route_executions"
|
||||
```
|
||||
|
||||
## Kubernetes Deployment
|
||||
@@ -417,7 +415,8 @@ The full stack is deployed to k3s via CI/CD on push to `main`. K8s manifests are
|
||||
|
||||
```
|
||||
cameleer namespace:
|
||||
ClickHouse (StatefulSet, 2Gi PVC) ← clickhouse:8123 (ClusterIP)
|
||||
PostgreSQL (StatefulSet, 10Gi PVC) ← postgres:5432 (ClusterIP)
|
||||
OpenSearch (StatefulSet, 10Gi PVC) ← opensearch:9200 (ClusterIP)
|
||||
cameleer3-server (Deployment) ← NodePort 30081
|
||||
cameleer3-ui (Deployment, Nginx) ← NodePort 30090
|
||||
Authentik Server (Deployment) ← NodePort 30950
|
||||
@@ -439,7 +438,7 @@ cameleer namespace:
|
||||
|
||||
Push to `main` triggers: **build** (UI npm + Maven, unit tests) → **docker** (buildx amd64 for server + UI, push to Gitea registry) → **deploy** (kubectl apply + rolling update).
|
||||
|
||||
Required Gitea org secrets: `REGISTRY_TOKEN`, `KUBECONFIG_BASE64`, `CAMELEER_AUTH_TOKEN`, `CAMELEER_JWT_SECRET`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CAMELEER_UI_USER` (optional), `CAMELEER_UI_PASSWORD` (optional), `AUTHENTIK_PG_PASSWORD`, `AUTHENTIK_SECRET_KEY`, `CAMELEER_OIDC_ENABLED`, `CAMELEER_OIDC_ISSUER`, `CAMELEER_OIDC_CLIENT_ID`, `CAMELEER_OIDC_CLIENT_SECRET`.
|
||||
Required Gitea org secrets: `REGISTRY_TOKEN`, `KUBECONFIG_BASE64`, `CAMELEER_AUTH_TOKEN`, `CAMELEER_JWT_SECRET`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, `OPENSEARCH_USER`, `OPENSEARCH_PASSWORD`, `CAMELEER_UI_USER` (optional), `CAMELEER_UI_PASSWORD` (optional), `AUTHENTIK_PG_USER`, `AUTHENTIK_PG_PASSWORD`, `AUTHENTIK_SECRET_KEY`, `CAMELEER_OIDC_ENABLED`, `CAMELEER_OIDC_ISSUER`, `CAMELEER_OIDC_CLIENT_ID`, `CAMELEER_OIDC_CLIENT_SECRET`.
|
||||
|
||||
### Manual K8s Commands
|
||||
|
||||
@@ -450,8 +449,11 @@ kubectl -n cameleer get pods
|
||||
# View server logs
|
||||
kubectl -n cameleer logs -f deploy/cameleer3-server
|
||||
|
||||
# View ClickHouse logs
|
||||
kubectl -n cameleer logs -f statefulset/clickhouse
|
||||
# View PostgreSQL logs
|
||||
kubectl -n cameleer logs -f statefulset/postgres
|
||||
|
||||
# View OpenSearch logs
|
||||
kubectl -n cameleer logs -f statefulset/opensearch
|
||||
|
||||
# Restart server
|
||||
kubectl -n cameleer rollout restart deployment/cameleer3-server
|
||||
|
||||
Reference in New Issue
Block a user