Commit Graph

307 Commits

Author SHA1 Message Date
hsiegeln
b393d262cb refactor: remove OIDC env var config and seeder
All checks were successful
CI / build (push) Successful in 1m7s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 39s
CI / deploy-feature (push) Has been skipped
OIDC configuration is already fully database-backed (oidc_config table,
admin API, OidcConfigRepository). Remove the redundant env var binding
(SecurityProperties.Oidc), the env-to-DB seeder (oidcConfigSeeder), and
the OIDC section from application.yml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 13:20:35 +01:00
hsiegeln
ff3a046f5a refactor: remove OIDC config from K8s manifests
All checks were successful
CI / build (push) Successful in 1m8s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 37s
CI / deploy-feature (push) Has been skipped
OIDC configuration should be managed by the server itself (database-backed),
not injected via K8s secrets. Remove all CAMELEER_OIDC_* env vars from
deployment manifests and the cameleer-oidc secret from CI. The server
defaults to OIDC disabled via application.yml.

This also fixes the Kustomize strategic merge conflict where the feature
overlay tried to set value on an env var that had valueFrom in the base.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 13:12:41 +01:00
hsiegeln
88df324b4b fix: preserve directory structure for feature overlay kustomize build
All checks were successful
CI / build (push) Successful in 1m7s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 14s
CI / deploy (push) Successful in 39s
CI / deploy-feature (push) Has been skipped
Kustomize rejects absolute paths for resource references. Instead of
rewriting ../../base to an absolute path, copy both base and overlay
into a temp directory preserving the relative path structure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 12:58:55 +01:00
hsiegeln
c1cf8ae260 chore: remove old flat deploy manifests superseded by Kustomize
Some checks failed
CI / build (push) Successful in 1m7s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 14s
CI / deploy (push) Has been skipped
CI / deploy-feature (push) Failing after 19s
deploy/server.yaml and deploy/ui.yaml are no longer referenced by CI,
which now uses deploy/base/ + deploy/overlays/main/ via kubectl apply -k.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 11:52:58 +01:00
hsiegeln
229463a2e8 fix: switch deploy containers from bitnami/kubectl to alpine/k8s
All checks were successful
CI / build (push) Successful in 1m8s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 13s
CI / deploy (push) Successful in 39s
CI / deploy-feature (push) Has been skipped
bitnami/kubectl lacks a package manager in the Gitea Actions runner,
so tool installation fails. alpine/k8s:1.32.3 ships with kubectl,
kustomize, git, jq, curl, and sed pre-installed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 11:39:58 +01:00
hsiegeln
15f20d22ad feat: add feature branch deployments with per-branch isolation
Some checks failed
CI / build (push) Successful in 1m8s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 42s
CI / deploy (push) Failing after 5s
CI / deploy-feature (push) Has been skipped
Enable deploying feature branches into isolated environments on the same
k3s cluster. Each branch gets its own namespace (cam-<slug>), PostgreSQL
schema, and OpenSearch index prefix for data isolation while sharing the
underlying infrastructure.

- Make OpenSearch index prefix and DB schema configurable via env vars
  (defaults preserve existing behavior)
- Restructure deploy/ into Kustomize base + overlays (main/feature)
- Extend CI to build Docker images for all branches, not just main
- Add deploy-feature job with namespace creation, secret copying,
  Traefik Ingress routing (<slug>-api/ui.cameleer.siegeln.net)
- Add cleanup-branch job to remove namespace, PG schema, OS indices
  on branch deletion
- Install required tools (git, jq, curl) in CI deploy containers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 11:35:07 +01:00
hsiegeln
672544660f fix: enable trackTotalHits for accurate OpenSearch result counts
All checks were successful
CI / build (push) Successful in 1m16s
CI / docker (push) Successful in 3m41s
CI / deploy (push) Successful in 44s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 09:54:50 +01:00
966db8545b Merge pull request 'fix: correct PostgreSQL mountPath and add external NodePort services' (#72) from feature/storage-layer-refactor into main
All checks were successful
CI / build (push) Successful in 1m9s
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 39s
Reviewed-on: cameleer/cameleer3-server#72
2026-03-17 01:04:06 +01:00
hsiegeln
c346babe33 fix: correct PostgreSQL mountPath and add external NodePort services
All checks were successful
CI / build (pull_request) Successful in 1m9s
CI / docker (pull_request) Has been skipped
CI / deploy (pull_request) Has been skipped
- Fix postgres.yaml mountPath to /home/postgres/pgdata matching timescaledb-ha PGDATA
- Add NodePort services for external access: PostgreSQL (30432), OpenSearch (30920)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 01:00:20 +01:00
8c2215ba58 Merge pull request 'refactor: replace ClickHouse with PostgreSQL/TimescaleDB + OpenSearch' (#71) from feature/storage-layer-refactor into main
All checks were successful
CI / build (push) Successful in 1m8s
CI / docker (push) Successful in 3m15s
CI / deploy (push) Successful in 3m34s
Reviewed-on: cameleer/cameleer3-server#71
2026-03-17 00:34:27 +01:00
hsiegeln
c316e80d7f chore: update docs and config for PostgreSQL/OpenSearch storage layer
All checks were successful
CI / build (pull_request) Successful in 1m20s
CI / docker (pull_request) Has been skipped
CI / deploy (pull_request) Has been skipped
- Set failsafe reuseForks=true to reuse JVM across IT classes (faster test suite)
- Replace ClickHouse with PostgreSQL+OpenSearch in docker-compose.yml
- Remove redundant docker-compose.dev.yml
- Update CLAUDE.md and HOWTO.md to reflect new storage stack

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 00:26:50 +01:00
hsiegeln
796be06a09 fix: resolve all integration test failures after storage layer refactor
- Use singleton container pattern for PostgreSQL + OpenSearch testcontainers
  (fixes container lifecycle issues with @TestInstance(PER_CLASS))
- Fix table name route_executions → executions in DetailControllerIT and
  ExecutionControllerIT
- Serialize processor headers as JSON (ObjectMapper) instead of Map.toString()
  for JSONB column compatibility
- Add nested mapping for processors field in OpenSearch index template
- Use .keyword sub-field for term queries on dynamically mapped text fields
- Add wildcard fallback queries for all text searches (substring matching)
- Isolate stats tests with unique route names to prevent data contamination
- Wait for OpenSearch indexing in SearchControllerIT with targeted Awaitility
- Reduce OpenSearch debounce to 100ms in test profile

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 00:02:19 +01:00
hsiegeln
26f5a2ce3b fix: update remaining ITs for synchronous ingestion and PostgreSQL storage
- SearchControllerIT: remove @TestInstance(PER_CLASS), use @BeforeEach with
  static guard, fix table name (route_executions -> executions), remove
  Awaitility polling
- OpenSearchIndexIT: replace Thread.sleep with explicit index refresh via
  OpenSearchClient
- DiagramLinkingIT: fix table name, remove Awaitility awaits (writes are
  synchronous)
- IngestionSchemaIT: rewrite queries for PostgreSQL relational model
  (processor_executions table instead of ClickHouse array columns)
- PostgresStatsStoreIT: use explicit time bounds in
  refresh_continuous_aggregate calls
- IngestionService: populate diagramContentHash during execution ingestion
  by looking up the latest diagram for the route+agent

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 22:03:29 +01:00
hsiegeln
d23b899f00 fix: prefix user tokens with 'user:' for JwtAuthenticationFilter routing 2026-03-16 21:01:57 +01:00
hsiegeln
288c7a86b5 chore: add docker-compose.dev.yml for local PostgreSQL + OpenSearch 2026-03-16 20:04:27 +01:00
hsiegeln
9f74e47ecf fix: use correct role-based JWT tokens in all integration tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 20:03:38 +01:00
hsiegeln
39f9925e71 fix: restore test config (bootstrap token, ingestion, agent-registry) and add @ActiveProfiles 2026-03-16 19:41:05 +01:00
hsiegeln
af03ecdf42 fix: use WITH NO DATA for continuous aggregates to avoid transaction block error 2026-03-16 19:32:54 +01:00
hsiegeln
0723f48e5b fix: disable Flyway transaction for continuous aggregate migration 2026-03-16 19:24:12 +01:00
hsiegeln
2634f60e59 fix: use timescaledb-ha image in K8s manifest for toolkit support 2026-03-16 19:14:15 +01:00
hsiegeln
3c0e615fb7 fix: use timescaledb-ha image which includes toolkit extension 2026-03-16 19:13:47 +01:00
hsiegeln
589da1b6d6 fix: use asCompatibleSubstituteFor for TimescaleDB Testcontainer image 2026-03-16 19:06:54 +01:00
hsiegeln
41e2038190 fix: use ChronoUnit for Instant arithmetic in PostgresStatsStoreIT 2026-03-16 19:04:42 +01:00
hsiegeln
ea687a342c deploy: remove obsolete ClickHouse K8s manifest 2026-03-16 19:01:26 +01:00
hsiegeln
cea16b38ed ci: update workflow for PostgreSQL + OpenSearch deployment
Replace ClickHouse credentials secret with postgres-credentials and
opensearch-credentials secrets. Update deploy step to apply postgres.yaml
and opensearch.yaml manifests instead of clickhouse.yaml, with appropriate
rollout status checks for each StatefulSet.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 19:00:20 +01:00
hsiegeln
a344be3a49 deploy: replace ClickHouse with PostgreSQL/TimescaleDB + OpenSearch in K8s manifests
- Dockerfile: update default SPRING_DATASOURCE_URL to jdbc:postgresql, add OPENSEARCH_URL default env
- deploy/postgres.yaml: new TimescaleDB StatefulSet + headless Service (10Gi PVC, pg_isready probes)
- deploy/opensearch.yaml: new OpenSearch 2.19.0 StatefulSet + headless Service (10Gi PVC, single-node, security disabled)
- deploy/server.yaml: switch datasource env from clickhouse-credentials to postgres-credentials, add OPENSEARCH_URL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:58:35 +01:00
hsiegeln
565b548ac1 refactor: remove all ClickHouse code, old interfaces, and SQL migrations
- Delete all ClickHouse storage implementations and config
- Delete old core interfaces (ExecutionRepository, DiagramRepository, MetricsRepository, SearchEngine, RawExecutionRow)
- Delete ClickHouse SQL migration files
- Delete AbstractClickHouseIT
- Update controllers to use new store interfaces (DiagramStore, ExecutionStore)
- Fix IngestionService calls in controllers for new synchronous API
- Migrate all ITs from AbstractClickHouseIT to AbstractPostgresIT
- Fix count() syntax and remove ClickHouse-specific test assertions
- Update TreeReconstructionTest for new buildTree() method

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:56:13 +01:00
hsiegeln
7dbfaf0932 feat: wire new storage beans, add MetricsFlushScheduler and RetentionScheduler
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:27:58 +01:00
hsiegeln
f7d7302694 feat: implement OpenSearchIndex with full-text and wildcard search
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:25:55 +01:00
hsiegeln
c48e0bdfde feat: implement debounced SearchIndexer for async OpenSearch indexing 2026-03-16 18:25:54 +01:00
hsiegeln
5932b5d969 feat: implement PostgresDiagramStore, PostgresUserRepository, PostgresOidcConfigRepository, PostgresMetricsStore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:23:21 +01:00
hsiegeln
527e2cf017 feat: implement PostgresStatsStore querying continuous aggregates 2026-03-16 18:22:44 +01:00
hsiegeln
9fd02c4edb feat: implement PostgresExecutionStore with upsert and dedup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:20:57 +01:00
hsiegeln
85ebe76111 refactor: IngestionService uses synchronous ExecutionStore writes with event publishing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:18:54 +01:00
hsiegeln
adf4b44d78 refactor: DetailService uses ExecutionStore, tree built from parentProcessorId 2026-03-16 18:18:53 +01:00
hsiegeln
84b93d74c7 refactor: SearchService uses SearchIndex + StatsStore instead of SearchEngine 2026-03-16 18:18:52 +01:00
hsiegeln
a55fc3c10d feat: add new storage interfaces for PostgreSQL/OpenSearch backends
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:16:53 +01:00
hsiegeln
55ed3be71a feat: add ExecutionDocument model and ExecutionUpdatedEvent
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:16:42 +01:00
hsiegeln
41a9a975fd config: switch datasource to PostgreSQL, add OpenSearch and Flyway config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:15:33 +01:00
hsiegeln
0eeae70369 test: add TimescaleDB test base class and Flyway migration smoke test 2026-03-16 18:15:32 +01:00
hsiegeln
8a637df65c feat: add Flyway migrations for PostgreSQL/TimescaleDB schema
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:13:53 +01:00
hsiegeln
5bed108d3b chore: swap ClickHouse deps for PostgreSQL, Flyway, OpenSearch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:13:45 +01:00
hsiegeln
ccc3f9fd92 Add storage layer refactor spec and implementation plan
All checks were successful
CI / build (push) Successful in 1m25s
CI / docker (push) Successful in 21s
CI / deploy (push) Successful in 32s
Design to replace ClickHouse with PostgreSQL/TimescaleDB + OpenSearch.
PostgreSQL as source of truth with continuous aggregates for analytics,
OpenSearch for full-text wildcard search. 21-task implementation plan.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:05:16 +01:00
hsiegeln
5ee78f7673 Reduce chart grid noise: subtle dashed Y-grid only, no X-grid or ticks
All checks were successful
CI / build (push) Successful in 1m13s
CI / docker (push) Successful in 52s
CI / deploy (push) Successful in 30s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:17:38 +01:00
hsiegeln
8c605d7523 Fix missing avg duration comparison on route Performance tab
All checks were successful
CI / build (push) Successful in 1m17s
CI / docker (push) Successful in 50s
CI / deploy (push) Successful in 29s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:14:11 +01:00
hsiegeln
4ea6814bb3 Fix unused import in AppScopedView
All checks were successful
CI / build (push) Successful in 1m8s
CI / docker (push) Successful in 53s
CI / deploy (push) Successful in 34s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 15:49:35 +01:00
hsiegeln
7fd8a787d0 UI overhaul: unified sidebar layout with app-scoped views
Some checks failed
CI / build (push) Failing after 48s
CI / docker (push) Has been skipped
CI / deploy (push) Has been skipped
Replace disconnected Transactions/Applications pages with a persistent
collapsible sidebar listing apps by health status. Add app-scoped view
(/apps/:group) with filtered stats, route chips, and scoped table.
Merge Processor Tree into diagram detail panel with Inspector/Tree
toggle and resizable divider. Remove max-width constraint for full
viewport usage. All view states are deep-linkable via URL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 15:47:33 +01:00
hsiegeln
0b56590e3f Fix Swagger UI CORS: add /api/v1 server URL to OpenAPI spec
All checks were successful
CI / build (push) Successful in 1m15s
CI / docker (push) Successful in 44s
CI / deploy (push) Successful in 29s
The empty servers list caused Swagger UI to construct request URLs
without the /api/v1 prefix, resulting in CORS/fetch failures.
Adding a relative server entry makes paths resolve correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 14:59:12 +01:00
hsiegeln
7dec8fbaff Add embedded Swagger UI page with auto-injected JWT auth
All checks were successful
CI / build (push) Successful in 1m16s
CI / docker (push) Successful in 1m9s
CI / deploy (push) Successful in 31s
- New /swagger route with lazy-loaded SwaggerPage that initializes
  swagger-ui-dist and injects the session JWT via requestInterceptor
- Move API link from primary nav to navRight utility area (pill style)
- Code-split swagger chunk (~1.4 MB) so main bundle stays lean

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 14:51:15 +01:00
hsiegeln
e466dc5861 Add API link in nav bar pointing to Swagger UI
All checks were successful
CI / build (push) Successful in 1m13s
CI / docker (push) Successful in 51s
CI / deploy (push) Successful in 35s
Opens /api/v1/swagger-ui in a new tab for manual endpoint testing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 14:35:23 +01:00