Commit Graph

110 Commits

Author SHA1 Message Date
hsiegeln
f42e6279e6 fix: null safety in role/group creation, add user create/update endpoints
- RoleAdminController.createRole: default null description to "" and null scope to "custom"
- RoleAdminController.updateRole: pass null audit details to avoid NPE when name is null
- GroupAdminController.updateGroup: pass null audit details to avoid NPE when name is null
- UserAdminController: add POST / createUser endpoint with default VIEWER role assignment
- UserAdminController: add PUT /{userId} updateUser endpoint for displayName/email updates
2026-03-17 18:49:34 +01:00
hsiegeln
4842507ff3 feat: seed built-in Admins group and assign admin users on login
- Add V2 Flyway migration to create built-in Admins group (id: ...0010) with ADMIN role
- Add ADMINS_GROUP_ID constant to SystemRole
- Add user to Admins group on successful local login alongside role assignment
2026-03-17 18:30:16 +01:00
hsiegeln
01295c84d8 feat: add Group, Role, and RBAC stats admin controllers
GroupAdminController with cycle detection, RoleAdminController
with system role protection, RbacStatsController for dashboard.
Rewrite UserAdminController to use RbacService.
2026-03-17 17:47:26 +01:00
hsiegeln
eb0cc8c141 feat: replace flat users.roles with relational RBAC model
New package com.cameleer3.server.core.rbac with SystemRole constants,
detail/summary records, GroupRepository, RoleRepository, RbacService.
Remove roles field from UserInfo. Implement PostgresGroupRepository,
PostgresRoleRepository, RbacServiceImpl with inheritance computation.
Update UiAuthController, OidcAuthController, AgentRegistrationController
to assign roles via user_roles table. JWT populated from effective system roles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 17:44:32 +01:00
hsiegeln
b06b3f52a8 refactor: consolidate V1-V10 Flyway migrations into single V1__init.sql
Add RBAC tables (roles, groups, group_roles, user_groups, user_roles)
with system role seeds and join indexes. Drop users.roles TEXT[] column.
2026-03-17 17:34:15 +01:00
hsiegeln
321b8808cc feat: add ThresholdAdminController and AuditLogController with integration tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:57:23 +01:00
hsiegeln
c6da858c2f feat: add OpenSearchAdminController with status, pipeline, indices, performance, and delete endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:57:18 +01:00
hsiegeln
c6b2f7c331 feat: add DatabaseAdminController with status, pool, tables, queries, and kill endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:57:14 +01:00
hsiegeln
0cea8af6bc feat: add response/request DTOs for admin infrastructure endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:51:31 +01:00
hsiegeln
1d6ae00b1c feat: wire AuditService, enable method security, retrofit audit logging into existing controllers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:51:22 +01:00
hsiegeln
e8842e3bdc feat: add Postgres implementations for AuditRepository and ThresholdRepository
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:51:13 +01:00
hsiegeln
a0944a1c72 feat: add audit domain model, repository interface, AuditService, and unit test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 15:36:21 +01:00
hsiegeln
fa3bc592d1 feat: add Flyway V9 (thresholds) and V10 (audit_log) migrations 2026-03-17 15:32:20 +01:00
hsiegeln
82117deaab fix: pass credentials to Flyway when using separate datasource URL
All checks were successful
CI / build (push) Successful in 1m6s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 42s
CI / deploy (push) Successful in 40s
CI / deploy-feature (push) Has been skipped
When spring.flyway.url is set independently, Spring Boot does not
inherit credentials from spring.datasource. Add explicit user/password
to both application.yml and K8s deployment to prevent "no password"
failures on feature branch deployments.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 13:34:41 +01:00
hsiegeln
247fdb01c0 fix: separate Flyway and app datasource search paths for schema isolation
Some checks failed
CI / build (push) Successful in 1m6s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 41s
CI / deploy (push) Failing after 2m19s
CI / deploy-feature (push) Has been skipped
Flyway needs public in the search_path to access TimescaleDB extension
functions (create_hypertable). The app datasource must NOT include public
to prevent accidental cross-schema reads from production data.

- spring.flyway.url: currentSchema=<branch>,public (extensions accessible)
- spring.datasource.url: currentSchema=<branch> (strict isolation)
- SPRING_FLYWAY_URL env var added to K8s base manifest

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 13:26:01 +01:00
hsiegeln
b393d262cb refactor: remove OIDC env var config and seeder
All checks were successful
CI / build (push) Successful in 1m7s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 39s
CI / deploy-feature (push) Has been skipped
OIDC configuration is already fully database-backed (oidc_config table,
admin API, OidcConfigRepository). Remove the redundant env var binding
(SecurityProperties.Oidc), the env-to-DB seeder (oidcConfigSeeder), and
the OIDC section from application.yml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 13:20:35 +01:00
hsiegeln
15f20d22ad feat: add feature branch deployments with per-branch isolation
Some checks failed
CI / build (push) Successful in 1m8s
CI / cleanup-branch (push) Has been skipped
CI / docker (push) Successful in 42s
CI / deploy (push) Failing after 5s
CI / deploy-feature (push) Has been skipped
Enable deploying feature branches into isolated environments on the same
k3s cluster. Each branch gets its own namespace (cam-<slug>), PostgreSQL
schema, and OpenSearch index prefix for data isolation while sharing the
underlying infrastructure.

- Make OpenSearch index prefix and DB schema configurable via env vars
  (defaults preserve existing behavior)
- Restructure deploy/ into Kustomize base + overlays (main/feature)
- Extend CI to build Docker images for all branches, not just main
- Add deploy-feature job with namespace creation, secret copying,
  Traefik Ingress routing (<slug>-api/ui.cameleer.siegeln.net)
- Add cleanup-branch job to remove namespace, PG schema, OS indices
  on branch deletion
- Install required tools (git, jq, curl) in CI deploy containers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 11:35:07 +01:00
hsiegeln
672544660f fix: enable trackTotalHits for accurate OpenSearch result counts
All checks were successful
CI / build (push) Successful in 1m16s
CI / docker (push) Successful in 3m41s
CI / deploy (push) Successful in 44s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 09:54:50 +01:00
hsiegeln
c316e80d7f chore: update docs and config for PostgreSQL/OpenSearch storage layer
All checks were successful
CI / build (pull_request) Successful in 1m20s
CI / docker (pull_request) Has been skipped
CI / deploy (pull_request) Has been skipped
- Set failsafe reuseForks=true to reuse JVM across IT classes (faster test suite)
- Replace ClickHouse with PostgreSQL+OpenSearch in docker-compose.yml
- Remove redundant docker-compose.dev.yml
- Update CLAUDE.md and HOWTO.md to reflect new storage stack

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 00:26:50 +01:00
hsiegeln
796be06a09 fix: resolve all integration test failures after storage layer refactor
- Use singleton container pattern for PostgreSQL + OpenSearch testcontainers
  (fixes container lifecycle issues with @TestInstance(PER_CLASS))
- Fix table name route_executions → executions in DetailControllerIT and
  ExecutionControllerIT
- Serialize processor headers as JSON (ObjectMapper) instead of Map.toString()
  for JSONB column compatibility
- Add nested mapping for processors field in OpenSearch index template
- Use .keyword sub-field for term queries on dynamically mapped text fields
- Add wildcard fallback queries for all text searches (substring matching)
- Isolate stats tests with unique route names to prevent data contamination
- Wait for OpenSearch indexing in SearchControllerIT with targeted Awaitility
- Reduce OpenSearch debounce to 100ms in test profile

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 00:02:19 +01:00
hsiegeln
26f5a2ce3b fix: update remaining ITs for synchronous ingestion and PostgreSQL storage
- SearchControllerIT: remove @TestInstance(PER_CLASS), use @BeforeEach with
  static guard, fix table name (route_executions -> executions), remove
  Awaitility polling
- OpenSearchIndexIT: replace Thread.sleep with explicit index refresh via
  OpenSearchClient
- DiagramLinkingIT: fix table name, remove Awaitility awaits (writes are
  synchronous)
- IngestionSchemaIT: rewrite queries for PostgreSQL relational model
  (processor_executions table instead of ClickHouse array columns)
- PostgresStatsStoreIT: use explicit time bounds in
  refresh_continuous_aggregate calls
- IngestionService: populate diagramContentHash during execution ingestion
  by looking up the latest diagram for the route+agent

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 22:03:29 +01:00
hsiegeln
d23b899f00 fix: prefix user tokens with 'user:' for JwtAuthenticationFilter routing 2026-03-16 21:01:57 +01:00
hsiegeln
9f74e47ecf fix: use correct role-based JWT tokens in all integration tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 20:03:38 +01:00
hsiegeln
39f9925e71 fix: restore test config (bootstrap token, ingestion, agent-registry) and add @ActiveProfiles 2026-03-16 19:41:05 +01:00
hsiegeln
af03ecdf42 fix: use WITH NO DATA for continuous aggregates to avoid transaction block error 2026-03-16 19:32:54 +01:00
hsiegeln
0723f48e5b fix: disable Flyway transaction for continuous aggregate migration 2026-03-16 19:24:12 +01:00
hsiegeln
3c0e615fb7 fix: use timescaledb-ha image which includes toolkit extension 2026-03-16 19:13:47 +01:00
hsiegeln
589da1b6d6 fix: use asCompatibleSubstituteFor for TimescaleDB Testcontainer image 2026-03-16 19:06:54 +01:00
hsiegeln
41e2038190 fix: use ChronoUnit for Instant arithmetic in PostgresStatsStoreIT 2026-03-16 19:04:42 +01:00
hsiegeln
565b548ac1 refactor: remove all ClickHouse code, old interfaces, and SQL migrations
- Delete all ClickHouse storage implementations and config
- Delete old core interfaces (ExecutionRepository, DiagramRepository, MetricsRepository, SearchEngine, RawExecutionRow)
- Delete ClickHouse SQL migration files
- Delete AbstractClickHouseIT
- Update controllers to use new store interfaces (DiagramStore, ExecutionStore)
- Fix IngestionService calls in controllers for new synchronous API
- Migrate all ITs from AbstractClickHouseIT to AbstractPostgresIT
- Fix count() syntax and remove ClickHouse-specific test assertions
- Update TreeReconstructionTest for new buildTree() method

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:56:13 +01:00
hsiegeln
7dbfaf0932 feat: wire new storage beans, add MetricsFlushScheduler and RetentionScheduler
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:27:58 +01:00
hsiegeln
f7d7302694 feat: implement OpenSearchIndex with full-text and wildcard search
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:25:55 +01:00
hsiegeln
5932b5d969 feat: implement PostgresDiagramStore, PostgresUserRepository, PostgresOidcConfigRepository, PostgresMetricsStore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:23:21 +01:00
hsiegeln
527e2cf017 feat: implement PostgresStatsStore querying continuous aggregates 2026-03-16 18:22:44 +01:00
hsiegeln
9fd02c4edb feat: implement PostgresExecutionStore with upsert and dedup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:20:57 +01:00
hsiegeln
41a9a975fd config: switch datasource to PostgreSQL, add OpenSearch and Flyway config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:15:33 +01:00
hsiegeln
0eeae70369 test: add TimescaleDB test base class and Flyway migration smoke test 2026-03-16 18:15:32 +01:00
hsiegeln
8a637df65c feat: add Flyway migrations for PostgreSQL/TimescaleDB schema
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:13:53 +01:00
hsiegeln
5bed108d3b chore: swap ClickHouse deps for PostgreSQL, Flyway, OpenSearch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 18:13:45 +01:00
hsiegeln
0b56590e3f Fix Swagger UI CORS: add /api/v1 server URL to OpenAPI spec
All checks were successful
CI / build (push) Successful in 1m15s
CI / docker (push) Successful in 44s
CI / deploy (push) Successful in 29s
The empty servers list caused Swagger UI to construct request URLs
without the /api/v1 prefix, resulting in CORS/fetch failures.
Adding a relative server entry makes paths resolve correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 14:59:12 +01:00
hsiegeln
48d944354a Fix ClickHouse OOM: PREWHERE on active-count query + per-query memory limits
All checks were successful
CI / build (push) Successful in 1m16s
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 33s
The active-count query scanned all wide rows on the base table, exceeding
the 3.6 GiB memory limit. Use PREWHERE status = 'RUNNING' so ClickHouse
reads only the status column first. Add SETTINGS max_memory_usage = 1 GiB
to all queries so concurrent requests degrade gracefully instead of crashing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 11:55:26 +01:00
hsiegeln
5ad0c75da8 Truncate rollup params to second precision for DateTime column
All checks were successful
CI / build (push) Successful in 1m10s
CI / docker (push) Successful in 42s
CI / deploy (push) Successful in 29s
The JDBC driver sends java.sql.Timestamp with nanoseconds as a string
(e.g. '2026-03-15 10:13:58.105931162') which DateTime('UTC') rejects.
Add bucketTimestamp() helper that truncates to seconds for all rollup
query parameters.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 11:15:42 +01:00
hsiegeln
8e6f8e2693 Fix bucket alignment: compute 5-min floor in Java, not ClickHouse SQL
All checks were successful
CI / build (push) Successful in 1m12s
CI / docker (push) Successful in 40s
CI / deploy (push) Successful in 31s
JDBC sends Timestamp params as strings, causing toStartOfFiveMinutes()
to fail with 'Illegal type String'. Floor to 5-minute boundaries in
Java instead and pass plain bucket >= ? comparisons.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 11:10:40 +01:00
hsiegeln
f660e88a17 Fix rollup queries: alias shadowed AggregateFunction column name
All checks were successful
CI / build (push) Successful in 1m12s
CI / docker (push) Successful in 41s
CI / deploy (push) Successful in 31s
countMerge(total_count) in the avg expression resolved to the UInt64
alias 'total_count' instead of the AggregateFunction column. Rename
SELECT aliases (cnt, failed, avg_ms, p99_ms) to avoid shadowing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 11:05:10 +01:00
hsiegeln
035356288f Fix stats rollup: AggregateFunction(count) takes no type argument
All checks were successful
CI / build (push) Successful in 1m13s
CI / docker (push) Successful in 40s
CI / deploy (push) Successful in 31s
ClickHouse count() accepts no arguments, so the column type must be
AggregateFunction(count) not AggregateFunction(count, UInt64). The
latter causes countMerge() to fail with ILLEGAL_TYPE_OF_ARGUMENT.
Drop and recreate the table/MV to apply the corrected schema.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 10:52:29 +01:00
hsiegeln
adf13f0430 Add 5-minute AggregatingMergeTree stats rollup for dashboard queries
All checks were successful
CI / build (push) Successful in 1m14s
CI / docker (push) Successful in 42s
CI / deploy (push) Successful in 30s
Pre-aggregate route execution stats into 5-minute buckets using a
materialized view with -State/-Merge combinators. Rewrite stats() and
timeseries() to query the rollup table instead of scanning the wide
base table. Active count remains a real-time query since RUNNING is
transient. Includes idempotent backfill migration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 10:46:26 +01:00
6f7c92f793 cameleer3-server-app/src/main/java/com/cameleer3/server/app/diagram/ElkDiagramRenderer.java aktualisiert
All checks were successful
CI / build (push) Successful in 1m13s
CI / docker (push) Successful in 40s
CI / deploy (push) Successful in 30s
increased node spacing to 90
2026-03-15 10:37:56 +01:00
hsiegeln
520590fbf4 Increase ELK node spacing and revert frontend node height to 40
All checks were successful
CI / build (push) Successful in 1m10s
CI / docker (push) Successful in 51s
CI / deploy (push) Successful in 33s
NODE_SPACING 40→60 gives edges more vertical room between nodes.
FIXED_H reverted to 40 to match backend NODE_HEIGHT.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 10:31:06 +01:00
hsiegeln
7778793e7b Add route diagram page with execution overlay and group-aware APIs
All checks were successful
CI / build (push) Successful in 1m10s
CI / docker (push) Successful in 1m3s
CI / deploy (push) Successful in 31s
Backend: Add group filtering to agent list, search, stats, and timeseries
endpoints. Add diagram lookup by group+routeId. Resolve application group
to agent IDs server-side for ClickHouse IN-clause queries.

Frontend: New route detail page at /apps/{group}/routes/{routeId} with
three tabs (Diagram, Performance, Processor Tree). SVG diagram rendering
with panzoom, execution overlay (glow effects, duration/sequence badges,
flow particles, minimap), and processor detail panel. uPlot charts for
performance tab replacing old SVG sparklines. Ctrl+Click from
ExecutionExplorer navigates to route diagram with overlay.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 21:35:42 +01:00
hsiegeln
b64edaa16f Server-side sorting for execution search results
All checks were successful
CI / build (push) Successful in 1m12s
CI / docker (push) Successful in 50s
CI / deploy (push) Successful in 33s
Sorting now applies to the entire result set via ClickHouse ORDER BY
instead of only sorting the current page client-side. Default sort
order is timestamp descending. Supported sort columns: startTime,
status, agentId, routeId, correlationId, durationMs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 19:34:22 +01:00