cameleer-saas/docs/architecture.md

# Cameleer SaaS Architecture

**Last updated:** 2026-04-05
**Status:** Living document -- update as the system evolves

---

## 1. System Overview

Cameleer SaaS is a multi-tenant platform that provides managed observability for
Apache Camel applications. Customers deploy their Camel JARs through the SaaS
platform and get zero-code instrumentation, execution tracing, route topology
visualization, and runtime control -- without running any observability
infrastructure themselves.

The system comprises three components:

**Cameleer Agent** (`cameleer` repo) -- A Java agent using ByteBuddy for
zero-code bytecode instrumentation. Captures route executions, processor traces,
payloads, metrics, and route graph topology. Deployed as a `-javaagent` JAR
alongside the customer's application.

**Cameleer Server** (`cameleer-server` repo) -- A Spring Boot observability
backend. Receives telemetry from agents via HTTP, pushes configuration and
commands to agents via SSE. Stores data in PostgreSQL and ClickHouse. Provides
a React SPA dashboard for direct observability access. JWT auth with Ed25519
config signing.

**Cameleer SaaS** (this repo) -- The multi-tenancy, deployment, and management
layer. Handles user authentication via Logto OIDC, tenant provisioning, JAR
upload and deployment, API key management, license generation, and audit
logging. Serves a React SPA that wraps the full user experience.

---

## 2. Component Topology

```
                     Internet / LAN
                          |
                    +-----+-----+
                    |  Traefik  |  :80 / :443
                    |  (v3)     |  Reverse proxy + TLS termination
                    +-----+-----+
                          |
          +---------------+---------------+-------------------+
          |               |               |                   |
   PathPrefix(/api)  PathPrefix(/)    PathPrefix(/oidc)  PathPrefix(/observe)
   PathPrefix(/api)  priority=1       PathPrefix(         PathPrefix(/dashboard)
          |               |           /interaction)            |
          v               v               v                    v
   +--------------+ +--------------+ +-----------+   +------------------+
   | cameleer-saas| | cameleer-saas| |   Logto   |   | cameleer-server |
   |  (API)       | |  (SPA)       | |           |   |                  |
   |  :8080       | |  :8080       | |  :3001    |   |  :8081           |
   +--------------+ +--------------+ +-----------+   +------------------+
          |                                |                  |
          +------+-------------------------+------------------+
                 |                         |                  |
          +------+------+          +------+------+    +------+------+
          |  PostgreSQL  |          |  PostgreSQL  |    |  ClickHouse |
          |  :5432       |          |  (logto DB)  |    |  :8123      |
          |  cameleer_   |          |  :5432       |    |  cameleer   |
          |  saas DB     |          +--------------+    +-------------+
          +--------------+
                                          |
                                   +------+------+
                                   | Customer    |
                                   | App + Agent |
                                   | (container) |
                                   +-------------+
```

### Services

| Service           | Image                                       | Internal Port | Network  | Purpose                          |
|-------------------|---------------------------------------------|---------------|----------|----------------------------------|
| traefik           | `traefik:v3`                                | 80, 443       | cameleer | Reverse proxy, TLS, routing      |
| postgres          | `postgres:16-alpine`                        | 5432          | cameleer | Shared PostgreSQL (3 databases)  |
| logto             | `ghcr.io/logto-io/logto:latest`             | 3001          | cameleer | OIDC identity provider           |
| logto-bootstrap   | `postgres:16-alpine` (ephemeral)            | --            | cameleer | One-shot bootstrap script        |
| cameleer-saas     | `gitea.siegeln.net/cameleer/cameleer-saas`  | 8080          | cameleer | SaaS API + SPA serving           |
| cameleer-server  | `gitea.siegeln.net/cameleer/cameleer-server`| 8081          | cameleer | Observability backend            |
| clickhouse        | `clickhouse/clickhouse-server:latest`       | 8123          | cameleer | Time-series telemetry storage    |

### Docker Network

All services share a single Docker bridge network named `cameleer`. Customer app
containers are also attached to this network so agents can reach the
cameleer-server.

### Volumes

| Volume          | Mounted By          | Purpose                                    |
|-----------------|---------------------|--------------------------------------------|
| `pgdata`        | postgres            | PostgreSQL data persistence                |
| `chdata`        | clickhouse          | ClickHouse data persistence                |
| `acme`          | traefik             | TLS certificate storage                    |
| `jardata`       | cameleer-saas       | Uploaded customer JAR files                |
| `bootstrapdata` | logto-bootstrap, cameleer-saas | Bootstrap output JSON (shared) |

### Databases on PostgreSQL

The shared PostgreSQL instance hosts three databases:

- `cameleer_saas` -- SaaS platform tables (tenants, environments, apps, etc.)
- `logto` -- Logto identity provider data
- `cameleer` -- cameleer-server operational data

The `docker/init-databases.sh` init script creates all three during first start.

---

## 3. Authentication & Authorization

### 3.1 Design Principles

1. **Logto is the single identity provider** for all human users.
2. **Zero trust** -- every service validates tokens independently via JWKS or its
   own signing key. No identity in HTTP headers.
3. **No custom crypto** -- standard protocols only (OAuth2, OIDC, JWT, SHA-256).
4. **API keys for agents** -- per-environment opaque secrets, exchanged for
   server-issued JWTs via the bootstrap registration flow.

### 3.2 Token Types

| Token              | Issuer          | Algorithm        | Validator            | Used By                        |
|--------------------|-----------------|------------------|----------------------|--------------------------------|
| Logto user JWT     | Logto           | ES384 (asymmetric)| Any service via JWKS | SaaS UI users, server users    |
| Logto M2M JWT      | Logto           | ES384 (asymmetric)| Any service via JWKS | SaaS platform -> server calls  |
| Server internal JWT| cameleer-server| HS256 (symmetric) | Issuing server only  | Agents (after registration)    |
| API key (opaque)   | SaaS platform   | N/A (SHA-256 hash)| cameleer-server     | Agent initial registration     |
| Ed25519 signature  | cameleer-server| EdDSA            | Agent                | Server -> agent command signing|

### 3.3 Scope Model

The Logto API resource `https://api.cameleer.local` has 10 scopes, created by
the bootstrap script (`docker/logto-bootstrap.sh`):

| Scope              | Description                    | Platform Admin | Org Admin | Org Member |
|--------------------|--------------------------------|:--------------:|:---------:|:----------:|
| `platform:admin`   | SaaS platform administration   | x              |           |            |
| `tenant:manage`    | Manage tenant settings         | x              | x         |            |
| `billing:manage`   | Manage billing                 | x              | x         |            |
| `team:manage`      | Manage team members            | x              | x         |            |
| `apps:manage`      | Create and delete apps         | x              | x         |            |
| `apps:deploy`      | Deploy apps                    | x              | x         | x          |
| `secrets:manage`   | Manage secrets                 | x              | x         |            |
| `observe:read`     | View observability data        | x              | x         | x          |
| `observe:debug`    | Debug and replay operations    | x              | x         | x          |
| `settings:manage`  | Manage settings                | x              | x         |            |

**Role hierarchy:**

- **Global role `platform-admin`** -- All 10 scopes. Assigned to SaaS owner.
- **Organization role `admin`** -- 9 tenant-level scopes (all except `platform:admin`).
- **Organization role `member`** -- 3 scopes: `apps:deploy`, `observe:read`,
  `observe:debug`.

### 3.4 Authentication Flows

**Human user -> SaaS Platform:**

```
  Browser                    Logto                   cameleer-saas
    |                          |                          |
    |--- OIDC auth code flow ->|                          |
    |<-- id_token, auth code --|                          |
    |                          |                          |
    |--- getAccessToken(resource, orgId) ---------------->|
    |    (org-scoped JWT with scope claim)                 |
    |                          |                          |
    |--- GET /api/me, Authorization: Bearer <jwt> ------->|
    |                          |    validate via JWKS      |
    |                          |    extract organization_id|
    |                          |    resolve to tenant      |
    |<-- { userId, tenants } -----------------------------|
```

1. User authenticates with Logto (OIDC authorization code flow via `@logto/react`).
2. Frontend obtains org-scoped access token via `getAccessToken(resource, orgId)`.
3. Backend validates via Logto JWKS (Spring OAuth2 Resource Server).
4. `organization_id` claim in JWT resolves to internal tenant ID via
   `TenantIsolationInterceptor`.

**SaaS platform -> cameleer-server API (M2M):**

1. SaaS platform obtains Logto M2M token (`client_credentials` grant) via
   `LogtoManagementClient`.
2. Calls server API with `Authorization: Bearer <logto-m2m-token>`.
3. Server validates via Logto JWKS (OIDC resource server support).
4. Server grants ADMIN role to valid M2M tokens.

**Agent -> cameleer-server:**

1. Agent reads `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` environment variable (API key).
2. Calls `POST /api/v1/agents/register` with the key as Bearer token.
3. Server validates via `BootstrapTokenValidator` (constant-time comparison).
4. Server issues internal HMAC JWT (access + refresh) + Ed25519 public key.
5. Agent uses JWT for all subsequent requests, refreshes on expiry.

**Server -> Agent (commands):**

1. Server signs command payload with Ed25519 private key.
2. Sends via SSE with signature field.
3. Agent verifies using server's public key (received at registration).
4. Destructive commands require a nonce (replay protection).

### 3.5 Spring Security Configuration

`SecurityConfig.java` configures a single stateless filter chain:

```java
@Configuration
@EnableWebSecurity
@EnableMethodSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http
            .csrf(csrf -> csrf.disable())
            .sessionManagement(s -> s.sessionCreationPolicy(STATELESS))
            .authorizeHttpRequests(auth -> auth
                .requestMatchers("/actuator/health").permitAll()
                .requestMatchers("/api/config").permitAll()
                .requestMatchers("/", "/index.html", "/login", "/callback",
                    "/environments/**", "/license", "/admin/**").permitAll()
                .requestMatchers("/assets/**", "/favicon.ico").permitAll()
                .anyRequest().authenticated()
            )
            .oauth2ResourceServer(oauth2 -> oauth2.jwt(jwt ->
                jwt.jwtAuthenticationConverter(jwtAuthenticationConverter())));
        return http.build();
    }
}
```

**JWT processing pipeline:**

1. `BearerTokenAuthenticationFilter` (Spring built-in) extracts the Bearer token.
2. `JwtDecoder` validates the token signature (ES384 via Logto JWKS) and issuer.
   Accepts both `JWT` and `at+jwt` token types (RFC 9068 / Logto convention).
3. `JwtAuthenticationConverter` maps the `scope` claim to Spring authorities:
   `scope: "platform:admin observe:read"` becomes `SCOPE_platform:admin` and
   `SCOPE_observe:read`.
4. `TenantIsolationInterceptor` (registered as a `HandlerInterceptor` on
   `/api/**` via `WebConfig`) reads `organization_id` from the JWT, resolves it
   to an internal tenant UUID via `TenantService.getByLogtoOrgId()`, stores it
   on `TenantContext` (ThreadLocal), and validates path variable isolation (see
   Section 8.1).

**Authorization enforcement** -- Every mutating API endpoint uses Spring
`@PreAuthorize` annotations with `SCOPE_` authorities. Read-only list/get
endpoints require authentication only (no specific scope). The scope-to-endpoint
mapping:

| Scope            | Endpoints                                                                |
|------------------|--------------------------------------------------------------------------|
| `platform:admin` | `GET /api/tenants` (list all), `POST /api/tenants` (create tenant)       |
| `apps:manage`    | Environment create/update/delete, app create/delete                      |
| `apps:deploy`    | JAR upload, routing patch, deploy/stop/restart                           |
| `billing:manage` | License generation                                                       |
| `observe:read`   | Log queries, agent status, observability status                          |
| *(auth only)*    | List/get-by-ID endpoints (environments, apps, deployments, licenses)     |

Example:

```java
@PreAuthorize("hasAuthority('SCOPE_apps:manage')")
public ResponseEntity<EnvironmentResponse> create(...) { ... }
```

### 3.6 Frontend Auth Architecture

**Logto SDK integration** (`main.tsx`):

The `LogtoProvider` is configured with scopes including `UserScope.Organizations`
and `UserScope.OrganizationRoles`, requesting organization-aware tokens from
Logto.

**Token management** (`TokenSync` component in `main.tsx`):

When an organization is selected, `setTokenProvider` is called with
`getAccessToken(resource, orgId)` to produce org-scoped JWTs. When no org is
selected, a non-org-scoped token is used.

**Organization resolution** (`OrgResolver.tsx`):

`OrgResolver` uses two separate `useEffect` hooks to keep org state and scopes
in sync:

- **Effect 1: Org population** (depends on `[me]`) -- Calls `GET /api/me` to
  fetch tenant memberships, maps them to `OrgInfo` objects in the Zustand org
  store, and auto-selects the first org if the user belongs to exactly one.
- **Effect 2: Scope fetching** (depends on `[me, currentOrgId]`) -- Fetches the
  API resource identifier from `/api/config`, then obtains an org-scoped access
  token (`getAccessToken(resource, orgId)`). Scopes are decoded from the JWT
  payload and written to the store via `setScopes()`. A single token fetch is
  sufficient because Logto merges all granted scopes (including global scopes
  like `platform:admin`) into the org-scoped token.

The two-effect split ensures scopes are re-fetched whenever the user switches
organizations, preventing stale scope sets from a previously selected org.

**Scope-based UI gating:**

The `useOrgStore` exposes a `scopes: Set<string>` that components check to
conditionally render UI elements. For example, admin-only controls check for
`platform:admin` in the scope set.

**Route protection** (`ProtectedRoute.tsx`):

Wraps authenticated routes. Redirects to `/login` when the user is not
authenticated. Uses a ref to avoid showing a spinner after the initial auth
check completes (the Logto SDK sets `isLoading=true` for every async method,
not just initial load).

---

## 4. Data Model

### 4.1 Entity Relationship Diagram

```
  +-------------------+
  |     tenants       |
  +-------------------+
  | id (PK, UUID)     |
  | name              |
  | slug (UNIQUE)     |
  | tier              |
  | status            |
  | logto_org_id      |
  | stripe_customer_id|
  | stripe_sub_id     |
  | settings (JSONB)  |
  | created_at        |
  | updated_at        |
  +--------+----------+
           |
     +-----+-----+------------------+
     |            |                  |
     v            v                  v
  +----------+ +----------+   +-----------+
  | licenses | | environ- |   | audit_log |
  |          | | ments    |   |           |
  +----------+ +----------+   +-----------+
  | id (PK)  | | id (PK)  |   | id (PK)  |
  | tenant_id| | tenant_id|   | tenant_id |
  | tier     | | slug     |   | actor_id  |
  | features | | display_ |   | action    |
  | limits   | |   name   |   | resource  |
  | token    | | status   |   | result    |
  | issued_at| | created_ |   | metadata  |
  | expires_ | |   at     |   | created_at|
  |   at     | +-----+----+   +-----------+
  +----------+       |
                +----+----+
                |         |
                v         v
          +----------+ +-----------+
          | api_keys | |   apps    |
          +----------+ +-----------+
          | id (PK)  | | id (PK)   |
          | environ_ | | environ_  |
          |  ment_id | |  ment_id  |
          | key_hash | | slug      |
          | key_     | | display_  |
          |  prefix  | |   name    |
          | status   | | jar_*     |
          | created_ | | exposed_  |
          |   at     | |   port    |
          | revoked_ | | current_  |
          |   at     | |  deploy_id|
          +----------+ | previous_ |
                       |  deploy_id|
                       +-----+-----+
                             |
                             v
                       +-------------+
                       | deployments |
                       +-------------+
                       | id (PK)     |
                       | app_id      |
                       | version     |
                       | image_ref   |
                       | desired_    |
                       |   status    |
                       | observed_   |
                       |   status    |
                       | orchestrator|
                       |  _metadata  |
                       | error_msg   |
                       | deployed_at |
                       | stopped_at  |
                       | created_at  |
                       +-------------+
```

### 4.2 Table Descriptions

**`tenants`** (V001) -- Top-level multi-tenancy entity. Each tenant maps to a
Logto organization via `logto_org_id`. The `tier` column (`LOW` default) drives
license feature gates. The `status` column tracks provisioning state
(`PROVISIONING`, `ACTIVE`, etc.). `settings` is a JSONB bag for tenant-specific
configuration. Stripe columns support future billing integration.

**`licenses`** (V002) -- Per-tenant license tokens with feature flags and usage
limits. The `token` column stores the generated license string. `features` and
`limits` are JSONB columns holding structured capability data. Licenses have
explicit expiry and optional revocation.

**`environments`** (V003) -- Logical deployment environments within a tenant
(e.g., `dev`, `staging`, `production`). Scoped by `(tenant_id, slug)` unique
constraint. Each environment gets its own set of API keys and apps.

**`api_keys`** (V004) -- Per-environment opaque API keys for agent
authentication. The plaintext is never stored -- only `key_hash` (SHA-256 hex,
64 chars) and `key_prefix` (first 12 chars of the `cmk_`-prefixed key, for
identification). Status lifecycle: `ACTIVE` -> `ROTATED` or `REVOKED`.

**`apps`** (V005) -- Customer applications within an environment. Tracks
uploaded JAR metadata (`jar_storage_path`, `jar_checksum`, `jar_size_bytes`,
`jar_original_filename`), optional `exposed_port` for inbound HTTP routing,
and deployment references (`current_deployment_id`, `previous_deployment_id`
for rollback).

**`deployments`** (V006) -- Versioned deployment records for each app. Tracks a
two-state lifecycle: `desired_status` (what the user wants: `RUNNING` or
`STOPPED`) and `observed_status` (what the system sees: `BUILDING`, `STARTING`,
`RUNNING`, `STOPPED`, `FAILED`). `orchestrator_metadata` (JSONB) stores the
Docker container ID. Versioned with `(app_id, version)` unique constraint.

**`audit_log`** (V007) -- Append-only audit trail. Records actor, tenant,
action, resource, environment, result, and optional metadata JSONB. Indexed
by `(tenant_id, created_at)`, `(actor_id, created_at)`, and
`(action, created_at)` for efficient querying.

### 4.3 Audit Actions

Defined in `AuditAction.java`:

| Category      | Actions                                                        |
|---------------|----------------------------------------------------------------|
| Auth          | `AUTH_REGISTER`, `AUTH_LOGIN`, `AUTH_LOGIN_FAILED`, `AUTH_LOGOUT`|
| Tenant        | `TENANT_CREATE`, `TENANT_UPDATE`, `TENANT_SUSPEND`, `TENANT_REACTIVATE`, `TENANT_DELETE` |
| Environment   | `ENVIRONMENT_CREATE`, `ENVIRONMENT_UPDATE`, `ENVIRONMENT_DELETE`|
| App lifecycle | `APP_CREATE`, `APP_DEPLOY`, `APP_PROMOTE`, `APP_ROLLBACK`, `APP_SCALE`, `APP_STOP`, `APP_DELETE` |
| Secrets       | `SECRET_CREATE`, `SECRET_READ`, `SECRET_UPDATE`, `SECRET_DELETE`, `SECRET_ROTATE` |
| Config        | `CONFIG_UPDATE`                                                |
| Team          | `TEAM_INVITE`, `TEAM_REMOVE`, `TEAM_ROLE_CHANGE`              |
| License       | `LICENSE_GENERATE`, `LICENSE_REVOKE`                           |

---

## 5. Deployment Model

### 5.1 Server-Per-Tenant

Each tenant gets a dedicated cameleer-server instance. The SaaS platform
provisions and manages these servers. In the current Docker Compose topology, a
single shared cameleer-server is used for the default tenant. Production
deployments will run per-tenant servers as separate containers or K8s pods.

### 5.2 Customer App Deployment Flow

The deployment lifecycle is managed by `DeploymentService`:

```
  User uploads JAR        Build Docker image       Start container
  via AppController  -->  from base image +   -->  on cameleer network
  (multipart POST)        uploaded JAR              with agent env vars
       |                       |                        |
       v                       v                        v
  apps.jar_storage_path   deployments.image_ref    deployments.orchestrator_metadata
  apps.jar_checksum       deployments.observed_    {"containerId": "..."}
  apps.jar_size_bytes       status = BUILDING
```

**Step-by-step (from `DeploymentService.deploy()`):**

1. **Validate** -- Ensure the app has an uploaded JAR.
2. **Version** -- Increment deployment version via
   `deploymentRepository.findMaxVersionByAppId()`.
3. **Image ref** -- Generate `cameleer-runtime-{env}-{app}:v{n}`.
4. **Persist** -- Save deployment record with `observed_status = BUILDING`.
5. **Audit** -- Log `APP_DEPLOY` action.
6. **Async execution** (`@Async("deploymentExecutor")`):
   a. Build Docker image from base image + customer JAR.
   b. Stop previous container if one exists.
   c. Start new container with environment variables:

   | Variable                    | Value                                  |
   |-----------------------------|----------------------------------------|
   | `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` | API key for agent registration |
   | `CAMELEER_EXPORT_TYPE`      | `HTTP`                                 |
   | `CAMELEER_SERVER_RUNTIME_SERVERURL` | cameleer-server internal URL  |
   | `CAMELEER_APPLICATION_ID`   | App slug                               |
   | `CAMELEER_ENVIRONMENT_ID`   | Environment slug                       |
   | `CAMELEER_DISPLAY_NAME`     | `{tenant}-{env}-{app}`                 |

   d. Apply resource limits (`container-memory-limit`, `container-cpu-shares`).
   e. Configure Traefik labels for inbound routing if `exposed_port` is set:
      `{app}.{env}.{tenant}.{domain}`.
   f. Poll container health for up to `health-check-timeout` seconds.
   g. Update deployment status to `RUNNING` or `FAILED`.
   h. Update app's `current_deployment_id` and `previous_deployment_id`.

### 5.3 Container Resource Limits

Configured via `RuntimeConfig`:

| Property                          | Default     | Description                 |
|-----------------------------------|-------------|-----------------------------|
| `cameleer.runtime.container-memory-limit` | `512m`  | Docker memory limit         |
| `cameleer.runtime.container-cpu-shares`   | `512`   | Docker CPU shares           |
| `cameleer.runtime.max-jar-size`           | `200MB` | Max upload size             |
| `cameleer.runtime.health-check-timeout`   | `60`    | Seconds to wait for healthy |
| `cameleer.runtime.deployment-thread-pool-size` | `4`| Concurrent deployments      |

---

## 6. Agent-Server Protocol

The agent-server protocol is defined in full in
`cameleer/cameleer-common/PROTOCOL.md`. This section summarizes the key
aspects relevant to the SaaS platform.

### 6.1 Agent Registration

1. Agent starts with `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` environment variable (an API key
   generated by the SaaS platform, prefixed with `cmk_`).
2. Agent calls `POST /api/v1/agents/register` on the cameleer-server with the
   API key as a Bearer token.
3. Server validates the key and returns:
   - HMAC JWT access token (short-lived, ~1 hour)
   - HMAC JWT refresh token (longer-lived, ~7 days)
   - Ed25519 public key (for verifying server commands)
4. Agent uses the access token for all subsequent API calls.
5. On access token expiry, agent uses refresh token to obtain a new pair.
6. On refresh token expiry, agent re-registers using the original API key.

### 6.2 Telemetry Ingestion

Agents send telemetry to the server via HTTP POST:
- Route executions with processor-level traces
- Payload captures (configurable granularity with redaction)
- Route graph topology (tree + graph dual representation)
- Metrics and heartbeats

### 6.3 Server-to-Agent Commands (SSE)

The server maintains an SSE (Server-Sent Events) push channel to each agent:
- Configuration changes (engine level, payload capture settings)
- Deep trace requests for specific correlation IDs
- Exchange replay commands
- Per-processor payload capture overrides

**Command signing:** All commands are signed with the server's Ed25519 private
key. The agent verifies signatures using the public key received during
registration. Destructive commands include a nonce for replay protection.

---

## 7. API Overview

All endpoints under `/api/` require authentication unless noted otherwise.
Authentication is via Logto JWT Bearer token. Mutating endpoints additionally
require specific scopes via `@PreAuthorize` (see Section 3.5 for the full
mapping). The Auth column below shows `JWT` for authentication-only endpoints
and the required scope name for scope-gated endpoints.

### 7.1 Platform Configuration

| Method | Path              | Auth     | Description                                |
|--------|-------------------|----------|--------------------------------------------|
| GET    | `/api/config`     | Public   | Frontend config (Logto endpoint, client ID, API resource, scopes) |
| GET    | `/api/health/secured` | JWT  | Auth verification endpoint                 |
| GET    | `/actuator/health`| Public   | Spring Boot health check                   |

`/api/config` response shape:

```json
{
  "logtoEndpoint": "http://localhost:3001",
  "logtoClientId": "<from bootstrap or env>",
  "logtoResource": "https://api.cameleer.local",
  "scopes": [
    "platform:admin", "tenant:manage", "billing:manage", "team:manage",
    "apps:manage", "apps:deploy", "secrets:manage", "observe:read",
    "observe:debug", "settings:manage"
  ]
}
```

The `scopes` array is authoritative -- the frontend reads it during Logto
provider initialization to request the correct API resource scopes during
sign-in. Scopes are defined as a constant list in `PublicConfigController`
rather than being queried from Logto at runtime.

### 7.2 Identity

| Method | Path              | Auth     | Description                                |
|--------|-------------------|----------|--------------------------------------------|
| GET    | `/api/me`         | JWT      | Current user info + tenant memberships     |

`MeController` extracts `organization_id` from the JWT to resolve the tenant.
For non-org-scoped tokens, it falls back to `LogtoManagementClient.getUserOrganizations()`
to enumerate all organizations the user belongs to.

### 7.3 Tenants

| Method | Path                       | Auth                             | Description            |
|--------|----------------------------|----------------------------------|------------------------|
| GET    | `/api/tenants`             | `SCOPE_platform:admin`           | List all tenants       |
| POST   | `/api/tenants`             | `SCOPE_platform:admin`           | Create tenant          |
| GET    | `/api/tenants/{id}`        | JWT                              | Get tenant by UUID     |
| GET    | `/api/tenants/by-slug/{slug}` | JWT                           | Get tenant by slug     |

### 7.4 Environments

| Method | Path                                              | Auth               | Description              |
|--------|----------------------------------------------------|---------------------|--------------------------|
| POST   | `/api/tenants/{tenantId}/environments`             | `apps:manage`       | Create environment       |
| GET    | `/api/tenants/{tenantId}/environments`             | JWT                 | List environments        |
| GET    | `/api/tenants/{tenantId}/environments/{envId}`     | JWT                 | Get environment          |
| PATCH  | `/api/tenants/{tenantId}/environments/{envId}`     | `apps:manage`       | Update display name      |
| DELETE | `/api/tenants/{tenantId}/environments/{envId}`     | `apps:manage`       | Delete environment       |

### 7.5 Apps

| Method | Path                                               | Auth            | Description            |
|--------|----------------------------------------------------|-----------------|------------------------|
| POST   | `/api/environments/{envId}/apps`                   | `apps:manage`   | Create app (multipart: metadata + JAR) |
| GET    | `/api/environments/{envId}/apps`                   | JWT             | List apps              |
| GET    | `/api/environments/{envId}/apps/{appId}`           | JWT             | Get app                |
| PUT    | `/api/environments/{envId}/apps/{appId}/jar`       | `apps:deploy`   | Re-upload JAR          |
| DELETE | `/api/environments/{envId}/apps/{appId}`           | `apps:manage`   | Delete app             |
| PATCH  | `/api/environments/{envId}/apps/{appId}/routing`   | `apps:deploy`   | Set exposed port       |

### 7.6 Deployments

| Method | Path                                              | Auth            | Description              |
|--------|----------------------------------------------------|-----------------|--------------------------|
| POST   | `/api/apps/{appId}/deploy`                         | `apps:deploy`   | Deploy app (async, 202)  |
| POST   | `/api/apps/{appId}/stop`                           | `apps:deploy`   | Stop running deployment  |
| POST   | `/api/apps/{appId}/restart`                        | `apps:deploy`   | Stop + redeploy          |
| GET    | `/api/apps/{appId}/deployments`                    | JWT             | List deployment history  |
| GET    | `/api/apps/{appId}/deployments/{deploymentId}`     | JWT             | Get deployment details   |

### 7.7 Observability

| Method | Path                                             | Auth            | Description               |
|--------|--------------------------------------------------|-----------------|---------------------------|
| GET    | `/api/apps/{appId}/agent-status`                 | `observe:read`  | Agent connectivity status |
| GET    | `/api/apps/{appId}/observability-status`         | `observe:read`  | Observability data status |
| GET    | `/api/apps/{appId}/logs`                         | `observe:read`  | Container logs (query params: `since`, `until`, `limit`, `stream`) |

### 7.8 Licenses

| Method | Path                                           | Auth              | Description              |
|--------|-------------------------------------------------|-------------------|--------------------------|
| POST   | `/api/tenants/{tenantId}/license`              | `billing:manage`  | Generate license (365d)  |
| GET    | `/api/tenants/{tenantId}/license`              | JWT               | Get active license       |

### 7.9 SPA Routing

The `SpaController` forwards all non-API paths to `index.html` for client-side
routing:

```java
@GetMapping(value = {"/", "/login", "/callback", "/environments/**", "/license"})
public String spa() { return "forward:/index.html"; }
```

---

## 8. Security Model

### 8.1 Tenant Isolation

Tenant isolation is enforced by a single Spring `HandlerInterceptor` --
`TenantIsolationInterceptor` -- registered on `/api/**` via `WebConfig`. It
handles both tenant resolution and ownership validation in one place:

**Resolution (every `/api/**` request):**

The interceptor's `preHandle()` reads the JWT's `organization_id` claim,
resolves it to an internal tenant UUID via `TenantService.getByLogtoOrgId()`,
and stores it on `TenantContext` (ThreadLocal). If no organization context is
resolved and the user is not a platform admin, the interceptor returns
**403 Forbidden**.

**Path variable validation (automatic, fail-closed):**

After resolution, the interceptor reads Spring's
`HandlerMapping.URI_TEMPLATE_VARIABLES_ATTRIBUTE` to inspect path variables
defined on the matched handler method. It checks three path variable names:

- `{tenantId}` -- Compared directly against the resolved tenant ID.
- `{environmentId}` -- The environment is loaded and its `tenantId` is compared.
- `{appId}` -- The app -> environment -> tenant chain is followed and compared.

If any path variable is present and the resolved tenant does not own that
resource, the interceptor returns **403 Forbidden**. This is **fail-closed**:
any new endpoint that uses these path variable names is automatically isolated
without requiring manual validation calls.

**Platform admin bypass:**

Users with `SCOPE_platform:admin` bypass all isolation checks. Their
`TenantContext` is left empty (null tenant ID), which downstream services
interpret as unrestricted access.

**Cleanup:**

`TenantContext.clear()` is called in `afterCompletion()` to prevent ThreadLocal
leaks regardless of whether the request succeeded or failed.

**Additional isolation boundaries:**

- Environment and app queries are scoped by tenant through foreign key
  relationships (`environments.tenant_id`).
- Customer app containers run in isolated Docker containers with per-container
  resource limits.

### 8.2 API Key Security

- Keys are generated with 32 bytes of `SecureRandom` entropy, prefixed with
  `cmk_` and Base64url-encoded.
- Only the SHA-256 hash is stored in the database (`key_hash` column, 64 hex
  chars). The `key_prefix` (first 12 chars) is stored for identification in
  UI listings.
- The plaintext key is returned exactly once at creation time and never stored.
- Key lifecycle: `ACTIVE` -> `ROTATED` (old keys remain for grace period) or
  `REVOKED` (immediately invalidated, `revoked_at` timestamp set).
- Validation is via SHA-256 hash comparison:
  `ApiKeyService.validate(plaintext)` -> hash -> lookup by hash and status.

### 8.3 Token Lifetimes

| Token                | Lifetime    | Notes                              |
|----------------------|-------------|------------------------------------|
| Logto access token   | ~1 hour     | Configured in Logto, refreshed by SDK |
| Logto refresh token  | ~14 days    | Used by `@logto/react` for silent refresh |
| Server agent JWT     | ~1 hour     | cameleer-server `CAMELEER_JWT_SECRET` |
| Server refresh token | ~7 days     | Agent re-registers when expired    |

### 8.4 Audit Logging

All state-changing operations are logged to the `audit_log` table via
`AuditService.log()`. Each entry records:

- `actor_id` -- UUID of the user (from JWT subject)
- `tenant_id` -- UUID of the affected tenant
- `action` -- Enum value from `AuditAction`
- `resource` -- Identifier of the affected resource (e.g., app slug)
- `environment` -- Environment slug if applicable
- `result` -- `SUCCESS` or error indicator
- `metadata` -- Optional JSONB for additional context

Audit entries are immutable (append-only, no UPDATE/DELETE operations).

### 8.5 Security Boundaries

- CSRF is disabled (stateless API, Bearer token auth only).
- Sessions are disabled (`SessionCreationPolicy.STATELESS`).
- The Docker socket is mounted read-write on cameleer-saas for container
  management. This is the highest-privilege access in the system.
- Logto's admin endpoint (`:3002`) is not exposed through Traefik.
- ClickHouse has no external port exposure.

---

## 9. Frontend Architecture

### 9.1 Stack

| Technology            | Purpose                                   |
|-----------------------|-------------------------------------------|
| React 19              | UI framework                              |
| Vite                  | Build tool and dev server                 |
| `@logto/react`        | OIDC SDK (auth code flow, token mgmt)     |
| Zustand               | Org/tenant state management (`useOrgStore`)|
| TanStack React Query  | Server state, caching, background refresh |
| React Router (v7)     | Client-side routing                       |
| `@cameleer/design-system` | Shared component library (Gitea npm)  |

### 9.2 Component Hierarchy

```
  <ThemeProvider>
    <ToastProvider>
      <BreadcrumbProvider>
        <GlobalFilterProvider>
          <CommandPaletteProvider>
            <LogtoProvider>
              <TokenSync />         -- Manages org-scoped token provider
              <QueryClientProvider>
                <BrowserRouter>
                  <AppRouter>
                    /login          -- LoginPage
                    /callback       -- CallbackPage (OIDC redirect)
                    <ProtectedRoute>
                      <OrgResolver>   -- Fetches /api/me, populates org store
                        <Layout>
                          /              -- DashboardPage
                          /environments  -- EnvironmentsPage
                          /environments/:envId         -- EnvironmentDetailPage
                          /environments/:envId/apps/:appId -- AppDetailPage
                          /license       -- LicensePage
                          /admin/tenants -- AdminTenantsPage
```

### 9.3 Auth Data Flow

```
  LogtoProvider          -- Configured with 10 API resource scopes from /api/config
      |
      v
  ProtectedRoute         -- Gates on isAuthenticated, redirects to /login
      |
      v
  OrgResolver            -- Effect 1 [me]: populate org store from /api/me
      |                  -- Effect 2 [me, currentOrgId]: fetch org-scoped
      |                  --   access token, decode scopes into Set
      |                  -- Re-runs Effect 2 on org switch (stale scope fix)
      v
  Layout + pages         -- Read from useOrgStore for tenant context
                         -- Read from useAuth() for auth state
                         -- Read scopes for UI gating
```

### 9.4 State Stores

**`useOrgStore`** (Zustand) -- `ui/src/auth/useOrganization.ts`:

| Field            | Type             | Purpose                            |
|------------------|------------------|------------------------------------|
| `currentOrgId`   | `string | null`  | Logto org ID (for token scoping)   |
| `currentTenantId`| `string | null`  | DB UUID (for API calls)            |
| `organizations`  | `OrgInfo[]`      | All orgs the user belongs to       |
| `scopes`         | `Set<string>`    | OAuth2 scopes from access token    |

**`useAuth()`** hook -- `ui/src/auth/useAuth.ts`:

Combines `@logto/react` state (`isAuthenticated`, `isLoading`) with org store
state (`currentTenantId`). Provides `logout` and `signIn` callbacks.

---

## 10. Configuration Reference

### 10.1 cameleer-saas

**Spring / Database:**

| Variable                     | Default                                      | Description                      |
|------------------------------|----------------------------------------------|----------------------------------|
| `SPRING_DATASOURCE_URL`     | `jdbc:postgresql://cameleer-postgres:5432/cameleer_saas` | PostgreSQL JDBC URL           |
| `SPRING_DATASOURCE_USERNAME`| `cameleer`                                   | PostgreSQL user                  |
| `SPRING_DATASOURCE_PASSWORD`| `cameleer_dev`                               | PostgreSQL password              |

**Identity / OIDC:**

| Variable                  | Default    | Description                                |
|---------------------------|------------|--------------------------------------------|
| `CAMELEER_SAAS_IDENTITY_LOGTOENDPOINT` | (empty) | Logto internal URL (Docker-internal) |
| `CAMELEER_SAAS_IDENTITY_LOGTOPUBLICENDPOINT` | (empty) | Logto public URL (browser-accessible) |
| `CAMELEER_SAAS_IDENTITY_M2MCLIENTID` | (empty) | M2M app client ID (from bootstrap)   |
| `CAMELEER_SAAS_IDENTITY_M2MCLIENTSECRET` | (empty) | M2M app client secret (from bootstrap) |
| `CAMELEER_SAAS_IDENTITY_SPACLIENTID` | (empty) | SPA app client ID (fallback; bootstrap preferred) |

**Provisioning** (`cameleer.saas.provisioning.*` / `CAMELEER_SAAS_PROVISIONING_*`):

| Variable                          | Default                            | Description                      |
|-----------------------------------|------------------------------------|----------------------------------|
| `CAMELEER_SAAS_PROVISIONING_SERVERIMAGE` | `gitea.siegeln.net/cameleer/cameleer-server:latest` | Docker image for per-tenant server |
| `CAMELEER_SAAS_PROVISIONING_SERVERUIIMAGE` | `gitea.siegeln.net/cameleer/cameleer-server-ui:latest` | Docker image for per-tenant UI |
| `CAMELEER_SAAS_PROVISIONING_NETWORKNAME` | `cameleer-saas_cameleer` | Shared services Docker network |
| `CAMELEER_SAAS_PROVISIONING_TRAEFIKNETWORK` | `cameleer-traefik` | Traefik Docker network |
| `CAMELEER_SAAS_PROVISIONING_PUBLICHOST` | `localhost` | Public hostname (same as infrastructure `PUBLIC_HOST`) |
| `CAMELEER_SAAS_PROVISIONING_PUBLICPROTOCOL` | `https` | Public protocol (same as infrastructure `PUBLIC_PROTOCOL`) |
| `CAMELEER_SAAS_PROVISIONING_DATASOURCEURL` | `jdbc:postgresql://cameleer-postgres:5432/cameleer` | PostgreSQL URL passed to tenant servers |
| `CAMELEER_SAAS_PROVISIONING_CLICKHOUSEURL` | `jdbc:clickhouse://cameleer-clickhouse:8123/cameleer` | ClickHouse URL passed to tenant servers |

### 10.2 cameleer-server (per-tenant)

Env vars injected into provisioned per-tenant server containers by `DockerTenantProvisioner`. All server properties use the `cameleer.server.*` prefix (env vars: `CAMELEER_SERVER_*`).

| Variable                     | Default / Value                              | Description                      |
|------------------------------|----------------------------------------------|----------------------------------|
| `SPRING_DATASOURCE_URL`     | `jdbc:postgresql://cameleer-postgres:5432/cameleer`  | PostgreSQL JDBC URL              |
| `SPRING_DATASOURCE_USERNAME`| `cameleer`                                   | PostgreSQL user                  |
| `SPRING_DATASOURCE_PASSWORD`| `cameleer_dev`                               | PostgreSQL password              |
| `CAMELEER_SERVER_CLICKHOUSE_URL` | `jdbc:clickhouse://cameleer-clickhouse:8123/cameleer` | ClickHouse JDBC URL         |
| `CAMELEER_SERVER_TENANT_ID` | *(tenant slug)*                              | Tenant identifier for data isolation |
| `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN` | *(generated)*                       | Agent bootstrap token            |
| `CAMELEER_SERVER_SECURITY_JWTSECRET` | *(generated)*                          | JWT signing secret               |
| `CAMELEER_SERVER_SECURITY_OIDC_ISSUERURI` | `${PUBLIC_PROTOCOL}://${PUBLIC_HOST}/oidc` | OIDC issuer for M2M tokens |
| `CAMELEER_SERVER_SECURITY_OIDC_JWKSETURI` | `http://cameleer-logto:3001/oidc/jwks`        | Docker-internal JWK fetch        |
| `CAMELEER_SERVER_SECURITY_OIDC_AUDIENCE` | `https://api.cameleer.local`          | JWT audience validation          |
| `CAMELEER_SERVER_SECURITY_CORSALLOWEDORIGINS` | `${PUBLIC_PROTOCOL}://${PUBLIC_HOST}` | CORS for browser requests |
| `CAMELEER_SERVER_RUNTIME_ENABLED` | `true`                                  | Enable Docker orchestration      |
| `CAMELEER_SERVER_RUNTIME_SERVERURL` | `http://cameleer-server-{slug}:8081`  | Per-tenant server URL            |
| `CAMELEER_SERVER_RUNTIME_ROUTINGDOMAIN` | `${PUBLIC_HOST}`                    | Domain for Traefik routing       |
| `CAMELEER_SERVER_RUNTIME_ROUTINGMODE` | `path`                               | `path` or `subdomain` routing    |
| `CAMELEER_SERVER_RUNTIME_JARSTORAGEPATH` | `/data/jars`                       | JAR file storage directory       |
| `CAMELEER_SERVER_RUNTIME_DOCKERNETWORK` | `cameleer-tenant-{slug}`            | Primary network for app containers |
| `CAMELEER_SERVER_RUNTIME_JARDOCKERVOLUME` | `cameleer-jars-{slug}`            | Docker volume for JAR sharing    |
| `CAMELEER_JWT_SECRET`       | `cameleer-dev-jwt-secret-...`                | HMAC secret for internal JWTs    |
| `CAMELEER_SERVER_TENANT_ID` | `default`                                    | Tenant slug for data isolation   |
| `CAMELEER_SERVER_SECURITY_OIDCISSUERURI` | (empty)                             | Logto issuer for M2M token validation |
| `CAMELEER_SERVER_SECURITY_OIDCAUDIENCE` | (empty)                              | Expected JWT audience            |

### 10.3 logto

| Variable            | Default                  | Description                     |
|---------------------|--------------------------|---------------------------------|
| `LOGTO_PUBLIC_ENDPOINT` | `http://localhost:3001`| Public-facing Logto URL         |
| `LOGTO_ADMIN_ENDPOINT`  | `http://localhost:3002`| Admin console URL (not exposed) |

### 10.4 postgres

| Variable            | Default           | Description                     |
|---------------------|-------------------|---------------------------------|
| `POSTGRES_DB`       | `cameleer_saas`   | Default database name           |
| `POSTGRES_USER`     | `cameleer`        | PostgreSQL superuser            |
| `POSTGRES_PASSWORD` | `cameleer_dev`    | PostgreSQL password             |

### 10.5 logto-bootstrap

| Variable             | Default                    | Description                    |
|----------------------|----------------------------|--------------------------------|
| `SAAS_ADMIN_USER`   | `admin`                    | Platform admin username        |
| `SAAS_ADMIN_PASS`   | `admin`                    | Platform admin password        |
| `TENANT_ADMIN_USER` | `camel`                    | Default tenant admin username  |
| `TENANT_ADMIN_PASS` | `camel`                    | Default tenant admin password  |
| `CAMELEER_SERVER_SECURITY_BOOTSTRAPTOKEN`| `default-bootstrap-token`  | Agent bootstrap token          |

### 10.6 Bootstrap Output

The bootstrap script writes `/data/logto-bootstrap.json` containing:

```json
{
  "spaClientId": "<auto-generated>",
  "m2mClientId": "<auto-generated>",
  "m2mClientSecret": "<auto-generated>",
  "tradAppId": "<auto-generated>",
  "tradAppSecret": "<auto-generated>",
  "apiResourceIndicator": "https://api.cameleer.local",
  "organizationId": "<auto-generated>",
  "tenantName": "Example Tenant",
  "tenantSlug": "default",
  "bootstrapToken": "<from env>",
  "platformAdminUser": "<from env>",
  "tenantAdminUser": "<from env>",
  "oidcIssuerUri": "http://cameleer-logto:3001/oidc",
  "oidcAudience": "https://api.cameleer.local"
}
```

This file is mounted read-only into cameleer-saas via the `bootstrapdata`
volume. `PublicConfigController` reads it to serve SPA client IDs and the API
resource indicator without requiring environment variable configuration. The
controller also includes a `scopes` array (see Section 7.1) so the frontend
can request the correct API resource scopes during Logto sign-in.

---

## Appendix: Key Source Files

| File | Purpose |
|------|---------|
| `docker-compose.yml` | Service topology and configuration |
| `docker/logto-bootstrap.sh` | Idempotent Logto + DB bootstrap |
| `src/.../config/SecurityConfig.java` | Spring Security filter chain |
| `src/.../config/TenantIsolationInterceptor.java` | JWT org_id -> tenant resolution + path variable ownership validation (fail-closed) |
| `src/.../config/WebConfig.java` | Registers `TenantIsolationInterceptor` on `/api/**` |
| `src/.../config/TenantContext.java` | ThreadLocal tenant ID holder |
| `src/.../config/MeController.java` | User identity + tenant endpoint |
| `src/.../config/PublicConfigController.java` | SPA configuration endpoint (Logto config + scopes) |
| `src/.../tenant/TenantController.java` | Tenant CRUD (platform:admin gated) |
| `src/.../environment/EnvironmentController.java` | Environment CRUD |
| `src/.../app/AppController.java` | App CRUD + JAR upload |
| `src/.../deployment/DeploymentService.java` | Async deployment orchestration |
| `src/.../deployment/DeploymentController.java` | Deploy/stop/restart endpoints |
| `src/.../apikey/ApiKeyService.java` | API key generation, rotation, revocation |
| `src/.../identity/LogtoManagementClient.java` | Logto Management API client |
| `src/.../audit/AuditService.java` | Audit log writer |
| `src/.../runtime/RuntimeConfig.java` | Container runtime configuration |
| `ui/src/main.tsx` | React app entry, Logto provider setup |
| `ui/src/router.tsx` | Client-side route definitions |
| `ui/src/auth/OrgResolver.tsx` | Org + scope resolution from JWT |
| `ui/src/auth/useOrganization.ts` | Zustand org/tenant store |
| `ui/src/auth/useAuth.ts` | Auth convenience hook |
| `ui/src/auth/ProtectedRoute.tsx` | Route guard component |