cameleer-server/docs/superpowers/plans/2026-03-17-infrastructure-overview.md

# Infrastructure Overview Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add Database, OpenSearch, and Audit Log admin pages with monitoring, basic management actions, configurable thresholds, and SOC2-compliant audit logging.

**Architecture:** Backend-proxied monitoring — new Spring Boot controllers query PostgreSQL and OpenSearch, aggregate data, and return structured JSON. Central `AuditService` records all admin actions to a database table. Frontend consumes these via React Query hooks with auto-refresh for lightweight endpoints.

**Tech Stack:** Java 17 / Spring Boot 3.4.3 / JdbcTemplate / OpenSearch Java Client / HikariCP MXBean / React 18 / TypeScript / TanStack React Query / React Router / openapi-fetch

**Spec:** `docs/superpowers/specs/2026-03-17-infrastructure-overview-design.md`

---

## File Map

### New Files — Backend (Core Module)

| File | Responsibility |
|------|---------------|
| `core/.../admin/AuditRecord.java` | Audit log record (immutable) |
| `core/.../admin/AuditCategory.java` | Enum: INFRA, AUTH, USER_MGMT, CONFIG |
| `core/.../admin/AuditResult.java` | Enum: SUCCESS, FAILURE |
| `core/.../admin/AuditRepository.java` | Interface: insert + paginated query |
| `core/.../admin/AuditService.java` | Central audit logging service |
| `core/.../admin/ThresholdConfig.java` | Threshold config record |
| `core/.../admin/ThresholdRepository.java` | Interface: find + save |
| `core/.../indexing/SearchIndexerStats.java` | Interface: queue depth, failed count, rate, etc. |

> `core/...` = `cameleer-server-core/src/main/java/com/cameleer/server/core`

### New Files — Backend (App Module)

| File | Responsibility |
|------|---------------|
| `app/.../storage/PostgresAuditRepository.java` | JdbcTemplate impl of AuditRepository |
| `app/.../storage/PostgresThresholdRepository.java` | JdbcTemplate impl of ThresholdRepository |
| `app/.../controller/DatabaseAdminController.java` | Database monitoring + kill query |
| `app/.../controller/OpenSearchAdminController.java` | OpenSearch monitoring + delete index |
| `app/.../controller/ThresholdAdminController.java` | Threshold CRUD |
| `app/.../controller/AuditLogController.java` | Audit log viewer endpoint |
| `app/.../dto/DatabaseStatusResponse.java` | DTO: version, host, schema, connected |
| `app/.../dto/ConnectionPoolResponse.java` | DTO: active, idle, pending, maxWait, maxSize |
| `app/.../dto/TableSizeResponse.java` | DTO: table name, rows, dataSize, indexSize |
| `app/.../dto/ActiveQueryResponse.java` | DTO: pid, duration, state, query |
| `app/.../dto/OpenSearchStatusResponse.java` | DTO: version, host, health, nodes |
| `app/.../dto/PipelineStatsResponse.java` | DTO: queueDepth, failed, debounce, rate, lastIndexed |
| `app/.../dto/IndexInfoResponse.java` | DTO: name, docs, size, health, shards |
| `app/.../dto/IndicesPageResponse.java` | DTO: paginated indices + summary |
| `app/.../dto/PerformanceResponse.java` | DTO: cache rates, latencies, JVM heap |
| `app/.../dto/AuditLogPageResponse.java` | DTO: paginated audit entries |
| `app/.../dto/ThresholdConfigRequest.java` | DTO: threshold save payload |
| `resources/db/migration/V9__admin_thresholds.sql` | Flyway: admin_thresholds table |
| `resources/db/migration/V10__audit_log.sql` | Flyway: audit_log table |

> `app/...` = `cameleer-server-app/src/main/java/com/cameleer/server/app`
> `resources/...` = `cameleer-server-app/src/main/resources`

### New Files — Backend (Tests)

| File | Responsibility |
|------|---------------|
| `test/.../controller/DatabaseAdminControllerIT.java` | Integration test: DB endpoints |
| `test/.../controller/OpenSearchAdminControllerIT.java` | Integration test: OS endpoints |
| `test/.../controller/AuditLogControllerIT.java` | Integration test: audit endpoints |
| `test/.../controller/ThresholdAdminControllerIT.java` | Integration test: threshold endpoints |
| `test/.../admin/AuditServiceTest.java` | Unit test: audit service logic |

> `test/...` = `cameleer-server-app/src/test/java/com/cameleer/server/app`

### New Files — Frontend

| File | Responsibility |
|------|---------------|
| `ui/src/api/queries/admin/database.ts` | React Query hooks: database endpoints |
| `ui/src/api/queries/admin/opensearch.ts` | React Query hooks: OpenSearch endpoints |
| `ui/src/api/queries/admin/thresholds.ts` | React Query hooks: threshold endpoints |
| `ui/src/api/queries/admin/audit.ts` | React Query hooks: audit log endpoint |
| `ui/src/components/admin/StatusBadge.tsx` | Green/yellow/red indicator |
| `ui/src/components/admin/StatusBadge.module.css` | Styles for StatusBadge |
| `ui/src/components/admin/RefreshableCard.tsx` | Card with refresh button |
| `ui/src/components/admin/RefreshableCard.module.css` | Styles for RefreshableCard |
| `ui/src/components/admin/ConfirmDeleteDialog.tsx` | Confirmation dialog requiring name input |
| `ui/src/components/admin/ConfirmDeleteDialog.module.css` | Styles for ConfirmDeleteDialog |
| `ui/src/pages/admin/DatabaseAdminPage.tsx` | Database monitoring page |
| `ui/src/pages/admin/DatabaseAdminPage.module.css` | Styles |
| `ui/src/pages/admin/OpenSearchAdminPage.tsx` | OpenSearch monitoring page |
| `ui/src/pages/admin/OpenSearchAdminPage.module.css` | Styles |
| `ui/src/pages/admin/AuditLogPage.tsx` | Audit log viewer page |
| `ui/src/pages/admin/AuditLogPage.module.css` | Styles |

### Modified Files

| File | Change |
|------|--------|
| `core/.../indexing/SearchIndexer.java` | Add stats counters, implement SearchIndexerStats |
| `app/.../security/SecurityConfig.java` | Add `@EnableMethodSecurity` |
| `app/.../controller/OidcConfigAdminController.java` | Add `@PreAuthorize`, inject AuditService |
| `app/.../controller/UserAdminController.java` | Add `@PreAuthorize`, inject AuditService |
| `app/.../security/UiAuthController.java` | Inject AuditService, log login/logout events |
| `app/.../security/OidcAuthController.java` | Inject AuditService, log OIDC login events |
| `app/.../config/StorageBeanConfig.java` | Wire new beans (AuditService, repositories) |
| `ui/src/router.tsx` | Add admin sub-routes, redirect `/admin` |
| `ui/src/components/layout/AppSidebar.tsx` | Collapsible admin sub-menu |

---

## Task Breakdown

### Task 1: Flyway Migrations

**Files:**
- Create: `cameleer-server-app/src/main/resources/db/migration/V9__admin_thresholds.sql`
- Create: `cameleer-server-app/src/main/resources/db/migration/V10__audit_log.sql`

- [ ] **Step 1: Create V9 migration**

```sql
CREATE TABLE admin_thresholds (
    id          INTEGER PRIMARY KEY DEFAULT 1,
    config      JSONB NOT NULL DEFAULT '{}',
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_by  TEXT NOT NULL,
    CONSTRAINT  single_row CHECK (id = 1)
);
```

- [ ] **Step 2: Create V10 migration**

```sql
CREATE TABLE audit_log (
    id          BIGSERIAL PRIMARY KEY,
    timestamp   TIMESTAMPTZ NOT NULL DEFAULT now(),
    username    TEXT NOT NULL,
    action      TEXT NOT NULL,
    category    TEXT NOT NULL,
    target      TEXT,
    detail      JSONB,
    result      TEXT NOT NULL,
    ip_address  TEXT,
    user_agent  TEXT
);

CREATE INDEX idx_audit_log_timestamp ON audit_log (timestamp DESC);
CREATE INDEX idx_audit_log_username ON audit_log (username);
CREATE INDEX idx_audit_log_category ON audit_log (category);
CREATE INDEX idx_audit_log_action ON audit_log (action);
CREATE INDEX idx_audit_log_target ON audit_log (target);
```

- [ ] **Step 3: Verify migrations compile**

Run: `cd cameleer-server && mvn clean compile -pl cameleer-server-app`
Expected: BUILD SUCCESS

- [ ] **Step 4: Commit**

```bash
git add cameleer-server-app/src/main/resources/db/migration/V9__admin_thresholds.sql \
       cameleer-server-app/src/main/resources/db/migration/V10__audit_log.sql
git commit -m "feat: add Flyway V9 (thresholds) and V10 (audit_log) migrations"
```

---

### Task 2: Core Module — Audit Domain Model + Repository Interface

**Files:**
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/AuditCategory.java`
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/AuditResult.java`
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/AuditRecord.java`
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/AuditRepository.java`
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/AuditService.java`

- [ ] **Step 1: Create AuditCategory enum**

```java
package com.cameleer.server.core.admin;

public enum AuditCategory {
    INFRA, AUTH, USER_MGMT, CONFIG
}
```

- [ ] **Step 2: Create AuditResult enum**

```java
package com.cameleer.server.core.admin;

public enum AuditResult {
    SUCCESS, FAILURE
}
```

- [ ] **Step 3: Create AuditRecord**

```java
package com.cameleer.server.core.admin;

import java.time.Instant;
import java.util.Map;

public record AuditRecord(
        long id,
        Instant timestamp,
        String username,
        String action,
        AuditCategory category,
        String target,
        Map<String, Object> detail,
        AuditResult result,
        String ipAddress,
        String userAgent
) {
    /** Factory for creating new records (id and timestamp assigned by DB) */
    public static AuditRecord create(String username, String action, AuditCategory category,
                                      String target, Map<String, Object> detail, AuditResult result,
                                      String ipAddress, String userAgent) {
        return new AuditRecord(0, null, username, action, category, target, detail, result, ipAddress, userAgent);
    }
}
```

- [ ] **Step 4: Create AuditRepository interface**

```java
package com.cameleer.server.core.admin;

import java.time.Instant;
import java.util.List;

public interface AuditRepository {

    void insert(AuditRecord record);

    record AuditQuery(
            String username,
            AuditCategory category,
            String search,
            Instant from,
            Instant to,
            String sort,
            String order,
            int page,
            int size
    ) {}

    record AuditPage(List<AuditRecord> items, long totalCount) {}

    AuditPage find(AuditQuery query);
}
```

- [ ] **Step 5: Create AuditService**

The service lives in core so it can be referenced by any controller. It depends on `AuditRepository` (interface) and uses SLF4J for dual logging. It extracts username/IP/user-agent from Spring Security context and servlet request.

```java
package com.cameleer.server.core.admin;

import jakarta.servlet.http.HttpServletRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;

import java.util.Map;

public class AuditService {
    private static final Logger log = LoggerFactory.getLogger(AuditService.class);
    private final AuditRepository repository;

    public AuditService(AuditRepository repository) {
        this.repository = repository;
    }

    /** Log an action using the current SecurityContext for username */
    public void log(String action, AuditCategory category, String target,
                    Map<String, Object> detail, AuditResult result,
                    HttpServletRequest request) {
        String username = extractUsername();
        log(username, action, category, target, detail, result, request);
    }

    /** Log an action with explicit username (for pre-auth contexts like login) */
    public void log(String username, String action, AuditCategory category, String target,
                    Map<String, Object> detail, AuditResult result,
                    HttpServletRequest request) {
        String ip = request != null ? request.getRemoteAddr() : null;
        String userAgent = request != null ? request.getHeader("User-Agent") : null;
        AuditRecord record = AuditRecord.create(username, action, category, target, detail, result, ip, userAgent);

        repository.insert(record);

        log.info("AUDIT: user={} action={} category={} target={} result={}",
                username, action, category, target, result);
    }

    private String extractUsername() {
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        if (auth != null && auth.getName() != null) {
            String name = auth.getName();
            return name.startsWith("user:") ? name.substring(5) : name;
        }
        return "unknown";
    }
}
```

**Note:** This class uses `jakarta.servlet` and `org.springframework.security` — the core POM must have these as `provided` scope dependencies if not already present. Check `cameleer-server-core/pom.xml` and add if needed:
```xml
<dependency>
    <groupId>jakarta.servlet</groupId>
    <artifactId>jakarta.servlet-api</artifactId>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-core</artifactId>
    <scope>provided</scope>
</dependency>
```

- [ ] **Step 6: Write AuditService unit test**

```java
class AuditServiceTest {
    private AuditRepository mockRepository;
    private AuditService auditService;

    @BeforeEach
    void setUp() {
        mockRepository = mock(AuditRepository.class);
        auditService = new AuditService(mockRepository);
    }

    @Test
    void log_withExplicitUsername_insertsRecordWithCorrectFields() {
        var request = mock(HttpServletRequest.class);
        when(request.getRemoteAddr()).thenReturn("192.168.1.1");
        when(request.getHeader("User-Agent")).thenReturn("Mozilla/5.0");

        auditService.log("admin", "kill_query", AuditCategory.INFRA, "PID 42",
                Map.of("query", "SELECT 1"), AuditResult.SUCCESS, request);

        var captor = ArgumentCaptor.forClass(AuditRecord.class);
        verify(mockRepository).insert(captor.capture());
        var record = captor.getValue();
        assertEquals("admin", record.username());
        assertEquals("kill_query", record.action());
        assertEquals(AuditCategory.INFRA, record.category());
        assertEquals("PID 42", record.target());
        assertEquals("192.168.1.1", record.ipAddress());
        assertEquals("Mozilla/5.0", record.userAgent());
    }

    @Test
    void log_withNullRequest_handlesGracefully() {
        auditService.log("admin", "test", AuditCategory.CONFIG, null, null, AuditResult.SUCCESS, null);
        verify(mockRepository).insert(any(AuditRecord.class));
    }
}
```

- [ ] **Step 7: Verify core module compiles and test passes**

Run: `mvn clean compile -pl cameleer-server-core`
Run: `mvn test -pl cameleer-server-app -Dtest=AuditServiceTest`
Expected: BUILD SUCCESS, tests PASS

- [ ] **Step 8: Commit**

```bash
git add cameleer-server-core/ cameleer-server-app/src/test/
git commit -m "feat: add audit domain model, repository interface, AuditService, and unit test"
```

---

### Task 3: Core Module — Threshold Model + Repository Interface

**Files:**
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/ThresholdConfig.java`
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/admin/ThresholdRepository.java`

- [ ] **Step 1: Create ThresholdConfig record**

```java
package com.cameleer.server.core.admin;

public record ThresholdConfig(
        DatabaseThresholds database,
        OpenSearchThresholds opensearch
) {
    public record DatabaseThresholds(
            int connectionPoolWarning,
            int connectionPoolCritical,
            double queryDurationWarning,
            double queryDurationCritical
    ) {
        public static DatabaseThresholds defaults() {
            return new DatabaseThresholds(80, 95, 1.0, 10.0);
        }
    }

    public record OpenSearchThresholds(
            String clusterHealthWarning,
            String clusterHealthCritical,
            int queueDepthWarning,
            int queueDepthCritical,
            int jvmHeapWarning,
            int jvmHeapCritical,
            int failedDocsWarning,
            int failedDocsCritical
    ) {
        public static OpenSearchThresholds defaults() {
            return new OpenSearchThresholds("YELLOW", "RED", 100, 500, 75, 90, 1, 10);
        }
    }

    public static ThresholdConfig defaults() {
        return new ThresholdConfig(DatabaseThresholds.defaults(), OpenSearchThresholds.defaults());
    }
}
```

- [ ] **Step 2: Create ThresholdRepository interface**

```java
package com.cameleer.server.core.admin;

import java.util.Optional;

public interface ThresholdRepository {
    Optional<ThresholdConfig> find();
    void save(ThresholdConfig config, String updatedBy);
}
```

- [ ] **Step 3: Compile and commit**

Run: `mvn clean compile -pl cameleer-server-core`

```bash
git add cameleer-server-core/
git commit -m "feat: add ThresholdConfig model and ThresholdRepository interface"
```

---

### Task 4: Core Module — SearchIndexerStats Interface + Instrumentation

**Files:**
- Create: `cameleer-server-core/src/main/java/com/cameleer/server/core/indexing/SearchIndexerStats.java`
- Modify: `cameleer-server-core/src/main/java/com/cameleer/server/core/indexing/SearchIndexer.java`

- [ ] **Step 1: Create SearchIndexerStats interface**

```java
package com.cameleer.server.core.indexing;

import java.time.Instant;

public interface SearchIndexerStats {
    int getQueueDepth();
    int getMaxQueueSize();
    long getFailedCount();
    long getIndexedCount();
    Instant getLastIndexedAt();
    long getDebounceMs();
    /** Approximate indexing rate in docs/sec over last measurement window */
    double getIndexingRate();
}
```

- [ ] **Step 2: Add stats counters to SearchIndexer**

Modify `SearchIndexer.java`:
- Add fields: `AtomicLong failedCount`, `AtomicLong indexedCount`, `volatile Instant lastIndexedAt`
- Add rate tracking: `AtomicLong rateWindowStart`, `AtomicLong rateWindowCount`
- Implement `SearchIndexerStats`
- Increment counters in `indexExecution()`: `indexedCount.incrementAndGet()` on success, `failedCount.incrementAndGet()` on catch
- Set `lastIndexedAt = Instant.now()` after successful indexing
- Return `pending.size()` for queue depth, `queueCapacity` for max size

Key changes to `indexExecution()` method:
```java
private void indexExecution(String executionId) {
    pending.remove(executionId);
    try {
        ExecutionRecord exec = executionStore.findById(executionId).orElse(null);
        if (exec == null) return;
        // ... existing indexing logic ...
        indexedCount.incrementAndGet();
        lastIndexedAt = Instant.now();
        updateRate();
    } catch (Exception e) {
        failedCount.incrementAndGet();
        log.error("Failed to index execution {}", executionId, e);
    }
}
```

Rate calculation approach: track count delta between 15-second measurement windows.
```java
private void updateRate() {
    long now = System.currentTimeMillis();
    long windowStart = rateWindowStartMs.get();
    long count = rateWindowCount.incrementAndGet();
    long elapsed = now - windowStart;
    if (elapsed >= 15_000) { // 15-second window
        lastRate = count / (elapsed / 1000.0);
        rateWindowStartMs.set(now);
        rateWindowCount.set(0);
    }
}
```

- [ ] **Step 3: Compile and commit**

Run: `mvn clean compile -pl cameleer-server-core`

```bash
git add cameleer-server-core/
git commit -m "feat: add SearchIndexerStats interface and instrument SearchIndexer"
```

---

### Task 5: App Module — Postgres Repository Implementations

**Files:**
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/PostgresAuditRepository.java`
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/storage/PostgresThresholdRepository.java`

- [ ] **Step 1: Create PostgresAuditRepository**

Follow the same `@Repository` + `JdbcTemplate` pattern used by `PostgresUserRepository` and `PostgresOidcConfigRepository`.

```java
@Repository
public class PostgresAuditRepository implements AuditRepository {
    private final JdbcTemplate jdbc;
    private final ObjectMapper objectMapper;

    public PostgresAuditRepository(JdbcTemplate jdbc, ObjectMapper objectMapper) {
        this.jdbc = jdbc;
        this.objectMapper = objectMapper;
    }

    @Override
    public void insert(AuditRecord record) {
        jdbc.update("""
                INSERT INTO audit_log (username, action, category, target, detail, result, ip_address, user_agent)
                VALUES (?, ?, ?, ?, ?::jsonb, ?, ?, ?)
                """,
                record.username(), record.action(), record.category().name(),
                record.target(), toJson(record.detail()), record.result().name(),
                record.ipAddress(), record.userAgent());
    }

    @Override
    public AuditPage find(AuditQuery query) {
        int effectiveSize = Math.min(query.size(), 100);
        var params = new ArrayList<>();
        var conditions = new ArrayList<String>();

        // Always filter by timestamp range
        conditions.add("timestamp >= ?");
        params.add(Timestamp.from(query.from()));
        conditions.add("timestamp <= ?");
        params.add(Timestamp.from(query.to()));

        if (query.username() != null && !query.username().isBlank()) {
            conditions.add("username = ?");
            params.add(query.username());
        }
        if (query.category() != null) {
            conditions.add("category = ?");
            params.add(query.category().name());
        }
        if (query.search() != null && !query.search().isBlank()) {
            conditions.add("(action ILIKE ? OR target ILIKE ?)");
            String pattern = "%" + query.search() + "%";
            params.add(pattern);
            params.add(pattern);
        }

        String where = "WHERE " + String.join(" AND ", conditions);

        // Validate sort column against allowlist
        String sortCol = switch (query.sort()) {
            case "username" -> "username";
            case "action" -> "action";
            case "category" -> "category";
            default -> "timestamp";
        };
        String orderDir = "asc".equalsIgnoreCase(query.order()) ? "ASC" : "DESC";

        // Count query
        long total = jdbc.queryForObject(
                "SELECT COUNT(*) FROM audit_log " + where, Long.class, params.toArray());

        // Data query with pagination
        String sql = "SELECT * FROM audit_log " + where
                + " ORDER BY " + sortCol + " " + orderDir
                + " LIMIT ? OFFSET ?";
        params.add(effectiveSize);
        params.add(query.page() * effectiveSize);

        var items = jdbc.query(sql, this::mapRow, params.toArray());
        return new AuditPage(items, total);
    }

    private String toJson(Map<String, Object> map) {
        if (map == null || map.isEmpty()) return null;
        try { return objectMapper.writeValueAsString(map); }
        catch (Exception e) { return "{}"; }
    }

    @SuppressWarnings("unchecked")
    private AuditRecord mapRow(ResultSet rs, int rowNum) throws SQLException {
        Map<String, Object> detail = null;
        String detailJson = rs.getString("detail");
        if (detailJson != null) {
            try { detail = objectMapper.readValue(detailJson, Map.class); }
            catch (Exception e) { detail = Map.of("_raw", detailJson); }
        }
        Timestamp ts = rs.getTimestamp("timestamp");
        return new AuditRecord(
                rs.getLong("id"),
                ts != null ? ts.toInstant() : null,
                rs.getString("username"),
                rs.getString("action"),
                AuditCategory.valueOf(rs.getString("category")),
                rs.getString("target"),
                detail,
                AuditResult.valueOf(rs.getString("result")),
                rs.getString("ip_address"),
                rs.getString("user_agent"));
    }
}
```

- [ ] **Step 2: Create PostgresThresholdRepository**

```java
@Repository
public class PostgresThresholdRepository implements ThresholdRepository {
    private final JdbcTemplate jdbc;
    private final ObjectMapper objectMapper;

    public PostgresThresholdRepository(JdbcTemplate jdbc, ObjectMapper objectMapper) {
        this.jdbc = jdbc;
        this.objectMapper = objectMapper;
    }

    @Override
    public Optional<ThresholdConfig> find() {
        var results = jdbc.query(
                "SELECT config FROM admin_thresholds WHERE id = 1",
                (rs, rowNum) -> {
                    try {
                        return objectMapper.readValue(rs.getString("config"), ThresholdConfig.class);
                    } catch (Exception e) {
                        return ThresholdConfig.defaults();
                    }
                });
        return results.isEmpty() ? Optional.empty() : Optional.of(results.get(0));
    }

    @Override
    public void save(ThresholdConfig config, String updatedBy) {
        String json;
        try { json = objectMapper.writeValueAsString(config); }
        catch (Exception e) { throw new RuntimeException("Failed to serialize thresholds", e); }

        jdbc.update("""
                INSERT INTO admin_thresholds (id, config, updated_by, updated_at)
                VALUES (1, ?::jsonb, ?, now())
                ON CONFLICT (id) DO UPDATE SET config = ?::jsonb, updated_by = ?, updated_at = now()
                """, json, updatedBy, json, updatedBy);
    }
}
```

- [ ] **Step 3: Compile and commit**

Run: `mvn clean compile -pl cameleer-server-app`

```bash
git add cameleer-server-app/src/main/java/com/cameleer/server/app/storage/
git commit -m "feat: add Postgres implementations for AuditRepository and ThresholdRepository"
```

---

### Task 6: App Module — Bean Wiring + Security Retrofit

**Files:**
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/config/StorageBeanConfig.java`
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/security/SecurityConfig.java`
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/OidcConfigAdminController.java`
- Modify: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/UserAdminController.java`

- [ ] **Step 1: Wire AuditService bean in StorageBeanConfig**

Add to `StorageBeanConfig.java`:
```java
@Bean
public AuditService auditService(AuditRepository auditRepository) {
    return new AuditService(auditRepository);
}
```

- [ ] **Step 2: Add @EnableMethodSecurity to SecurityConfig**

```java
@Configuration
@EnableWebSecurity
@EnableMethodSecurity  // <-- add this
public class SecurityConfig {
```

- [ ] **Step 3: Add @PreAuthorize to existing admin controllers**

Add `@PreAuthorize("hasRole('ADMIN')")` to both `OidcConfigAdminController` and `UserAdminController` class-level annotations.

- [ ] **Step 4: Inject AuditService into OidcConfigAdminController**

Add `AuditService` as constructor parameter. Add audit logging calls:
- `save()` → `auditService.log("update_oidc", AuditCategory.CONFIG, "oidc", Map.of(...), AuditResult.SUCCESS, request)`
- `delete()` → `auditService.log("delete_oidc", AuditCategory.CONFIG, "oidc", null, AuditResult.SUCCESS, request)`
- `testConnection()` → `auditService.log("test_oidc", AuditCategory.CONFIG, "oidc", Map.of("result", ...), result, request)`

Add `HttpServletRequest request` parameter to each endpoint method.

- [ ] **Step 5: Inject AuditService into UserAdminController**

Same pattern. Log:
- `updateRoles()` → `auditService.log("update_roles", AuditCategory.USER_MGMT, userId, Map.of("roles", roles), AuditResult.SUCCESS, request)`
- `deleteUser()` → `auditService.log("delete_user", AuditCategory.USER_MGMT, userId, null, AuditResult.SUCCESS, request)`

- [ ] **Step 6: Inject AuditService into UiAuthController**

The file is `UiAuthController.java` (not `AuthController`). Log login success/failure and logout. Use the explicit-username overload for login (SecurityContext not yet populated):
- Login success: `auditService.log(username, "login", AuditCategory.AUTH, null, null, AuditResult.SUCCESS, request)`
- Login failure: `auditService.log(username, "login_failed", AuditCategory.AUTH, null, Map.of("reason", reason), AuditResult.FAILURE, request)`

- [ ] **Step 7: Inject AuditService into OidcAuthController**

Log OIDC login:
- OIDC callback success: `auditService.log(username, "login_oidc", AuditCategory.AUTH, null, Map.of("provider", issuerUri), AuditResult.SUCCESS, request)`

- [ ] **Step 8: Compile and commit**

Run: `mvn clean compile`

```bash
git add cameleer-server-app/
git commit -m "feat: wire AuditService, enable method security, retrofit audit logging into existing controllers"
```

---

### Task 7: App Module — Response DTOs

**Files:**
- Create all DTO files listed in the File Map under `app/.../dto/`

- [ ] **Step 1: Create database-related DTOs**

All as Java records with `@Schema` annotations for OpenAPI:

```java
// DatabaseStatusResponse
public record DatabaseStatusResponse(
    @Schema(description = "Whether the database is reachable") boolean connected,
    @Schema(description = "PostgreSQL version string") String version,
    @Schema(description = "JDBC host") String host,
    @Schema(description = "Current schema") String schema,
    @Schema(description = "Whether TimescaleDB extension is present") boolean timescaleDb
) {}

// ConnectionPoolResponse
public record ConnectionPoolResponse(
    int activeConnections, int idleConnections, int pendingThreads,
    long maxWaitMs, int maxPoolSize
) {}

// TableSizeResponse
public record TableSizeResponse(String tableName, long rowCount, String dataSize, String indexSize, long dataSizeBytes, long indexSizeBytes) {}

// ActiveQueryResponse
public record ActiveQueryResponse(int pid, double durationSeconds, String state, String query) {}
```

- [ ] **Step 2: Create OpenSearch-related DTOs**

```java
// OpenSearchStatusResponse
public record OpenSearchStatusResponse(
    boolean reachable, String clusterHealth, String version, int nodeCount, String host
) {}

// PipelineStatsResponse
public record PipelineStatsResponse(
    int queueDepth, int maxQueueSize, long failedCount, long indexedCount,
    long debounceMs, double indexingRate, Instant lastIndexedAt
) {}

// IndexInfoResponse
public record IndexInfoResponse(String name, long docCount, String size, long sizeBytes, String health, int primaryShards, int replicaShards) {}

// IndicesPageResponse
public record IndicesPageResponse(
    List<IndexInfoResponse> indices, long totalIndices, long totalDocs,
    String totalSize, int page, int pageSize, int totalPages
) {}

// PerformanceResponse
public record PerformanceResponse(
    double queryCacheHitRate, double requestCacheHitRate,
    double searchLatencyMs, double indexingLatencyMs,
    long jvmHeapUsedBytes, long jvmHeapMaxBytes
) {}
```

- [ ] **Step 3: Create audit + threshold DTOs**

```java
// AuditLogPageResponse
public record AuditLogPageResponse(
    List<AuditRecord> items, long totalCount, int page, int pageSize, int totalPages
) {}

// ThresholdConfigRequest
public record ThresholdConfigRequest(
    @Valid DatabaseThresholdsRequest database,
    @Valid OpenSearchThresholdsRequest opensearch
) {
    public record DatabaseThresholdsRequest(
        @Min(0) @Max(100) int connectionPoolWarning,
        @Min(0) @Max(100) int connectionPoolCritical,
        @Positive double queryDurationWarning,
        @Positive double queryDurationCritical
    ) {}

    public record OpenSearchThresholdsRequest(
        @NotBlank String clusterHealthWarning,
        @NotBlank String clusterHealthCritical,
        @Min(0) int queueDepthWarning,
        @Min(0) int queueDepthCritical,
        @Min(0) @Max(100) int jvmHeapWarning,
        @Min(0) @Max(100) int jvmHeapCritical,
        @Min(0) int failedDocsWarning,
        @Min(0) int failedDocsCritical
    ) {}

    /** Convert to domain model after validation */
    public ThresholdConfig toConfig() {
        return new ThresholdConfig(
            new ThresholdConfig.DatabaseThresholds(
                database.connectionPoolWarning, database.connectionPoolCritical,
                database.queryDurationWarning, database.queryDurationCritical),
            new ThresholdConfig.OpenSearchThresholds(
                opensearch.clusterHealthWarning, opensearch.clusterHealthCritical,
                opensearch.queueDepthWarning, opensearch.queueDepthCritical,
                opensearch.jvmHeapWarning, opensearch.jvmHeapCritical,
                opensearch.failedDocsWarning, opensearch.failedDocsCritical));
    }

    /** Custom validation: warning <= critical for all pairs */
    public List<String> validate() {
        var errors = new ArrayList<String>();
        if (database.connectionPoolWarning > database.connectionPoolCritical)
            errors.add("database.connectionPoolWarning must be <= connectionPoolCritical");
        if (database.queryDurationWarning > database.queryDurationCritical)
            errors.add("database.queryDurationWarning must be <= queryDurationCritical");
        if (opensearch.queueDepthWarning > opensearch.queueDepthCritical)
            errors.add("opensearch.queueDepthWarning must be <= queueDepthCritical");
        if (opensearch.jvmHeapWarning > opensearch.jvmHeapCritical)
            errors.add("opensearch.jvmHeapWarning must be <= jvmHeapCritical");
        if (opensearch.failedDocsWarning > opensearch.failedDocsCritical)
            errors.add("opensearch.failedDocsWarning must be <= failedDocsCritical");
        // Cluster health severity: GREEN < YELLOW < RED
        var severity = Map.of("GREEN", 0, "YELLOW", 1, "RED", 2);
        int warnSev = severity.getOrDefault(opensearch.clusterHealthWarning.toUpperCase(), -1);
        int critSev = severity.getOrDefault(opensearch.clusterHealthCritical.toUpperCase(), -1);
        if (warnSev < 0) errors.add("opensearch.clusterHealthWarning must be GREEN, YELLOW, or RED");
        if (critSev < 0) errors.add("opensearch.clusterHealthCritical must be GREEN, YELLOW, or RED");
        if (warnSev >= 0 && critSev >= 0 && warnSev > critSev)
            errors.add("opensearch.clusterHealthWarning must be less severe than clusterHealthCritical");
        return errors;
    }
}
```

- [ ] **Step 4: Compile and commit**

Run: `mvn clean compile -pl cameleer-server-app`

```bash
git add cameleer-server-app/src/main/java/com/cameleer/server/app/dto/
git commit -m "feat: add response/request DTOs for admin infrastructure endpoints"
```

---

### Task 8: App Module — DatabaseAdminController

**Files:**
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/DatabaseAdminController.java`
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/controller/DatabaseAdminControllerIT.java`

- [ ] **Step 1: Write integration test**

Extend `AbstractPostgresIT`. Test all 5 database endpoints:
- `GET /api/v1/admin/database/status` → 200, contains version, connected=true
- `GET /api/v1/admin/database/pool` → 200, contains activeConnections >= 0
- `GET /api/v1/admin/database/tables` → 200, contains list with at least "users" table
- `GET /api/v1/admin/database/queries` → 200, returns list
- `POST /api/v1/admin/database/queries/99999/kill` → 404 (non-existent PID)

Use `TestRestTemplate` with admin JWT for authentication. Create a helper that generates an admin JWT using `JwtService`.

- [ ] **Step 2: Run test to verify it fails**

Run: `mvn test -pl cameleer-server-app -Dtest=DatabaseAdminControllerIT`
Expected: FAIL — controller class does not exist

- [ ] **Step 3: Implement DatabaseAdminController**

```java
@RestController
@RequestMapping("/api/v1/admin/database")
@PreAuthorize("hasRole('ADMIN')")
@Tag(name = "Database Admin", description = "Database monitoring and management (ADMIN only)")
public class DatabaseAdminController {

    private final JdbcTemplate jdbc;
    private final DataSource dataSource;
    private final AuditService auditService;

    // Constructor injection

    @GetMapping("/status")
    @Operation(summary = "Get database connection status and version")
    public ResponseEntity<DatabaseStatusResponse> getStatus() {
        try {
            String version = jdbc.queryForObject("SELECT version()", String.class);
            boolean timescaleDb = Boolean.TRUE.equals(
                jdbc.queryForObject("SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'timescaledb')", Boolean.class));
            String schema = jdbc.queryForObject("SELECT current_schema()", String.class);
            // Extract host from DataSource URL
            String host = extractHost(dataSource);
            return ResponseEntity.ok(new DatabaseStatusResponse(true, version, host, schema, timescaleDb));
        } catch (Exception e) {
            return ResponseEntity.ok(new DatabaseStatusResponse(false, null, null, null, false));
        }
    }

    @GetMapping("/pool")
    @Operation(summary = "Get HikariCP connection pool stats")
    public ResponseEntity<ConnectionPoolResponse> getPool() {
        HikariDataSource hds = (HikariDataSource) dataSource;
        HikariPoolMXBean pool = hds.getHikariPoolMXBean();
        return ResponseEntity.ok(new ConnectionPoolResponse(
            pool.getActiveConnections(), pool.getIdleConnections(),
            pool.getThreadsAwaitingConnection(), hds.getConnectionTimeout(),
            hds.getMaximumPoolSize()));
    }

    @GetMapping("/tables")
    @Operation(summary = "Get table sizes and row counts")
    public ResponseEntity<List<TableSizeResponse>> getTables() {
        // Query pg_stat_user_tables + pg_total_relation_size + pg_indexes_size
        var tables = jdbc.query("""
            SELECT schemaname || '.' || relname AS table_name,
                   n_live_tup AS row_count,
                   pg_size_pretty(pg_total_relation_size(relid)) AS data_size,
                   pg_total_relation_size(relid) AS data_size_bytes,
                   pg_size_pretty(pg_indexes_size(relid)) AS index_size,
                   pg_indexes_size(relid) AS index_size_bytes
            FROM pg_stat_user_tables
            ORDER BY pg_total_relation_size(relid) DESC
            """, (rs, row) -> new TableSizeResponse(
                rs.getString("table_name"), rs.getLong("row_count"),
                rs.getString("data_size"), rs.getString("index_size"),
                rs.getLong("data_size_bytes"), rs.getLong("index_size_bytes")));
        return ResponseEntity.ok(tables);
    }

    @GetMapping("/queries")
    @Operation(summary = "Get active queries")
    public ResponseEntity<List<ActiveQueryResponse>> getQueries() {
        var queries = jdbc.query("""
            SELECT pid, EXTRACT(EPOCH FROM (now() - query_start)) AS duration_seconds,
                   state, query
            FROM pg_stat_activity
            WHERE state != 'idle' AND pid != pg_backend_pid()
            ORDER BY query_start ASC
            """, (rs, row) -> new ActiveQueryResponse(
                rs.getInt("pid"), rs.getDouble("duration_seconds"),
                rs.getString("state"), rs.getString("query")));
        return ResponseEntity.ok(queries);
    }

    @PostMapping("/queries/{pid}/kill")
    @Operation(summary = "Terminate a query by PID")
    public ResponseEntity<Void> killQuery(@PathVariable int pid, HttpServletRequest request) {
        // Check PID exists first
        var exists = jdbc.queryForObject(
            "SELECT EXISTS(SELECT 1 FROM pg_stat_activity WHERE pid = ? AND pid != pg_backend_pid())",
            Boolean.class, pid);
        if (!Boolean.TRUE.equals(exists)) {
            throw new ResponseStatusException(HttpStatus.NOT_FOUND, "No active query with PID " + pid);
        }
        jdbc.queryForObject("SELECT pg_terminate_backend(?)", Boolean.class, pid);
        auditService.log("kill_query", AuditCategory.INFRA, "PID " + pid, null, AuditResult.SUCCESS, request);
        return ResponseEntity.ok().build();
    }
}
```

- [ ] **Step 4: Run integration test**

Run: `mvn test -pl cameleer-server-app -Dtest=DatabaseAdminControllerIT`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add cameleer-server-app/
git commit -m "feat: add DatabaseAdminController with status, pool, tables, queries, and kill endpoints"
```

---

### Task 9: App Module — OpenSearchAdminController

**Files:**
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/OpenSearchAdminController.java`
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/controller/OpenSearchAdminControllerIT.java`

- [ ] **Step 1: Write integration test**

Extend `AbstractPostgresIT` (which starts both PG and OpenSearch containers). Test:
- `GET /api/v1/admin/opensearch/status` → 200, reachable=true, clusterHealth in [green, yellow]
- `GET /api/v1/admin/opensearch/pipeline` → 200, contains queueDepth >= 0
- `GET /api/v1/admin/opensearch/indices` → 200, returns paginated response
- `GET /api/v1/admin/opensearch/performance` → 200, contains jvmHeapMaxBytes > 0
- `DELETE /api/v1/admin/opensearch/indices/nonexistent` → 404

- [ ] **Step 2: Run test to verify it fails**

Run: `mvn test -pl cameleer-server-app -Dtest=OpenSearchAdminControllerIT`
Expected: FAIL

- [ ] **Step 3: Implement OpenSearchAdminController**

```java
@RestController
@RequestMapping("/api/v1/admin/opensearch")
@PreAuthorize("hasRole('ADMIN')")
@Tag(name = "OpenSearch Admin", description = "OpenSearch monitoring and management (ADMIN only)")
public class OpenSearchAdminController {

    private final OpenSearchClient client;
    private final RestClient restClient;
    private final SearchIndexerStats indexerStats;
    private final AuditService auditService;
    @Value("${opensearch.url:http://localhost:9200}")
    private String opensearchUrl;

    // Constructor injection

    @GetMapping("/status")
    public ResponseEntity<OpenSearchStatusResponse> getStatus() {
        try {
            var health = client.cluster().health();
            var info = client.info();
            return ResponseEntity.ok(new OpenSearchStatusResponse(
                true, health.status().jsonValue(), info.version().number(),
                health.numberOfNodes(), opensearchUrl));
        } catch (Exception e) {
            return ResponseEntity.ok(new OpenSearchStatusResponse(
                false, "UNREACHABLE", null, 0, opensearchUrl));
        }
    }

    @GetMapping("/pipeline")
    public ResponseEntity<PipelineStatsResponse> getPipeline() {
        return ResponseEntity.ok(new PipelineStatsResponse(
            indexerStats.getQueueDepth(), indexerStats.getMaxQueueSize(),
            indexerStats.getFailedCount(), indexerStats.getIndexedCount(),
            indexerStats.getDebounceMs(), indexerStats.getIndexingRate(),
            indexerStats.getLastIndexedAt()));
    }

    @GetMapping("/indices")
    public ResponseEntity<IndicesPageResponse> getIndices(
            @RequestParam(defaultValue = "") String search,
            @RequestParam(defaultValue = "ALL") String health,
            @RequestParam(defaultValue = "name") String sort,
            @RequestParam(defaultValue = "asc") String order,
            @RequestParam(defaultValue = "0") int page,
            @RequestParam(defaultValue = "10") int size) {
        int effectiveSize = Math.min(size, 100);
        try {
            // Use RestClient for _cat/indices with JSON format (richer than Java client's API)
            Request catRequest = new Request("GET", "/_cat/indices?format=json&h=index,health,docs.count,store.size,pri,rep&bytes=b");
            Response catResponse = restClient.performRequest(catRequest);
            var allIndices = objectMapper.readValue(
                    catResponse.getEntity().getContent(),
                    new TypeReference<List<Map<String, String>>>() {});

            // Filter by search pattern
            var filtered = allIndices.stream()
                    .filter(idx -> search.isEmpty() || idx.get("index").contains(search))
                    .filter(idx -> "ALL".equals(health) || health.equalsIgnoreCase(idx.get("health")))
                    .toList();

            // Compute summary totals before pagination
            long totalDocs = filtered.stream().mapToLong(idx -> parseLong(idx.get("docs.count"))).sum();
            long totalBytes = filtered.stream().mapToLong(idx -> parseLong(idx.get("store.size"))).sum();

            // Sort
            Comparator<Map<String, String>> comparator = switch (sort) {
                case "docs" -> Comparator.comparingLong(m -> parseLong(m.get("docs.count")));
                case "size" -> Comparator.comparingLong(m -> parseLong(m.get("store.size")));
                case "health" -> Comparator.comparing(m -> m.get("health"));
                default -> Comparator.comparing(m -> m.get("index"));
            };
            if ("desc".equalsIgnoreCase(order)) comparator = comparator.reversed();

            // Paginate
            var sorted = filtered.stream().sorted(comparator).toList();
            int fromIndex = page * effectiveSize;
            int toIndex = Math.min(fromIndex + effectiveSize, sorted.size());
            var pageItems = fromIndex < sorted.size()
                    ? sorted.subList(fromIndex, toIndex) : List.<Map<String, String>>of();

            var indices = pageItems.stream().map(m -> new IndexInfoResponse(
                    m.get("index"), parseLong(m.get("docs.count")),
                    humanSize(parseLong(m.get("store.size"))),
                    parseLong(m.get("store.size")),
                    m.getOrDefault("health", "unknown"),
                    parseInt(m.get("pri")), parseInt(m.get("rep"))
            )).toList();

            int totalPages = (int) Math.ceil((double) filtered.size() / effectiveSize);
            return ResponseEntity.ok(new IndicesPageResponse(
                    indices, filtered.size(), totalDocs,
                    humanSize(totalBytes), page, effectiveSize, totalPages));
        } catch (Exception e) {
            throw new ResponseStatusException(HttpStatus.BAD_GATEWAY, "OpenSearch unreachable");
        }
    }

    private long parseLong(String s) { try { return Long.parseLong(s); } catch (Exception e) { return 0; } }
    private int parseInt(String s) { try { return Integer.parseInt(s); } catch (Exception e) { return 0; } }
    private String humanSize(long bytes) {
        if (bytes < 1024) return bytes + " B";
        if (bytes < 1024 * 1024) return String.format("%.1f KB", bytes / 1024.0);
        if (bytes < 1024L * 1024 * 1024) return String.format("%.1f MB", bytes / (1024.0 * 1024));
        return String.format("%.1f GB", bytes / (1024.0 * 1024 * 1024));
    }

    @DeleteMapping("/indices/{name}")
    public ResponseEntity<Void> deleteIndex(@PathVariable String name, HttpServletRequest request) {
        try {
            var exists = client.indices().exists(e -> e.index(name));
            if (!exists.value()) {
                throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Index not found: " + name);
            }
            client.indices().delete(d -> d.index(name));
            auditService.log("delete_index", AuditCategory.INFRA, name, null, AuditResult.SUCCESS, request);
            return ResponseEntity.ok().build();
        } catch (ResponseStatusException e) { throw e; }
        catch (Exception e) {
            throw new ResponseStatusException(HttpStatus.BAD_GATEWAY, "OpenSearch unreachable");
        }
    }

    @GetMapping("/performance")
    public ResponseEntity<PerformanceResponse> getPerformance() {
        try {
            // Use RestClient for _nodes/stats — the Java client exposes this via NodesStatsRequest
            Request statsRequest = new Request("GET", "/_nodes/stats/jvm,indices");
            Response statsResponse = restClient.performRequest(statsRequest);
            var statsJson = objectMapper.readTree(statsResponse.getEntity().getContent());

            // Aggregate across all nodes
            var nodes = statsJson.get("nodes");
            long jvmHeapUsed = 0, jvmHeapMax = 0;
            long queryCacheHits = 0, queryCacheMisses = 0;
            long requestCacheHits = 0, requestCacheMisses = 0;
            long searchTotal = 0, searchTimeMs = 0;
            long indexingTotal = 0, indexingTimeMs = 0;

            var nodeIter = nodes.fields();
            while (nodeIter.hasNext()) {
                var node = nodeIter.next().getValue();
                var jvm = node.get("jvm").get("mem");
                jvmHeapUsed += jvm.get("heap_used_in_bytes").asLong();
                jvmHeapMax += jvm.get("heap_max_in_bytes").asLong();

                var indices = node.get("indices");
                var qc = indices.get("query_cache");
                queryCacheHits += qc.get("hit_count").asLong();
                queryCacheMisses += qc.get("miss_count").asLong();
                var rc = indices.get("request_cache");
                requestCacheHits += rc.get("hit_count").asLong();
                requestCacheMisses += rc.get("miss_count").asLong();
                var search = indices.get("search");
                searchTotal += search.get("query_total").asLong();
                searchTimeMs += search.get("query_time_in_millis").asLong();
                var indexing = indices.get("indexing");
                indexingTotal += indexing.get("index_total").asLong();
                indexingTimeMs += indexing.get("index_time_in_millis").asLong();
            }

            double qcHitRate = (queryCacheHits + queryCacheMisses) > 0
                    ? 100.0 * queryCacheHits / (queryCacheHits + queryCacheMisses) : 0;
            double rcHitRate = (requestCacheHits + requestCacheMisses) > 0
                    ? 100.0 * requestCacheHits / (requestCacheHits + requestCacheMisses) : 0;
            double avgSearchLatency = searchTotal > 0 ? (double) searchTimeMs / searchTotal : 0;
            double avgIndexingLatency = indexingTotal > 0 ? (double) indexingTimeMs / indexingTotal : 0;

            return ResponseEntity.ok(new PerformanceResponse(
                    qcHitRate, rcHitRate, avgSearchLatency, avgIndexingLatency,
                    jvmHeapUsed, jvmHeapMax));
        } catch (Exception e) {
            throw new ResponseStatusException(HttpStatus.BAD_GATEWAY, "OpenSearch unreachable");
        }
    }
}
```

**Note on OpenSearch client access:** The `OpenSearchClient` bean is already available. For some operations (like `_cat/indices` with sorting), you may need to use the underlying `RestClient` directly with a raw HTTP request, since the Java client may not expose all `_cat` parameters. Alternatively, use the `_cat/indices` API via `RestClient.performRequest()` with `format=json`.

- [ ] **Step 4: Run integration test**

Run: `mvn test -pl cameleer-server-app -Dtest=OpenSearchAdminControllerIT`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add cameleer-server-app/
git commit -m "feat: add OpenSearchAdminController with status, pipeline, indices, performance, and delete endpoints"
```

---

### Task 10: App Module — ThresholdAdminController + AuditLogController

**Files:**
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/ThresholdAdminController.java`
- Create: `cameleer-server-app/src/main/java/com/cameleer/server/app/controller/AuditLogController.java`
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/controller/ThresholdAdminControllerIT.java`
- Create: `cameleer-server-app/src/test/java/com/cameleer/server/app/controller/AuditLogControllerIT.java`

- [ ] **Step 1: Write ThresholdAdminController integration test**

- `GET /api/v1/admin/thresholds` → 200, returns defaults if no row exists
- `PUT /api/v1/admin/thresholds` with valid payload → 200
- `GET /api/v1/admin/thresholds` after save → returns saved values
- `PUT /api/v1/admin/thresholds` with warning > critical → 400

- [ ] **Step 2: Write AuditLogController integration test**

- `GET /api/v1/admin/audit` → 200, returns paginated results
- `GET /api/v1/admin/audit?category=INFRA` → 200, only INFRA entries
- `GET /api/v1/admin/audit?search=kill_query` → 200, filtered results
- Verify that the audit log captures actions from other admin endpoints (e.g., call kill query, then check audit log)

- [ ] **Step 3: Implement ThresholdAdminController**

```java
@RestController
@RequestMapping("/api/v1/admin/thresholds")
@PreAuthorize("hasRole('ADMIN')")
@Tag(name = "Threshold Admin", description = "Threshold configuration (ADMIN only)")
public class ThresholdAdminController {
    private final ThresholdRepository repository;
    private final AuditService auditService;

    @GetMapping
    public ResponseEntity<ThresholdConfig> get() {
        return ResponseEntity.ok(repository.find().orElse(ThresholdConfig.defaults()));
    }

    @PutMapping
    public ResponseEntity<?> save(@Valid @RequestBody ThresholdConfigRequest request,
                                   HttpServletRequest httpRequest) {
        var errors = request.validate();
        if (!errors.isEmpty()) {
            throw new ResponseStatusException(HttpStatus.BAD_REQUEST,
                    "Validation failed: " + String.join("; ", errors));
        }
        ThresholdConfig config = request.toConfig();
        // Extract username from SecurityContext (same approach as AuditService)
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        String username = auth != null ? auth.getName() : "unknown";
        if (username.startsWith("user:")) username = username.substring(5);

        repository.save(config, username);
        auditService.log("update_thresholds", AuditCategory.INFRA, "thresholds",
                Map.of("config", config), AuditResult.SUCCESS, httpRequest);
        return ResponseEntity.ok(config);
    }
}
```

- [ ] **Step 4: Implement AuditLogController**

```java
@RestController
@RequestMapping("/api/v1/admin/audit")
@PreAuthorize("hasRole('ADMIN')")
@Tag(name = "Audit Log", description = "Audit log viewer (ADMIN only)")
public class AuditLogController {
    private final AuditRepository repository;

    @GetMapping
    public ResponseEntity<AuditLogPageResponse> getAuditLog(
            @RequestParam(required = false) String username,
            @RequestParam(required = false) AuditCategory category,
            @RequestParam(required = false) String search,
            @RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate from,
            @RequestParam(required = false) @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate to,
            @RequestParam(defaultValue = "timestamp") String sort,
            @RequestParam(defaultValue = "desc") String order,
            @RequestParam(defaultValue = "0") int page,
            @RequestParam(defaultValue = "25") int size) {
        size = Math.min(size, 100);
        // Convert LocalDate to Instant (start of day / end of day in UTC)
        Instant fromInstant = (from != null ? from : LocalDate.now().minusDays(7))
                .atStartOfDay(ZoneOffset.UTC).toInstant();
        Instant toInstant = (to != null ? to.plusDays(1) : LocalDate.now().plusDays(1))
                .atStartOfDay(ZoneOffset.UTC).toInstant();
        var query = new AuditRepository.AuditQuery(username, category, search, fromInstant, toInstant, sort, order, page, size);
        var result = repository.find(query);
        int totalPages = (int) Math.ceil((double) result.totalCount() / size);
        return ResponseEntity.ok(new AuditLogPageResponse(result.items(), result.totalCount(), page, size, totalPages));
    }
}
```

- [ ] **Step 5: Run tests**

Run: `mvn test -pl cameleer-server-app -Dtest="ThresholdAdminControllerIT,AuditLogControllerIT"`
Expected: PASS

- [ ] **Step 6: Commit**

```bash
git add cameleer-server-app/
git commit -m "feat: add ThresholdAdminController and AuditLogController with integration tests"
```

---

### Task 11: Regenerate OpenAPI Spec

**Files:**
- Modify: `ui/src/api/schema/openapi.json` (regenerated)
- Modify: `ui/src/api/schema.d.ts` (regenerated)

- [ ] **Step 1: Start the server and regenerate**

The project regenerates `openapi.json` from the running server. Follow the pattern from `feedback_regenerate_openapi.md`:

1. Start the server (needs PG + OpenSearch running, or use test profile)
2. Fetch OpenAPI JSON: `curl http://localhost:8081/api/v1/api-docs > ui/src/api/schema/openapi.json`
3. Regenerate TypeScript types: `cd ui && npx openapi-typescript src/api/schema/openapi.json -o src/api/schema.d.ts`

- [ ] **Step 2: Verify frontend compiles**

Run: `cd ui && npm run build`
Expected: BUILD SUCCESS (or only warnings, no errors)

- [ ] **Step 3: Commit**

```bash
git add ui/src/api/schema/
git commit -m "chore: regenerate OpenAPI spec and TypeScript types for admin endpoints"
```

---

### Task 12: Frontend — Shared Admin Components

**Files:**
- Create: `ui/src/components/admin/StatusBadge.tsx` + `.module.css`
- Create: `ui/src/components/admin/RefreshableCard.tsx` + `.module.css`
- Create: `ui/src/components/admin/ConfirmDeleteDialog.tsx` + `.module.css`

- [ ] **Step 1: Create StatusBadge component**

A small colored dot + label. Props: `status: 'healthy' | 'warning' | 'critical' | 'unknown'`, `label?: string`. Colors: green/yellow/red/gray. Uses CSS modules consistent with existing component styling.

```tsx
import styles from './StatusBadge.module.css';

type Status = 'healthy' | 'warning' | 'critical' | 'unknown';

export function StatusBadge({ status, label }: { status: Status; label?: string }) {
    return (
        <span className={`${styles.badge} ${styles[status]}`}>
            <span className={styles.dot} />
            {label && <span className={styles.label}>{label}</span>}
        </span>
    );
}
```

- [ ] **Step 2: Create RefreshableCard component**

A collapsible card with title, optional auto-refresh indicator, and manual refresh button. Props: `title`, `onRefresh`, `isRefreshing`, `autoRefresh?: boolean`, `children`.

- [ ] **Step 3: Create ConfirmDeleteDialog component**

Modal dialog. Props: `isOpen`, `onClose`, `onConfirm`, `resourceName`, `resourceType`. Requires user to type the resource name to confirm. Disabled confirm button until input matches.

- [ ] **Step 4: Verify frontend compiles**

Run: `cd ui && npm run build`

- [ ] **Step 5: Commit**

```bash
git add ui/src/components/admin/
git commit -m "feat: add shared admin UI components (StatusBadge, RefreshableCard, ConfirmDeleteDialog)"
```

---

### Task 13: Frontend — Admin Sidebar + Routing

**Files:**
- Modify: `ui/src/components/layout/AppSidebar.tsx`
- Modify: `ui/src/router.tsx`

- [ ] **Step 1: Update router.tsx**

Add new admin routes inside the `AppShell` children array. Add redirect for `/admin`:

```tsx
{ path: 'admin', element: <Navigate to="/admin/database" replace /> },
{ path: 'admin/database', element: <DatabaseAdminPage /> },
{ path: 'admin/opensearch', element: <OpenSearchAdminPage /> },
{ path: 'admin/audit', element: <AuditLogPage /> },
{ path: 'admin/oidc', element: <OidcAdminPage /> },
```

Use lazy imports for the new pages (consistent with SwaggerPage pattern).

- [ ] **Step 2: Refactor AppSidebar admin section**

Replace the single gear-icon link with a collapsible section:

```tsx
// Admin section (visible only to ADMIN role)
{roles.includes('ADMIN') && (
    <div className={styles.adminSection}>
        <button className={styles.adminHeader} onClick={toggleAdmin}>
            <SettingsIcon /> Admin {adminOpen ? '▾' : '▸'}
        </button>
        {adminOpen && (
            <nav className={styles.adminNav}>
                <NavLink to="/admin/database">Database</NavLink>
                <NavLink to="/admin/opensearch">OpenSearch</NavLink>
                <NavLink to="/admin/audit">Audit Log</NavLink>
                <NavLink to="/admin/oidc">OIDC</NavLink>
            </nav>
        )}
    </div>
)}
```

Persist `adminOpen` state in localStorage (key: `cameleer-admin-sidebar-open`). Default: collapsed.

- [ ] **Step 3: Verify frontend compiles and routing works**

Run: `cd ui && npm run build`

- [ ] **Step 4: Commit**

```bash
git add ui/src/router.tsx ui/src/components/layout/
git commit -m "feat: restructure admin sidebar with collapsible sub-navigation and new routes"
```

---

### Task 14: Frontend — React Query Hooks

**Files:**
- Create: `ui/src/api/queries/admin/database.ts`
- Create: `ui/src/api/queries/admin/opensearch.ts`
- Create: `ui/src/api/queries/admin/thresholds.ts`
- Create: `ui/src/api/queries/admin/audit.ts`

- [ ] **Step 1: Create database query hooks**

Follow the exact pattern from `oidc-admin.ts`:

```typescript
import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query';
import { api } from '../../client';

export function useDatabaseStatus() {
    return useQuery({
        queryKey: ['admin', 'database', 'status'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/database/status');
            if (error) throw new Error('Failed to load database status');
            return data!;
        },
    });
}

export function useDatabasePool() {
    return useQuery({
        queryKey: ['admin', 'database', 'pool'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/database/pool');
            if (error) throw new Error('Failed to load pool stats');
            return data!;
        },
        refetchInterval: 15_000,
    });
}

export function useDatabaseTables() {
    return useQuery({
        queryKey: ['admin', 'database', 'tables'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/database/tables');
            if (error) throw new Error('Failed to load table sizes');
            return data!;
        },
        // NO refetchInterval — manual refresh only
    });
}

export function useDatabaseQueries() {
    return useQuery({
        queryKey: ['admin', 'database', 'queries'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/database/queries');
            if (error) throw new Error('Failed to load active queries');
            return data!;
        },
        refetchInterval: 15_000,
    });
}

export function useKillQuery() {
    const qc = useQueryClient();
    return useMutation({
        mutationFn: async (pid: number) => {
            const { error } = await api.POST('/admin/database/queries/{pid}/kill', { params: { path: { pid } } });
            if (error) throw new Error('Failed to kill query');
        },
        onSuccess: () => qc.invalidateQueries({ queryKey: ['admin', 'database', 'queries'] }),
    });
}
```

- [ ] **Step 2: Create OpenSearch query hooks**

Same pattern as database hooks. Create:

```typescript
export function useOpenSearchStatus() {
    return useQuery({
        queryKey: ['admin', 'opensearch', 'status'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/opensearch/status');
            if (error) throw new Error('Failed to load OpenSearch status');
            return data!;
        },
    });
}

export function usePipelineStats() {
    return useQuery({
        queryKey: ['admin', 'opensearch', 'pipeline'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/opensearch/pipeline');
            if (error) throw new Error('Failed to load pipeline stats');
            return data!;
        },
        refetchInterval: 15_000,
    });
}

export function useIndices(params: { search?: string; health?: string; sort?: string; order?: string; page?: number; size?: number }) {
    return useQuery({
        queryKey: ['admin', 'opensearch', 'indices', params],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/opensearch/indices', { params: { query: params } });
            if (error) throw new Error('Failed to load indices');
            return data!;
        },
        // NO refetchInterval — manual refresh only
    });
}

export function usePerformanceStats() {
    return useQuery({
        queryKey: ['admin', 'opensearch', 'performance'],
        queryFn: async () => {
            const { data, error } = await api.GET('/admin/opensearch/performance');
            if (error) throw new Error('Failed to load performance stats');
            return data!;
        },
        refetchInterval: 15_000,
    });
}

export function useDeleteIndex() {
    const qc = useQueryClient();
    return useMutation({
        mutationFn: async (name: string) => {
            const { error } = await api.DELETE('/admin/opensearch/indices/{name}', { params: { path: { name } } });
            if (error) throw new Error('Failed to delete index');
        },
        onSuccess: () => qc.invalidateQueries({ queryKey: ['admin', 'opensearch', 'indices'] }),
    });
}
```

- [ ] **Step 3: Create threshold hooks**

`useThresholds()` query + `useSaveThresholds()` mutation.

- [ ] **Step 4: Create audit log hooks**

`useAuditLog()` query that accepts filter params (username, category, search, from, to, page, size) and includes them in the query key for proper cache management.

- [ ] **Step 5: Verify frontend compiles**

Run: `cd ui && npm run build`

- [ ] **Step 6: Commit**

```bash
git add ui/src/api/queries/admin/
git commit -m "feat: add React Query hooks for database, OpenSearch, threshold, and audit log admin endpoints"
```

---

### Task 15: Frontend — Database Admin Page

**Files:**
- Create: `ui/src/pages/admin/DatabaseAdminPage.tsx` + `.module.css`

- [ ] **Step 1: Implement DatabaseAdminPage**

Page structure (follow `OidcAdminPage` patterns for role check, loading states, error handling):

1. **Header**: StatusBadge (connected/disconnected) + version + host + schema + manual refresh-all button
2. **Connection Pool card** (RefreshableCard, auto-refresh): progress bar showing active/max, metrics grid
3. **Table Sizes card** (RefreshableCard, manual): table with sortable columns, summary row
4. **Active Queries card** (RefreshableCard, auto-refresh): table with Kill button per row (behind confirmation dialog)
5. **Maintenance card** (Phase 2 placeholder): greyed-out buttons with tooltip
6. **Thresholds section** (collapsible): form inputs for warning/critical values, save button

Use the `useThresholds()` hook to load current threshold values and apply them to StatusBadge calculations (e.g., pool usage % > warning → yellow badge).

- [ ] **Step 2: Style the page**

CSS modules, consistent with existing admin page styling from `OidcAdminPage.module.css`.

- [ ] **Step 3: Verify page renders**

Run: `cd ui && npm run build`

- [ ] **Step 4: Commit**

```bash
git add ui/src/pages/admin/DatabaseAdminPage.tsx ui/src/pages/admin/DatabaseAdminPage.module.css
git commit -m "feat: add Database admin page with pool, tables, queries, and thresholds UI"
```

---

### Task 16: Frontend — OpenSearch Admin Page

**Files:**
- Create: `ui/src/pages/admin/OpenSearchAdminPage.tsx` + `.module.css`

- [ ] **Step 1: Implement OpenSearchAdminPage**

Page structure:

1. **Header**: StatusBadge (cluster health) + version + nodes + host + manual refresh-all
2. **Indexing Pipeline card** (auto-refresh): queue depth bar, metrics grid, status badge
3. **Indices card** (manual refresh): search input, health filter dropdown, sortable/paginated table, delete button per row (ConfirmDeleteDialog with name-typing confirmation), summary row above table
4. **Performance card** (auto-refresh): cache hit rates, latencies, JVM heap bar
5. **Operations card** (Phase 2 placeholder): greyed-out buttons
6. **Thresholds section** (collapsible): form inputs for OpenSearch-specific thresholds

- [ ] **Step 2: Style the page**

- [ ] **Step 3: Verify page renders**

Run: `cd ui && npm run build`

- [ ] **Step 4: Commit**

```bash
git add ui/src/pages/admin/OpenSearchAdminPage.tsx ui/src/pages/admin/OpenSearchAdminPage.module.css
git commit -m "feat: add OpenSearch admin page with pipeline, indices, performance, and thresholds UI"
```

---

### Task 17: Frontend — Audit Log Page

**Files:**
- Create: `ui/src/pages/admin/AuditLogPage.tsx` + `.module.css`

- [ ] **Step 1: Implement AuditLogPage**

Page structure:

1. **Header**: total event count + date range picker (two date inputs, default last 7 days)
2. **Filters row**: username dropdown (populated from distinct values or free text), category dropdown (INFRA/AUTH/USER_MGMT/CONFIG/All), free text search input
3. **Table**: columns — Timestamp, User, Category, Action, Target, Result. Click row to expand and show full `detail` JSON (formatted/pretty-printed)
4. **Pagination**: page controls below table, showing "Showing X-Y of Z"

No auto-refresh. Read-only — no edit or delete buttons anywhere.

- [ ] **Step 2: Style the page**

- [ ] **Step 3: Verify page renders**

Run: `cd ui && npm run build`

- [ ] **Step 4: Commit**

```bash
git add ui/src/pages/admin/AuditLogPage.tsx ui/src/pages/admin/AuditLogPage.module.css
git commit -m "feat: add Audit Log admin page with filtering, pagination, and detail expansion"
```

---

### Task 18: End-to-End Verification

- [ ] **Step 1: Run full backend build**

Run: `mvn clean verify`
Expected: BUILD SUCCESS with all tests passing

- [ ] **Step 2: Run full frontend build**

Run: `cd ui && npm run build`
Expected: BUILD SUCCESS

- [ ] **Step 3: Manual smoke test**

Start the full stack (PG + OpenSearch + server) and verify:
1. Admin sidebar shows collapsible sub-menu with Database, OpenSearch, Audit Log, OIDC
2. Database page shows connection status, pool stats, table sizes, active queries
3. OpenSearch page shows cluster health, pipeline stats, indices list, performance
4. Audit Log page shows entries from admin actions
5. Kill query and delete index work with confirmation dialogs
6. Thresholds save and load correctly
7. Non-admin users cannot see admin sidebar or access admin API endpoints (verify 403)

- [ ] **Step 4: Final commit if any fixes needed**

- [ ] **Step 5: Update HOWTO.md**

Add section for new admin pages — how to access, what each page shows, how thresholds work.

```bash
git add HOWTO.md
git commit -m "docs: update HOWTO.md with admin infrastructure pages"
```

- [ ] **Step 6: Push to Gitea**

```bash
git push origin main
```