# Streaming Views: Multi-Select Filters + Infinite Scroll

**Date:** 2026-04-17
**Status:** Draft

## Problem

In the Runtime page's **Application Log**, the `App / Agent / Container` source buttons are single-select, and the **Level** buttons are multi-select but filtered **client-side**. Both are flawed for the same reason: when one source or level floods the result set (e.g. hundreds of INFO-level app logs), less-frequent entries (container logs, ERROR lines) can be pushed past the fetched page and never appear.

The page is also a single `useQuery` that fetches at most `limit` rows. There is no way to browse older logs.

The **Timeline** (agent events) has the same single-page limitation.

## Goal

Establish a consistent pattern for streaming views (log-like lists + event feeds):

1. **Multi-select server-side filters** joined with `OR` — the server decides what's included, so no category is ever silently starved.
2. **Infinite scroll** via cursor pagination — user scrolls down to load older entries.
3. **Top-gated auto-refetch** — polling only runs while the user is viewing the newest entries; when they scroll away to inspect history, polling pauses so their viewport cannot shift under them.
4. **Client-side text filter** only — text search filters pages already loaded without re-querying.

Apply this pattern to:
- `AgentHealth.tsx` — Application Log + Timeline
- `AgentInstance.tsx` — Application Log + Timeline
- Shared components/hooks so future streaming views reuse the same primitives.

Out of scope for infinite scroll (bounded data, don't need it):
- `LogTab.tsx` — per-exchange logs (bounded, one exchange, already capped at 500)
- `StartupLogPanel.tsx` — container logs during a single deployment startup

These two keep `useLogs` / `useStartupLogs` as-is. They will benefit automatically if `LogEntry.source` gets pushed through to `LogViewer` for badge rendering (already supported by DS).

## Design

### 1. Backend: multi-value `source`

The log endpoint already supports comma-split `level`. Mirror that for `source`.

**`cameleer-server-core/.../search/LogSearchRequest.java`**
- Change `String source` → `List<String> sources` (default empty list).

**`cameleer-server-app/.../controller/LogQueryController.java`**
- `@RequestParam(required = false) String source` stays.
- Parse same way as `level` (comma-split, trim, drop blanks) into `List<String>`.

**`cameleer-server-app/.../search/ClickHouseLogStore.java`**
- Replace:
  ```
  if (request.source() != null && !request.source().isEmpty()) {
      baseConditions.add("source = ?");
      baseParams.add(request.source());
  }
  ```
  with a list-driven `source IN (?, ?, …)` built when `request.sources()` is non-empty.

No schema change. No migration.

### 2. Backend: cursor pagination for agent events

Agent events currently return `List<AgentEventResponse>` with `limit` (default 50). No cursor. To support infinite scroll we need stable desc ordering + a cursor.

Events have no stable ID (the ClickHouse repository synthesizes `id = 0`). The stable ordering is `(timestamp DESC, instance_id ASC)`. Cursor encodes that tuple as an opaque base64 string, matching how `ClickHouseLogStore` builds log cursors.

**`cameleer-server-core/.../agent/AgentEventService.java` / `AgentEventRepository.java`**
- Add overload `queryEvents(appId, agentId, env, from, to, cursor, limit)` returning a `AgentEventPage { data, nextCursor, hasMore }`.
- Keep the existing un-cursored `queryEvents` if used elsewhere (audit usage; if not, delete it).

**`cameleer-server-app/.../storage/ClickHouseAgentEventRepository.java`**
- New method: on cursor, decode `(cursorTs, cursorInstance)` and add `(timestamp, instance_id) < (?, ?)` predicate.
- Always `ORDER BY timestamp DESC, instance_id ASC`.
- Always fetch `limit + 1` rows; last row determines `hasMore` + `nextCursor`.

**`cameleer-server-app/.../controller/AgentEventsController.java`**
- Accept optional `?cursor=`; return `AgentEventPageResponse { data, nextCursor, hasMore }` instead of a bare list.

### 3. Shared UI primitives

Two thin primitives. No new design-system package; live under `ui/src/components/` and `ui/src/hooks/` so they're reused from the app only.

**`ui/src/hooks/useInfiniteStream.ts`** — thin `useInfiniteQuery` wrapper:

```ts
interface StreamPage<T> { data: T[]; nextCursor: string | null; hasMore: boolean }

interface UseInfiniteStreamArgs<T> {
  queryKey: unknown[];
  fetchPage: (cursor: string | undefined) => Promise<StreamPage<T>>;
  enabled?: boolean;
  isAtTop: boolean;     // drives refetchInterval on/off
  refetchMs?: number;   // default 15000
}

interface UseInfiniteStreamResult<T> {
  items: T[];
  fetchNextPage: () => void;
  hasNextPage: boolean;
  isFetchingNextPage: boolean;
  isLoading: boolean;
  refresh: () => void;  // invalidates and re-fetches from page 1
}
```

Internals: wraps `useInfiniteQuery` with `getNextPageParam: (last) => last.hasMore ? last.nextCursor : undefined`, flattens pages, exposes `refresh` that calls `queryClient.invalidateQueries(queryKey)`.

**`ui/src/components/InfiniteScrollArea.tsx`** — scrollable container:

```tsx
interface InfiniteScrollAreaProps {
  onEndReached: () => void;          // call fetchNextPage
  onTopVisibilityChange: (b: boolean) => void;
  isFetchingNextPage: boolean;
  hasNextPage: boolean;
  maxHeight?: number | string;
  children: ReactNode;
}
```

Renders:
```
<div ref={scrollRef} style={{ overflowY: 'auto', maxHeight }}>
  <div ref={topSentinelRef} />     // IntersectionObserver -> onTopVisibilityChange
  {children}
  <div ref={bottomSentinelRef} />  // IntersectionObserver -> onEndReached
  {isFetchingNextPage && <Spinner />}
  {!hasNextPage && <EndMarker />}
</div>
```

A single `IntersectionObserver` per sentinel with `rootMargin: '100px'` for the bottom so the next page is prefetched before the user hits the literal bottom.

### 4. New hooks

**`ui/src/api/queries/logs.ts`**

Add `useInfiniteApplicationLogs`:

```ts
export function useInfiniteApplicationLogs(args: {
  application?: string;
  agentId?: string;
  sources?: string[];   // multi-select, server-side
  levels?: string[];    // multi-select, server-side
  exchangeId?: string;
  isAtTop: boolean;
}): UseInfiniteStreamResult<LogEntryResponse>
```

Under the hood: cursor-paginated calls to `/environments/{env}/logs`; `fetchPage(cursor)` sets `source=a,b`, `level=ERROR,WARN`, and time range from the global filter store.

Keep `useLogs` and `useApplicationLogs` (the existing single-page wrapper) so `LogTab` and `StartupLogPanel` remain untouched. Mark `useApplicationLogs` internally as "bounded consumers only" via a short TSDoc line.

**`ui/src/api/queries/agents.ts`**

Add `useInfiniteAgentEvents`:

```ts
export function useInfiniteAgentEvents(args: {
  appId?: string;
  agentId?: string;
  isAtTop: boolean;
}): UseInfiniteStreamResult<AgentEventResponse>
```

Existing `useAgentEvents` is only used in `AgentHealth`/`AgentInstance` for the Timeline. It gets replaced by the new hook — no other consumers, no backwards-compat shim.

### 5. Page wiring — `AgentHealth.tsx` and `AgentInstance.tsx`

**Application Log block:**

- State
  - `logSources: Set<string>` (new, replaces `logSource: string`).
  - `logLevels: Set<string>` (unchanged shape, now passed through as `levels`).
  - `logSearch: string` (unchanged, client-side).
  - `logSortAsc: boolean` (unchanged).
  - `isLogAtTop: boolean` (new, from `InfiniteScrollArea`).

- Data
  - `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteApplicationLogs({ application: appId, agentId, sources: [...logSources], levels: [...logLevels], isAtTop: isLogAtTop });`

- Map → `LogEntry[]`
  - Same mapping as today plus `source: l.source ?? undefined` so `LogViewer`'s source badge lights up.

- Client-side
  - Text filter (`logSearch`) applied after flattening.
  - Sort reversal (`logSortAsc`) applied after.

- ButtonGroup wiring
  - `<ButtonGroup items={LOG_SOURCE_ITEMS} value={logSources} onChange={setLogSources} />`
  - Level `<ButtonGroup>` unchanged (`value={logLevels}`), but `onChange` now implicitly re-queries.
  - A "Clear" button next to each group when the set is non-empty (source has 3 options; still low-cost and symmetric with level).

- Layout
  - `<LogViewer entries={filteredLogs} />` (no `maxHeight`) wrapped in `<InfiniteScrollArea maxHeight={360} ...>`.

**Timeline block:**

- `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteAgentEvents({ appId, isAtTop: isTimelineAtTop });`
- Map to `FeedEvent[]` (same transform as today).
- `<EventFeed events={feedEvents} />` inside `<InfiniteScrollArea>`.
- Timeline has no filters today; none added in this spec.

**Refresh buttons:**
Both log and timeline `<RefreshCw>` buttons call `refresh()` from the stream hook. `logRefreshTo`/`eventRefreshTo` states are retired.

### 6. Auto-refetch gating semantics

- `InfiniteScrollArea` fires `onTopVisibilityChange(true)` when the top sentinel is fully in view, `false` when it scrolls out. Implemented via `IntersectionObserver` with threshold `1.0`.
- `useInfiniteStream` reads `isAtTop` and sets `refetchInterval: isAtTop ? refetchMs : false`. Tanstack re-evaluates on every render, so toggling the prop takes effect on the next cycle.
- Manual `refresh()` always works and scrolls back to top (scroll reset is owned by the page, calling `scrollRef.current?.scrollTo({ top: 0 })` after the refresh settles — wired in `AgentHealth` / `AgentInstance`).
- No viewport-preservation logic. The contract "refetch must not move the user's viewport" is satisfied by disabling refetch while the user is scrolled away.

### 7. Bounded views — minimal changes

- `LogTab.tsx`: map `e.source` into the rendered `LogEntry` so source badges appear. No other changes.
- `StartupLogPanel.tsx`: no change; startup-log rows are container-sourced by definition.

### 8. OpenAPI regeneration

After controller/DTO changes:
```
cd ui && npm run generate-api:live
```
Commit the resulting `openapi.json` + `schema.d.ts` in the same change. TypeScript will surface any SPA call sites that need adjustment; fix all of them before testing in the browser.

### 9. `.claude/rules/` updates

- `app-classes.md` — `LogQueryController` entry: add "multi-value `source` (comma-split)"; `AgentEventsController` entry: add "cursor-paginated, returns `{ data, nextCursor, hasMore }`".
- `ui.md` — add entry for `InfiniteScrollArea` component and note the "streaming views use `useInfiniteStream` + `InfiniteScrollArea`" convention.

## Testing

- **Backend unit**: `ClickHouseLogStoreTest` for `source IN (...)` predicate (single, multi, empty). `ClickHouseAgentEventRepositoryTest` for cursor ordering and `hasMore` boundary.
- **Controller**: `LogQueryControllerTest` (existing) — add `source=app,container` case. `AgentEventsControllerTest` — cursor round-trip.
- **UI manual smoke**: scroll long log streams, mix source filters, verify auto-refetch toggles at top, confirm text filter stays local, confirm refresh resets scroll.

## Risks & mitigations

| Risk | Mitigation |
|---|---|
| Event cursor using `(timestamp, instance_id)` — collisions if two events share both | In practice events are recorded one-at-a-time per instance; tuple is stable. Tie-breaker fine as both fields are indexed. |
| `useInfiniteQuery` refetch reloads all pages → visible flicker | Use `placeholderData: (prev) => prev` so stale data stays during refetch. Tested pattern already used by `useApplicationLogs`. |
| IntersectionObserver on the top sentinel fires both on scroll and on list grow | Debounce via React state, not a ref — state updates coalesce per render. |
| DS `LogViewer` may virtualize internally, hiding sentinels | `LogViewer` today has no virtualization (DS 0.1.49 props are minimal). If that changes, move sentinels into the outer scroll container (already the plan). |
| Level client→server change breaks existing users with saved filter state | No persisted filter state exists for log levels; this is in-memory only. |

## Decision log

- **Server-side for source and level, client-side for text.** Confirmed during design — flooding argument applies symmetrically to source and level.
- **Gate auto-refetch on top visibility** instead of preserving viewport. Rationale: viewport preservation requires stable row IDs (logs have timestamp collisions) and is fragile under virtualization. Gating is simple, predictable, and meets the "user never loses their line" constraint.
- **Bounded log views (LogTab, StartupLogPanel) keep single-page hooks.** Infinite scroll isn't useful for capped data; shared primitives are available if they ever need it.
- **Agent events get cursor pagination**, matching logs. Single-source-of-truth pattern for streaming lists.
