From b14551de4e627c7f4dfdc70d81b4f32f99e8d6bf Mon Sep 17 00:00:00 2001 From: hsiegeln <37154749+hsiegeln@users.noreply.github.com> Date: Fri, 17 Apr 2026 11:26:39 +0200 Subject: [PATCH] docs: spec for multi-select log filters + infinite scroll Establishes the shared pattern for streaming views (logs + agent events): server-side multi-select source/level filters, cursor-based infinite scroll, and top-gated auto-refetch. Scoped to AgentHealth and AgentInstance; bounded views (LogTab, StartupLogPanel) keep single-page hooks. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...ters-multiselect-infinite-scroll-design.md | 252 ++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-17-log-filters-multiselect-infinite-scroll-design.md diff --git a/docs/superpowers/specs/2026-04-17-log-filters-multiselect-infinite-scroll-design.md b/docs/superpowers/specs/2026-04-17-log-filters-multiselect-infinite-scroll-design.md new file mode 100644 index 00000000..92c805ec --- /dev/null +++ b/docs/superpowers/specs/2026-04-17-log-filters-multiselect-infinite-scroll-design.md @@ -0,0 +1,252 @@ +# Streaming Views: Multi-Select Filters + Infinite Scroll + +**Date:** 2026-04-17 +**Status:** Draft + +## Problem + +In the Runtime page's **Application Log**, the `App / Agent / Container` source buttons are single-select, and the **Level** buttons are multi-select but filtered **client-side**. Both are flawed for the same reason: when one source or level floods the result set (e.g. hundreds of INFO-level app logs), less-frequent entries (container logs, ERROR lines) can be pushed past the fetched page and never appear. + +The page is also a single `useQuery` that fetches at most `limit` rows. There is no way to browse older logs. + +The **Timeline** (agent events) has the same single-page limitation. + +## Goal + +Establish a consistent pattern for streaming views (log-like lists + event feeds): + +1. **Multi-select server-side filters** joined with `OR` — the server decides what's included, so no category is ever silently starved. +2. **Infinite scroll** via cursor pagination — user scrolls down to load older entries. +3. **Top-gated auto-refetch** — polling only runs while the user is viewing the newest entries; when they scroll away to inspect history, polling pauses so their viewport cannot shift under them. +4. **Client-side text filter** only — text search filters pages already loaded without re-querying. + +Apply this pattern to: +- `AgentHealth.tsx` — Application Log + Timeline +- `AgentInstance.tsx` — Application Log + Timeline +- Shared components/hooks so future streaming views reuse the same primitives. + +Out of scope for infinite scroll (bounded data, don't need it): +- `LogTab.tsx` — per-exchange logs (bounded, one exchange, already capped at 500) +- `StartupLogPanel.tsx` — container logs during a single deployment startup + +These two keep `useLogs` / `useStartupLogs` as-is. They will benefit automatically if `LogEntry.source` gets pushed through to `LogViewer` for badge rendering (already supported by DS). + +## Design + +### 1. Backend: multi-value `source` + +The log endpoint already supports comma-split `level`. Mirror that for `source`. + +**`cameleer-server-core/.../search/LogSearchRequest.java`** +- Change `String source` → `List sources` (default empty list). + +**`cameleer-server-app/.../controller/LogQueryController.java`** +- `@RequestParam(required = false) String source` stays. +- Parse same way as `level` (comma-split, trim, drop blanks) into `List`. + +**`cameleer-server-app/.../search/ClickHouseLogStore.java`** +- Replace: + ``` + if (request.source() != null && !request.source().isEmpty()) { + baseConditions.add("source = ?"); + baseParams.add(request.source()); + } + ``` + with a list-driven `source IN (?, ?, …)` built when `request.sources()` is non-empty. + +No schema change. No migration. + +### 2. Backend: cursor pagination for agent events + +Agent events currently return `List` with `limit` (default 50). No cursor. To support infinite scroll we need stable desc ordering + a cursor. + +Events have no stable ID (the ClickHouse repository synthesizes `id = 0`). The stable ordering is `(timestamp DESC, instance_id ASC)`. Cursor encodes that tuple as an opaque base64 string, matching how `ClickHouseLogStore` builds log cursors. + +**`cameleer-server-core/.../agent/AgentEventService.java` / `AgentEventRepository.java`** +- Add overload `queryEvents(appId, agentId, env, from, to, cursor, limit)` returning a `AgentEventPage { data, nextCursor, hasMore }`. +- Keep the existing un-cursored `queryEvents` if used elsewhere (audit usage; if not, delete it). + +**`cameleer-server-app/.../storage/ClickHouseAgentEventRepository.java`** +- New method: on cursor, decode `(cursorTs, cursorInstance)` and add `(timestamp, instance_id) < (?, ?)` predicate. +- Always `ORDER BY timestamp DESC, instance_id ASC`. +- Always fetch `limit + 1` rows; last row determines `hasMore` + `nextCursor`. + +**`cameleer-server-app/.../controller/AgentEventsController.java`** +- Accept optional `?cursor=`; return `AgentEventPageResponse { data, nextCursor, hasMore }` instead of a bare list. + +### 3. Shared UI primitives + +Two thin primitives. No new design-system package; live under `ui/src/components/` and `ui/src/hooks/` so they're reused from the app only. + +**`ui/src/hooks/useInfiniteStream.ts`** — thin `useInfiniteQuery` wrapper: + +```ts +interface StreamPage { data: T[]; nextCursor: string | null; hasMore: boolean } + +interface UseInfiniteStreamArgs { + queryKey: unknown[]; + fetchPage: (cursor: string | undefined) => Promise>; + enabled?: boolean; + isAtTop: boolean; // drives refetchInterval on/off + refetchMs?: number; // default 15000 +} + +interface UseInfiniteStreamResult { + items: T[]; + fetchNextPage: () => void; + hasNextPage: boolean; + isFetchingNextPage: boolean; + isLoading: boolean; + refresh: () => void; // invalidates and re-fetches from page 1 +} +``` + +Internals: wraps `useInfiniteQuery` with `getNextPageParam: (last) => last.hasMore ? last.nextCursor : undefined`, flattens pages, exposes `refresh` that calls `queryClient.invalidateQueries(queryKey)`. + +**`ui/src/components/InfiniteScrollArea.tsx`** — scrollable container: + +```tsx +interface InfiniteScrollAreaProps { + onEndReached: () => void; // call fetchNextPage + onTopVisibilityChange: (b: boolean) => void; + isFetchingNextPage: boolean; + hasNextPage: boolean; + maxHeight?: number | string; + children: ReactNode; +} +``` + +Renders: +``` +
+
// IntersectionObserver -> onTopVisibilityChange + {children} +
// IntersectionObserver -> onEndReached + {isFetchingNextPage && } + {!hasNextPage && } +
+``` + +A single `IntersectionObserver` per sentinel with `rootMargin: '100px'` for the bottom so the next page is prefetched before the user hits the literal bottom. + +### 4. New hooks + +**`ui/src/api/queries/logs.ts`** + +Add `useInfiniteApplicationLogs`: + +```ts +export function useInfiniteApplicationLogs(args: { + application?: string; + agentId?: string; + sources?: string[]; // multi-select, server-side + levels?: string[]; // multi-select, server-side + exchangeId?: string; + isAtTop: boolean; +}): UseInfiniteStreamResult +``` + +Under the hood: cursor-paginated calls to `/environments/{env}/logs`; `fetchPage(cursor)` sets `source=a,b`, `level=ERROR,WARN`, and time range from the global filter store. + +Keep `useLogs` and `useApplicationLogs` (the existing single-page wrapper) so `LogTab` and `StartupLogPanel` remain untouched. Mark `useApplicationLogs` internally as "bounded consumers only" via a short TSDoc line. + +**`ui/src/api/queries/agents.ts`** + +Add `useInfiniteAgentEvents`: + +```ts +export function useInfiniteAgentEvents(args: { + appId?: string; + agentId?: string; + isAtTop: boolean; +}): UseInfiniteStreamResult +``` + +Existing `useAgentEvents` is only used in `AgentHealth`/`AgentInstance` for the Timeline. It gets replaced by the new hook — no other consumers, no backwards-compat shim. + +### 5. Page wiring — `AgentHealth.tsx` and `AgentInstance.tsx` + +**Application Log block:** + +- State + - `logSources: Set` (new, replaces `logSource: string`). + - `logLevels: Set` (unchanged shape, now passed through as `levels`). + - `logSearch: string` (unchanged, client-side). + - `logSortAsc: boolean` (unchanged). + - `isLogAtTop: boolean` (new, from `InfiniteScrollArea`). + +- Data + - `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteApplicationLogs({ application: appId, agentId, sources: [...logSources], levels: [...logLevels], isAtTop: isLogAtTop });` + +- Map → `LogEntry[]` + - Same mapping as today plus `source: l.source ?? undefined` so `LogViewer`'s source badge lights up. + +- Client-side + - Text filter (`logSearch`) applied after flattening. + - Sort reversal (`logSortAsc`) applied after. + +- ButtonGroup wiring + - `` + - Level `` unchanged (`value={logLevels}`), but `onChange` now implicitly re-queries. + - A "Clear" button next to each group when the set is non-empty (source has 3 options; still low-cost and symmetric with level). + +- Layout + - `` (no `maxHeight`) wrapped in ``. + +**Timeline block:** + +- `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteAgentEvents({ appId, isAtTop: isTimelineAtTop });` +- Map to `FeedEvent[]` (same transform as today). +- `` inside ``. +- Timeline has no filters today; none added in this spec. + +**Refresh buttons:** +Both log and timeline `` buttons call `refresh()` from the stream hook. `logRefreshTo`/`eventRefreshTo` states are retired. + +### 6. Auto-refetch gating semantics + +- `InfiniteScrollArea` fires `onTopVisibilityChange(true)` when the top sentinel is fully in view, `false` when it scrolls out. Implemented via `IntersectionObserver` with threshold `1.0`. +- `useInfiniteStream` reads `isAtTop` and sets `refetchInterval: isAtTop ? refetchMs : false`. Tanstack re-evaluates on every render, so toggling the prop takes effect on the next cycle. +- Manual `refresh()` always works and scrolls back to top (scroll reset is owned by the page, calling `scrollRef.current?.scrollTo({ top: 0 })` after the refresh settles — wired in `AgentHealth` / `AgentInstance`). +- No viewport-preservation logic. The contract "refetch must not move the user's viewport" is satisfied by disabling refetch while the user is scrolled away. + +### 7. Bounded views — minimal changes + +- `LogTab.tsx`: map `e.source` into the rendered `LogEntry` so source badges appear. No other changes. +- `StartupLogPanel.tsx`: no change; startup-log rows are container-sourced by definition. + +### 8. OpenAPI regeneration + +After controller/DTO changes: +``` +cd ui && npm run generate-api:live +``` +Commit the resulting `openapi.json` + `schema.d.ts` in the same change. TypeScript will surface any SPA call sites that need adjustment; fix all of them before testing in the browser. + +### 9. `.claude/rules/` updates + +- `app-classes.md` — `LogQueryController` entry: add "multi-value `source` (comma-split)"; `AgentEventsController` entry: add "cursor-paginated, returns `{ data, nextCursor, hasMore }`". +- `ui.md` — add entry for `InfiniteScrollArea` component and note the "streaming views use `useInfiniteStream` + `InfiniteScrollArea`" convention. + +## Testing + +- **Backend unit**: `ClickHouseLogStoreTest` for `source IN (...)` predicate (single, multi, empty). `ClickHouseAgentEventRepositoryTest` for cursor ordering and `hasMore` boundary. +- **Controller**: `LogQueryControllerTest` (existing) — add `source=app,container` case. `AgentEventsControllerTest` — cursor round-trip. +- **UI manual smoke**: scroll long log streams, mix source filters, verify auto-refetch toggles at top, confirm text filter stays local, confirm refresh resets scroll. + +## Risks & mitigations + +| Risk | Mitigation | +|---|---| +| Event cursor using `(timestamp, instance_id)` — collisions if two events share both | In practice events are recorded one-at-a-time per instance; tuple is stable. Tie-breaker fine as both fields are indexed. | +| `useInfiniteQuery` refetch reloads all pages → visible flicker | Use `placeholderData: (prev) => prev` so stale data stays during refetch. Tested pattern already used by `useApplicationLogs`. | +| IntersectionObserver on the top sentinel fires both on scroll and on list grow | Debounce via React state, not a ref — state updates coalesce per render. | +| DS `LogViewer` may virtualize internally, hiding sentinels | `LogViewer` today has no virtualization (DS 0.1.49 props are minimal). If that changes, move sentinels into the outer scroll container (already the plan). | +| Level client→server change breaks existing users with saved filter state | No persisted filter state exists for log levels; this is in-memory only. | + +## Decision log + +- **Server-side for source and level, client-side for text.** Confirmed during design — flooding argument applies symmetrically to source and level. +- **Gate auto-refetch on top visibility** instead of preserving viewport. Rationale: viewport preservation requires stable row IDs (logs have timestamp collisions) and is fragile under virtualization. Gating is simple, predictable, and meets the "user never loses their line" constraint. +- **Bounded log views (LogTab, StartupLogPanel) keep single-page hooks.** Infinite scroll isn't useful for capped data; shared primitives are available if they ever need it. +- **Agent events get cursor pagination**, matching logs. Single-source-of-truth pattern for streaming lists.