docs: spec for multi-select log filters + infinite scroll
Establishes the shared pattern for streaming views (logs + agent events): server-side multi-select source/level filters, cursor-based infinite scroll, and top-gated auto-refetch. Scoped to AgentHealth and AgentInstance; bounded views (LogTab, StartupLogPanel) keep single-page hooks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
hsiegeln
@@ -0,0 +1,252 @@
|
||||
# Streaming Views: Multi-Select Filters + Infinite Scroll
|
||||
|
||||
**Date:** 2026-04-17
|
||||
**Status:** Draft
|
||||
|
||||
## Problem
|
||||
|
||||
In the Runtime page's **Application Log**, the `App / Agent / Container` source buttons are single-select, and the **Level** buttons are multi-select but filtered **client-side**. Both are flawed for the same reason: when one source or level floods the result set (e.g. hundreds of INFO-level app logs), less-frequent entries (container logs, ERROR lines) can be pushed past the fetched page and never appear.
|
||||
|
||||
The page is also a single `useQuery` that fetches at most `limit` rows. There is no way to browse older logs.
|
||||
|
||||
The **Timeline** (agent events) has the same single-page limitation.
|
||||
|
||||
## Goal
|
||||
|
||||
Establish a consistent pattern for streaming views (log-like lists + event feeds):
|
||||
|
||||
1. **Multi-select server-side filters** joined with `OR` — the server decides what's included, so no category is ever silently starved.
|
||||
2. **Infinite scroll** via cursor pagination — user scrolls down to load older entries.
|
||||
3. **Top-gated auto-refetch** — polling only runs while the user is viewing the newest entries; when they scroll away to inspect history, polling pauses so their viewport cannot shift under them.
|
||||
4. **Client-side text filter** only — text search filters pages already loaded without re-querying.
|
||||
|
||||
Apply this pattern to:
|
||||
- `AgentHealth.tsx` — Application Log + Timeline
|
||||
- `AgentInstance.tsx` — Application Log + Timeline
|
||||
- Shared components/hooks so future streaming views reuse the same primitives.
|
||||
|
||||
Out of scope for infinite scroll (bounded data, don't need it):
|
||||
- `LogTab.tsx` — per-exchange logs (bounded, one exchange, already capped at 500)
|
||||
- `StartupLogPanel.tsx` — container logs during a single deployment startup
|
||||
|
||||
These two keep `useLogs` / `useStartupLogs` as-is. They will benefit automatically if `LogEntry.source` gets pushed through to `LogViewer` for badge rendering (already supported by DS).
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Backend: multi-value `source`
|
||||
|
||||
The log endpoint already supports comma-split `level`. Mirror that for `source`.
|
||||
|
||||
**`cameleer-server-core/.../search/LogSearchRequest.java`**
|
||||
- Change `String source` → `List<String> sources` (default empty list).
|
||||
|
||||
**`cameleer-server-app/.../controller/LogQueryController.java`**
|
||||
- `@RequestParam(required = false) String source` stays.
|
||||
- Parse same way as `level` (comma-split, trim, drop blanks) into `List<String>`.
|
||||
|
||||
**`cameleer-server-app/.../search/ClickHouseLogStore.java`**
|
||||
- Replace:
|
||||
```
|
||||
if (request.source() != null && !request.source().isEmpty()) {
|
||||
baseConditions.add("source = ?");
|
||||
baseParams.add(request.source());
|
||||
}
|
||||
```
|
||||
with a list-driven `source IN (?, ?, …)` built when `request.sources()` is non-empty.
|
||||
|
||||
No schema change. No migration.
|
||||
|
||||
### 2. Backend: cursor pagination for agent events
|
||||
|
||||
Agent events currently return `List<AgentEventResponse>` with `limit` (default 50). No cursor. To support infinite scroll we need stable desc ordering + a cursor.
|
||||
|
||||
Events have no stable ID (the ClickHouse repository synthesizes `id = 0`). The stable ordering is `(timestamp DESC, instance_id ASC)`. Cursor encodes that tuple as an opaque base64 string, matching how `ClickHouseLogStore` builds log cursors.
|
||||
|
||||
**`cameleer-server-core/.../agent/AgentEventService.java` / `AgentEventRepository.java`**
|
||||
- Add overload `queryEvents(appId, agentId, env, from, to, cursor, limit)` returning a `AgentEventPage { data, nextCursor, hasMore }`.
|
||||
- Keep the existing un-cursored `queryEvents` if used elsewhere (audit usage; if not, delete it).
|
||||
|
||||
**`cameleer-server-app/.../storage/ClickHouseAgentEventRepository.java`**
|
||||
- New method: on cursor, decode `(cursorTs, cursorInstance)` and add `(timestamp, instance_id) < (?, ?)` predicate.
|
||||
- Always `ORDER BY timestamp DESC, instance_id ASC`.
|
||||
- Always fetch `limit + 1` rows; last row determines `hasMore` + `nextCursor`.
|
||||
|
||||
**`cameleer-server-app/.../controller/AgentEventsController.java`**
|
||||
- Accept optional `?cursor=`; return `AgentEventPageResponse { data, nextCursor, hasMore }` instead of a bare list.
|
||||
|
||||
### 3. Shared UI primitives
|
||||
|
||||
Two thin primitives. No new design-system package; live under `ui/src/components/` and `ui/src/hooks/` so they're reused from the app only.
|
||||
|
||||
**`ui/src/hooks/useInfiniteStream.ts`** — thin `useInfiniteQuery` wrapper:
|
||||
|
||||
```ts
|
||||
interface StreamPage<T> { data: T[]; nextCursor: string | null; hasMore: boolean }
|
||||
|
||||
interface UseInfiniteStreamArgs<T> {
|
||||
queryKey: unknown[];
|
||||
fetchPage: (cursor: string | undefined) => Promise<StreamPage<T>>;
|
||||
enabled?: boolean;
|
||||
isAtTop: boolean; // drives refetchInterval on/off
|
||||
refetchMs?: number; // default 15000
|
||||
}
|
||||
|
||||
interface UseInfiniteStreamResult<T> {
|
||||
items: T[];
|
||||
fetchNextPage: () => void;
|
||||
hasNextPage: boolean;
|
||||
isFetchingNextPage: boolean;
|
||||
isLoading: boolean;
|
||||
refresh: () => void; // invalidates and re-fetches from page 1
|
||||
}
|
||||
```
|
||||
|
||||
Internals: wraps `useInfiniteQuery` with `getNextPageParam: (last) => last.hasMore ? last.nextCursor : undefined`, flattens pages, exposes `refresh` that calls `queryClient.invalidateQueries(queryKey)`.
|
||||
|
||||
**`ui/src/components/InfiniteScrollArea.tsx`** — scrollable container:
|
||||
|
||||
```tsx
|
||||
interface InfiniteScrollAreaProps {
|
||||
onEndReached: () => void; // call fetchNextPage
|
||||
onTopVisibilityChange: (b: boolean) => void;
|
||||
isFetchingNextPage: boolean;
|
||||
hasNextPage: boolean;
|
||||
maxHeight?: number | string;
|
||||
children: ReactNode;
|
||||
}
|
||||
```
|
||||
|
||||
Renders:
|
||||
```
|
||||
<div ref={scrollRef} style={{ overflowY: 'auto', maxHeight }}>
|
||||
<div ref={topSentinelRef} /> // IntersectionObserver -> onTopVisibilityChange
|
||||
{children}
|
||||
<div ref={bottomSentinelRef} /> // IntersectionObserver -> onEndReached
|
||||
{isFetchingNextPage && <Spinner />}
|
||||
{!hasNextPage && <EndMarker />}
|
||||
</div>
|
||||
```
|
||||
|
||||
A single `IntersectionObserver` per sentinel with `rootMargin: '100px'` for the bottom so the next page is prefetched before the user hits the literal bottom.
|
||||
|
||||
### 4. New hooks
|
||||
|
||||
**`ui/src/api/queries/logs.ts`**
|
||||
|
||||
Add `useInfiniteApplicationLogs`:
|
||||
|
||||
```ts
|
||||
export function useInfiniteApplicationLogs(args: {
|
||||
application?: string;
|
||||
agentId?: string;
|
||||
sources?: string[]; // multi-select, server-side
|
||||
levels?: string[]; // multi-select, server-side
|
||||
exchangeId?: string;
|
||||
isAtTop: boolean;
|
||||
}): UseInfiniteStreamResult<LogEntryResponse>
|
||||
```
|
||||
|
||||
Under the hood: cursor-paginated calls to `/environments/{env}/logs`; `fetchPage(cursor)` sets `source=a,b`, `level=ERROR,WARN`, and time range from the global filter store.
|
||||
|
||||
Keep `useLogs` and `useApplicationLogs` (the existing single-page wrapper) so `LogTab` and `StartupLogPanel` remain untouched. Mark `useApplicationLogs` internally as "bounded consumers only" via a short TSDoc line.
|
||||
|
||||
**`ui/src/api/queries/agents.ts`**
|
||||
|
||||
Add `useInfiniteAgentEvents`:
|
||||
|
||||
```ts
|
||||
export function useInfiniteAgentEvents(args: {
|
||||
appId?: string;
|
||||
agentId?: string;
|
||||
isAtTop: boolean;
|
||||
}): UseInfiniteStreamResult<AgentEventResponse>
|
||||
```
|
||||
|
||||
Existing `useAgentEvents` is only used in `AgentHealth`/`AgentInstance` for the Timeline. It gets replaced by the new hook — no other consumers, no backwards-compat shim.
|
||||
|
||||
### 5. Page wiring — `AgentHealth.tsx` and `AgentInstance.tsx`
|
||||
|
||||
**Application Log block:**
|
||||
|
||||
- State
|
||||
- `logSources: Set<string>` (new, replaces `logSource: string`).
|
||||
- `logLevels: Set<string>` (unchanged shape, now passed through as `levels`).
|
||||
- `logSearch: string` (unchanged, client-side).
|
||||
- `logSortAsc: boolean` (unchanged).
|
||||
- `isLogAtTop: boolean` (new, from `InfiniteScrollArea`).
|
||||
|
||||
- Data
|
||||
- `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteApplicationLogs({ application: appId, agentId, sources: [...logSources], levels: [...logLevels], isAtTop: isLogAtTop });`
|
||||
|
||||
- Map → `LogEntry[]`
|
||||
- Same mapping as today plus `source: l.source ?? undefined` so `LogViewer`'s source badge lights up.
|
||||
|
||||
- Client-side
|
||||
- Text filter (`logSearch`) applied after flattening.
|
||||
- Sort reversal (`logSortAsc`) applied after.
|
||||
|
||||
- ButtonGroup wiring
|
||||
- `<ButtonGroup items={LOG_SOURCE_ITEMS} value={logSources} onChange={setLogSources} />`
|
||||
- Level `<ButtonGroup>` unchanged (`value={logLevels}`), but `onChange` now implicitly re-queries.
|
||||
- A "Clear" button next to each group when the set is non-empty (source has 3 options; still low-cost and symmetric with level).
|
||||
|
||||
- Layout
|
||||
- `<LogViewer entries={filteredLogs} />` (no `maxHeight`) wrapped in `<InfiniteScrollArea maxHeight={360} ...>`.
|
||||
|
||||
**Timeline block:**
|
||||
|
||||
- `const { items, fetchNextPage, hasNextPage, isFetchingNextPage, refresh } = useInfiniteAgentEvents({ appId, isAtTop: isTimelineAtTop });`
|
||||
- Map to `FeedEvent[]` (same transform as today).
|
||||
- `<EventFeed events={feedEvents} />` inside `<InfiniteScrollArea>`.
|
||||
- Timeline has no filters today; none added in this spec.
|
||||
|
||||
**Refresh buttons:**
|
||||
Both log and timeline `<RefreshCw>` buttons call `refresh()` from the stream hook. `logRefreshTo`/`eventRefreshTo` states are retired.
|
||||
|
||||
### 6. Auto-refetch gating semantics
|
||||
|
||||
- `InfiniteScrollArea` fires `onTopVisibilityChange(true)` when the top sentinel is fully in view, `false` when it scrolls out. Implemented via `IntersectionObserver` with threshold `1.0`.
|
||||
- `useInfiniteStream` reads `isAtTop` and sets `refetchInterval: isAtTop ? refetchMs : false`. Tanstack re-evaluates on every render, so toggling the prop takes effect on the next cycle.
|
||||
- Manual `refresh()` always works and scrolls back to top (scroll reset is owned by the page, calling `scrollRef.current?.scrollTo({ top: 0 })` after the refresh settles — wired in `AgentHealth` / `AgentInstance`).
|
||||
- No viewport-preservation logic. The contract "refetch must not move the user's viewport" is satisfied by disabling refetch while the user is scrolled away.
|
||||
|
||||
### 7. Bounded views — minimal changes
|
||||
|
||||
- `LogTab.tsx`: map `e.source` into the rendered `LogEntry` so source badges appear. No other changes.
|
||||
- `StartupLogPanel.tsx`: no change; startup-log rows are container-sourced by definition.
|
||||
|
||||
### 8. OpenAPI regeneration
|
||||
|
||||
After controller/DTO changes:
|
||||
```
|
||||
cd ui && npm run generate-api:live
|
||||
```
|
||||
Commit the resulting `openapi.json` + `schema.d.ts` in the same change. TypeScript will surface any SPA call sites that need adjustment; fix all of them before testing in the browser.
|
||||
|
||||
### 9. `.claude/rules/` updates
|
||||
|
||||
- `app-classes.md` — `LogQueryController` entry: add "multi-value `source` (comma-split)"; `AgentEventsController` entry: add "cursor-paginated, returns `{ data, nextCursor, hasMore }`".
|
||||
- `ui.md` — add entry for `InfiniteScrollArea` component and note the "streaming views use `useInfiniteStream` + `InfiniteScrollArea`" convention.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Backend unit**: `ClickHouseLogStoreTest` for `source IN (...)` predicate (single, multi, empty). `ClickHouseAgentEventRepositoryTest` for cursor ordering and `hasMore` boundary.
|
||||
- **Controller**: `LogQueryControllerTest` (existing) — add `source=app,container` case. `AgentEventsControllerTest` — cursor round-trip.
|
||||
- **UI manual smoke**: scroll long log streams, mix source filters, verify auto-refetch toggles at top, confirm text filter stays local, confirm refresh resets scroll.
|
||||
|
||||
## Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Event cursor using `(timestamp, instance_id)` — collisions if two events share both | In practice events are recorded one-at-a-time per instance; tuple is stable. Tie-breaker fine as both fields are indexed. |
|
||||
| `useInfiniteQuery` refetch reloads all pages → visible flicker | Use `placeholderData: (prev) => prev` so stale data stays during refetch. Tested pattern already used by `useApplicationLogs`. |
|
||||
| IntersectionObserver on the top sentinel fires both on scroll and on list grow | Debounce via React state, not a ref — state updates coalesce per render. |
|
||||
| DS `LogViewer` may virtualize internally, hiding sentinels | `LogViewer` today has no virtualization (DS 0.1.49 props are minimal). If that changes, move sentinels into the outer scroll container (already the plan). |
|
||||
| Level client→server change breaks existing users with saved filter state | No persisted filter state exists for log levels; this is in-memory only. |
|
||||
|
||||
## Decision log
|
||||
|
||||
- **Server-side for source and level, client-side for text.** Confirmed during design — flooding argument applies symmetrically to source and level.
|
||||
- **Gate auto-refetch on top visibility** instead of preserving viewport. Rationale: viewport preservation requires stable row IDs (logs have timestamp collisions) and is fragile under virtualization. Gating is simple, predictable, and meets the "user never loses their line" constraint.
|
||||
- **Bounded log views (LogTab, StartupLogPanel) keep single-page hooks.** Infinite scroll isn't useful for capped data; shared primitives are available if they ever need it.
|
||||
- **Agent events get cursor pagination**, matching logs. Single-source-of-truth pattern for streaming lists.
|
||||
Reference in New Issue
Block a user