cameleer-server/.planning/phases/02-transaction-search-diagrams/02-CONTEXT.md

Phase 2: Transaction Search + Diagrams - Context

Gathered: 2026-03-11 Status: Ready for planning

## Phase Boundary

Users can find any transaction by status, time, duration, correlation ID, or content, view execution detail trees, and see versioned route diagrams linked to transactions. This phase delivers the REST API query layer on top of the ClickHouse data ingested in Phase 1. No web UI (that's v2).

## Implementation Decisions

Search API shape

• Both GET and POST endpoints for search
• GET /api/v1/search/executions supports basic filters: status, timeFrom, timeTo, correlationId
• POST /api/v1/search/executions accepts JSON body for advanced filters: all basic filters + duration range, full-text, per-field text targeting
• Response envelope: { "data": [...], "total": N, "offset": 0, "limit": 50 } — wrapped with metadata for pagination
• Pagination approach: Claude's discretion (offset/limit is the natural fit for ClickHouse)

Full-text search scope

• Extend the ClickHouse schema to store message bodies, headers, and exchange properties — choose a data structure that fits performance, search, and storage requirements
• Store everything the agent sends — no server-side truncation. The agent is configured to control what data it captures, but the server must store all of it
• Substring matching (LIKE '%term%') for full-text search — not token-based only
• Global text parameter searches all text fields at once (error_message, error_stacktrace, bodies, headers)
• Optional per-field targeting: textInBody, textInHeaders, textInErrors for power users who want to narrow scope
• Keep in mind that full-text search may be offloaded to OpenSearch later in the project — design the search service interface to be swappable

Transaction detail

• Nested JSON tree returned by the server — server reconstructs the processor tree from flat storage
• Add depth and parent index arrays to the ClickHouse schema (processor_depths, processor_parent_indexes) — populated at ingestion time from the tree structure
• Verify that the agent sends the required tree structure information for reconstruction
• Exchange snapshot data (body, headers, properties) fetched separately per processor — not inlined in the detail response. Keeps the initial tree response lightweight
• Diagram accessed via separate endpoint (not embedded in detail response) — detail response includes the diagram content hash for linking

Diagram rendering

• Both SVG and JSON layout formats, selected via Accept header content negotiation
  • Accept: image/svg+xml -> SVG rendering
  • Accept: application/json -> JSON layout with node positions/coordinates
  • Default to SVG if no preference
• Functional, clean SVG — not full mockup fidelity. The polished interactive version (glow, animations, execution overlay) is a UI responsibility
• No execution overlay in server-rendered SVG — the UI handles overlay with theme support (dark/light)
• Top-to-bottom node layout flow
• Nested processors (inside for-each, split, try-catch) rendered in swimlanes to highlight nesting/scope
• Reference: cameleer agent repo examples/route-diagram-example.html for visual style inspiration (color-coded node types, EIP icons)

Claude's Discretion

• Pagination implementation details (offset/limit vs cursor)
• ClickHouse schema extension approach for exchange snapshot storage (parallel arrays, JSON columns, or separate table)
• SVG rendering library choice (Batik, jsvg, manual SVG generation, etc.)
• Layout algorithm for diagram node positioning
• Search service abstraction layer design (for future OpenSearch swap)

## Specific Ideas

• "We want a cmd+k type of search in the UI" — see cameleer/examples/cmd-k-search-example.html for the target UX. Key features:
  • Cross-entity search: single query hits Executions, Routes, Exchanges, Agents with scope tabs and counts
  • Filter chips in the input (e.g., route:order prefix filtering)
  • Inline preview pane with JSON syntax highlighting for selected result
  • Keyboard navigation (arrows, enter, tab for filter, # for scope)
• The Phase 2 API should be designed so this cmd+k UI pattern works when the web UI arrives in v2. This means:
  • Consistent response shapes across entity search endpoints
  • Support for cross-entity or parallel entity search
  • Result counts per entity type
  • Text highlighting or match context in results
• "See route-diagram-example.html — it was really liked by our target audience" — the diagram rendering should match the topology style (color-coded nodes by type: blue endpoints, purple EIPs, green processors, red errors, cyan cross-route)
• Nested processors in swimlanes: "for-each loops could be rendered in additional swimlanes to highlight the nesting"

<code_context>

Existing Code Insights

Reusable Assets

• WriteBuffer<T> (core module): Generic buffer with backpressure — reusable for any new buffered operations
• IngestionService (core module): Orchestrates buffer writes — pattern for new service classes
• ClickHouseExecutionRepository: JDBC batch insert pattern with parallel arrays — template for query implementations
• AbstractClickHouseIT: Testcontainers base class — reuse for all Phase 2 integration tests
• ProtocolVersionInterceptor: Request validation pattern — reusable for search request validation

Established Patterns

• Core module holds interfaces + domain logic; app module holds Spring Boot + ClickHouse implementations
• Controllers accept raw String body (for single/array flexibility); services handle deserialization
• JdbcTemplate with BatchPreparedStatementSetter for ClickHouse writes
• Processors stored as depth-first-flattened parallel arrays (needs depth/parent extension for tree reconstruction)
• route_diagrams uses ReplacingMergeTree with content_hash for dedup — already supports content-addressable versioning

Integration Points

• Search endpoints join existing /api/v1/ path structure with ProtocolVersionInterceptor
• route_executions table is the primary query target — needs schema extension for exchange snapshot data
• route_diagrams table already stores definitions — needs query methods and rendering layer
• DiagramRepository already has findByContentHash and findContentHashForRoute — ready for linking

</code_context>

## Deferred Ideas

• Cursor-based pagination (ASRCH-01) — explicitly v2
• Saved search queries (ASRCH-02) — explicitly v2
• Web UI with cmd+k search overlay — v2, but Phase 2 API designed to support it
• Execution overlay on diagrams — UI responsibility (needs theme support)
• OpenSearch for full-text search — evaluate during/after Phase 2 if ClickHouse skip indexes aren't sufficient

---

Phase: 02-transaction-search-diagrams Context gathered: 2026-03-11