cameleer/docs/superpowers/specs/2026-04-14-simplified-log-forwarding-design.md

# Simplified Log Forwarding Architecture

**Date:** 2026-04-14
**Status:** Approved (conversation design review)

## Problem

The current log forwarding has three buffering layers (BridgeAccess early buffer → LogForwarder deferred exporter → ChunkedExporter) to cover the timing gap between appender registration and server connection. This creates:

- Framework-specific early hooks: `SpringContextAdvice` hooks `AbstractApplicationContext.refresh()` for Spring apps; `CameleerHookInstaller.preInstall()` has a fallback path for non-Spring apps
- `BridgeAccess` early buffer (5000 entries) with synchronized drain logic and classloader reflection
- `LogForwarder` deferred exporter pattern (no-arg constructor, `setExporter()` called later)
- `LogForwardingConsumer` wrapper class to avoid ByteBuddy lambda inlining issues
- Cross-classloader complexity in `SpringContextAdvice` (loads LogForwarder on system CL via reflection)

Despite this complexity, **bootstrap failures before CamelContext.start() are still not captured** — the server connection only happens in `postInstall`, so if the app dies during Spring context initialization, buffered logs die with the process.

## Decision

1. **Move server connection to `preInstall`** — connect before routes start, so the exporter is live when the appender is registered. No deferred pattern needed.
2. **External infrastructure handles pre-agent logs** — Docker/K8s logging drivers, DaemonSet shippers, or kubelet events capture container lifecycle before the JVM starts. The server team will implement this.
3. **Remove all early buffering machinery** — BridgeAccess buffer, deferred exporter, SpringContextAdvice log code, LogForwardingConsumer.
4. **Unified appender registration path** — same code for Spring Boot, Quarkus, and Plain Java, executed in preInstall/configure.

### Why preInstall instead of premain?

The original conversation explored moving the server connection to `premain()`. However, the agent uses an `AgentClassLoader` (child-first) that creates a separate class namespace from the system classloader. Objects created in `premain()` (system CL) cannot be passed to `CameleerHookInstaller` (AgentClassLoader) without complex cross-classloader reflection using only primitives. Since `preInstall` already fires before any route processes exchanges (it runs at the start of `CamelContext.start()`), it meets the requirement of "fully initialized before the first route executes" without the classloader complexity.

## Architecture

### Three-Phase Log Coverage Model

| Phase | Coverage | Mechanism |
|-------|----------|-----------|
| Pre-JVM (container fails before Java runs) | Server team / infra | kubelet events, Docker logging drivers, DaemonSet shipper |
| JVM start → preInstall (framework bootstrap) | Not captured by agent | Infra-level capture; acceptable gap |
| preInstall onward (routes executing) | Agent | In-agent appender with structured log forwarding |

### New Startup Sequence (Agent Mode)

```
premain()
  → install CamelContext transformer (existing)
  → install SendDynamicAware transformer (existing)
  → [REMOVED: Spring AbstractApplicationContext transformer]

CamelContext.start() intercepted → preInstall()
  → install InterceptStrategy (existing)
  → earlyConnect: ServerConnection + register (minimal) + ChunkedExporter
  → register log appender on root logger (unified, all frameworks)
  → create LogForwarder WITH live exporter
  → set bridge handler on LogEventBridge
  → ✅ log forwarding LIVE before any route processes an exchange

CamelContext.start() completes → postInstall()
  → create EventNotifier (existing)
  → PostStartSetup: diagrams, metrics, re-register with full info, heartbeat, SSE
```

### New Startup Sequence (Extension Mode)

```
@PostConstruct (CameleerConfigAdapter)
  → set system properties from Quarkus config
  → [REMOVED: appender registration and earlyLogForwarder]

configure() (CameleerLifecycle - before CamelContext.start())
  → earlyConnect: ServerConnection + register (minimal) + ChunkedExporter
  → create collector + InterceptStrategy + EventNotifier (existing)
  → register log appender on root logger
  → create LogForwarder WITH live exporter
  → set bridge handler
  → ✅ log forwarding LIVE

CamelContextStartedEvent
  → PostStartSetup: diagrams, metrics, re-register, heartbeat, SSE
```

### Server-Side Cutover

The `logForwarding` capability in the registration/heartbeat capabilities map signals when agent log forwarding is active. The server team can use this to stop ingesting infra-level logs for that instance, avoiding duplicates.

## Changes

### New: `ServerSetup.earlyConnect(CameleerAgentConfig)`

Minimal server registration before routes start. Returns `EarlyConnection(ServerConnection, ChunkedExporter)` or null on failure.

- Creates `ServerConnection` with endpoint + auth token
- Registers with: instanceId, applicationId, environmentId, version, empty routeIds, minimal capabilities
- Creates `ChunkedExporter` wired to the connection
- On failure: logs warning, returns null (agent falls back to `LogExporter`)

### New: `ServerSetup.setupLogForwarding(config, exporter, appClassLoader)`

Shared method for both agent and extension to register the appender and create a LogForwarder in one call.

### Modified: `ServerSetup.connect(ConnectionContext)`

- `ConnectionContext` gains an `earlyConnection` field (replaces `earlyLogForwarder`)
- If `earlyConnection` is present: reuses `ServerConnection` and `ChunkedExporter`, re-registers with full route IDs and capabilities
- If absent: full connection as before (backward compat for edge cases)
- `installLogForwarding()` removed (log forwarding already set up in preInstall)

### Modified: `PostStartSetup.Context`

- `earlyLogForwarder` field replaced by `earlyConnection` (ServerSetup.EarlyConnection)
- `logForwarder` passed separately (already created in preInstall)

### Modified: `LogForwarder`

- No-arg constructor **removed**
- Constructor always requires a non-null `Exporter`
- `setExporter()` **kept** for one case: early connect failed (LogExporter), late connect succeeded (swap to ChunkedExporter)
- Internal queue and flush scheduler unchanged

### Simplified: `BridgeAccess`

- `earlyBuffer` (ConcurrentLinkedQueue) **removed**
- `drainBuffer()` **removed**
- `MAX_BUFFER_SIZE` **removed**
- `forward()` calls handler directly; if no handler, entry is silently dropped
- `setHandler()`, `getHandler()`, `resolveField()` unchanged
- Test utility methods updated

### Removed: `SpringContextAdvice`

- Entire class deleted (only purpose was early log registration)

### Removed: `LogForwardingConsumer`

- Entire class deleted (was only needed by SpringContextAdvice)

### Removed: `SpringContextTransformer`

- Entire class deleted (ByteBuddy transformer for SpringContextAdvice)

### Modified: `CameleerAgent.premain()`

- Remove `AbstractApplicationContext` transformer installation (lines 67-72)

### Modified: `CameleerHookInstaller`

- `preInstall()`: calls `earlyConnect()`, registers appender, creates `LogForwarder` with live exporter, stores `EarlyConnection` for postInstall
- `postInstall()`: no longer picks up `SpringContextAdvice.earlyLogForwarder`; passes `EarlyConnection` + `logForwarder` to `PostStartSetup`

### Modified: `CameleerConfigAdapter` (Extension)

- Remove `earlyLogForwarder` field and `getEarlyLogForwarder()`
- Remove appender registration from `@PostConstruct`

### Modified: `CameleerLifecycle` (Extension)

- `configure()`: calls `earlyConnect()`, registers appender, creates `LogForwarder`
- `onCamelContextStarted()`: passes `EarlyConnection` + `logForwarder` to `PostStartSetup`

## Test Changes

- `LogForwarderTest`: remove `deferredExporter_buffersUntilExporterSet` test (deferred pattern no longer exists)
- `BridgeAccessTest`: remove buffer tests (`forward_buffersWhenNoHandler`, `forward_drainsBufferWhenHandlerSet`, `forward_bufferCapped`); add test for silent drop when no handler
- Remaining tests unchanged (they already pass exporter to constructor)

## Constraints

- Must not steal the customer's log stream — appender adds alongside existing appenders, never replaces
- Must work with Datadog, Elastic, and other APM agents that also add appenders
- Extension path (Quarkus native, no agent) remains separate but follows the same simplified pattern
- If server is unreachable in preInstall, graceful fallback to LogExporter; postInstall retries connection