Make agent failures inspectable
When an agent gives a wrong answer, the final response rarely explains whether the problem came from the model, a tool, or a delegated task. OrcView combines a Python agent runtime with a live dashboard so developers can inspect those steps, their costs, and the condition that stopped the run.
An explicit runtime, from execution to replay
The runtime uses four states: Start → Reasoning ⇄ Acting → Done. Reasoning requests the next model response; Acting executes requested tools and returns their results to the model. State transitions are emitted by the execution loop itself, so the dashboard follows the control flow that actually ran.
A child agent implements the same callable interface as a tool. Its events carry a sub-run identifier through the parent’s event stream, while the parent model receives the child’s answer rather than its internal trace. This keeps delegation and ordinary tool use on one execution path and lets the dashboard drill into child runs through one server-sent event connection.
Each run captures the model, prompt, enabled tools, and execution limits at startup. The recorder collects and forwards events in memory. Local SDK operators can explicitly enable SQLite recording with record_runs=True; it is disabled by default. The public dashboard isolates temporary configuration/history per browser session instead of sharing a database.
Opening a recorded child run filters events already loaded in the browser and feeds them through the same event renderer. This avoids another model call or server request for the drill-down.
Runtime and delegation source · Event stream and history endpoints · Replay renderer
Controls at the boundary where failures occur
| Failure | Implemented control | Boundary |
|---|---|---|
| Repeated reasoning or tool use | Default 12-cycle ceiling; optional completed-tool-call limit checked before the next tool executes | Bounds execution, without establishing answer correctness |
| Delegation returns to an ancestor | Agent labels are checked against an immutable ancestor set before a model call | Labels must uniquely identify agents; this is delegation-cycle detection, not detection of semantically repeated answers |
| Malformed final output | Optional strict Pydantic schema validation with field-level error events | Validates the final JSON response, not intermediate tool messages or factual truth |
| Transient service failure | Opt-in retry policy with bounded attempts and exponential backoff; each attempt and delay is traced | Permanent errors are surfaced; clients must configure their own network deadlines |
| Browser disconnect | Cooperative cancellation signal checked between reasoning steps | Does not interrupt an in-flight external operation |
The ancestor set is copied when delegation extends the chain, rather than shared as a mutable list. Independent branches therefore cannot overwrite each other’s ancestry. Retry and validation policies remain explicit application choices instead of silently changing every agent’s behavior.
Five controlled failures
The saved deterministic fault suite passed 5/5 injected scenarios using fake model responses. These checks isolate enforcement and event recording; they do not measure production reliability or model accuracy.
| Injected scenario | Observed behavior |
|---|---|
| Malformed structured output | One validation failure recorded; invalid output rejected |
| Timeout | First attempt failed; traced retry succeeded on attempt two |
| HTTP 429 | First attempt failed; traced retry succeeded on attempt two |
| A → B → A delegation cycle | One cycle error traced; parent continued and returned “cycle handled” |
| Two tool requests with a one-call limit | Only the first tool executed; guardrail and stop error recorded |
In the cycle case, A delegates to B, and B requests A. The ancestor check rejects that request before another model call. B emits the error and closes its run; A receives the child result and produces its final response. The saved event sequence connects the originating tool call, nested failure, and parent continuation instead of reducing the whole interaction to a single error message.
The accompanying microbenchmark measured 115.3 ns of incremental recording overhead per event over a direct callback, using nine repeats of 250,000 events. This isolates appending and forwarding an event in-process; database writes, dashboard rendering, network traffic, and model latency are excluded.
Saved fault outcomes and overhead measurements · Reproduction script
Task-evaluation summary
Routine battery: 15/15 deterministic value checks and 15/15 model-judge checks passed in the saved run.
Hard computable tier: 10/10 strict value checks in the saved run.
Routine median latency: 6.0 seconds; range 3.2–15.2 seconds.
Routine run cost: $0.2489 for 15 tasks; model pricing is time-dependent.
Task difficulty and tool limits
| Layer | What it measures | Observed | Interpretation |
|---|---|---|---|
| Routine questions | Single-step and modest multi-step aggregation over a fixed 200-row sales CSV | 15/15 deterministic checks; 15/15 model-judge checks | Passed saved battery |
| Hard, computable | Derived percentages, contrasts, medians, and multi-call synthesis available through the tool set | 10/10 strict value checks | Passed saved battery |
| Beyond tool limits | Compound filters, percentiles, and correlation requests unsupported by individual tools | Raw evaluator (known scoring defect): 2 correct, 1 declined, 3 wrong | Wrong answers on unsupported tasks |
A trace exposed a scoring defect
One beyond-tool result expected Eva, but the agent concluded Bob. The saved grader still marked it correct because “Eva” appeared elsewhere in the answer table. The raw artifact remains available, but the aggregate must be regenerated with a stricter final-answer extractor before it is reported as a result.
Complete answers and tool traces are retained so aggregate scoring can be audited.
What each recording retains
| Behavior | Persisted evidence |
|---|---|
| Tool-use sequence | Ordered model/tool events with inputs, outputs, state, and cycle number |
| Cycle and budget state | Per-run cycle count, token/cost accounting, and stop reason |
| Replay and comparison | Opt-in local recordings or temporary browser-session history for behavioral inspection |
Evaluation artifacts
Routine raw results — Every question, expected value, answer, tool sequence, cycles, latency, cost, and grader outputs.
Hard raw results — Value checks for computable tasks (Tier A) and correct / declined / wrong classifications for beyond-tool tasks (Tier B).
Routine evaluator — Fixed test cases and deterministic grading code.
Hard evaluator — Tests at the limits of the available tools and grading logic for unsupported-task responses.
Evaluation limits
- The saved evaluations use one synthetic 200-row sales dataset and do not establish general data-analysis accuracy.
- Model-judge agreement is secondary to deterministic value checks and can share model biases.
- Latency and cost are properties of one saved run and model configuration, not service-level guarantees.
- The beyond-tool tests show that recording agent behavior can expose failures; it does not itself prevent fabricated answers.