FRESH MATCHED RETRIEVAL EXPERIMENT
Which setup makes fewer mistakes?
The same hard trips. Two models. Three ways to supply guidance, with and without a critic.
Loading recorded results…
Blue: Haiku. Coral: Sonnet. Points and thin ranges: estimated probabilities and 95% model-based ranges. Bayesian methods update these estimates from reviewed outcomes. Pale bars: simultaneous 95% confidence bounds across all 12 setups, task error and the loss-exceedance curve. Click a row to inspect it.
Inspect a setup
Inspect a recorded mistake
Which mistakes occurred?
Timing, tokens and cost
What the three setups receive
Methods and evidence
Frozen protocol ↗ · Experiment & results ↗
Proposed losses use synthetic cancellation and rebooking penalties capped at $1,000, before the execution gate. Unnecessary escalation counts as a task error even with no dollar loss. Missing losses span $0–$1,000. Deployment projections require traffic and untested-failure assumptions.