PRIVATE INFORMATION & KNOWLEDGE QUALITY
What could an assistant reveal or invent?
Compare privacy leaks, unsupported claims and unreliable citations. Estimate their probabilities, then explore possible costs.
Loading recorded fictional cases…
Synthetic records, invalid test identifiers and nonfunctional credentials. Cost ranges are planning assumptions.
Each graph row: Haiku is blue and above; Sonnet is rust and below.
Which mistake types occurred?
Dots show Bayesian probability estimates; thin lines show 95% ranges. Pale bars are simultaneous confidence bounds across the comparisons. Small samples produce wide ranges, including after zero observed errors.
Set the cost and unseen-risk assumptions
Estimated cost = sum of (mistake probability × possible cost of that mistake)
Severity = response expenses + hourly rate × response hours + optional value × impact score. The impact score is an ordinal scenario input; assigning dollars to it is a sensitivity choice. Default dollar value is zero.
For severity support [l, u], tail risk lies between Σ p × 1(l > threshold) and Σ p × 1(u > threshold). Unseen failures add an explicit sensitivity component with an assumed $30,000 cap. This does not assign a measured price to privacy, trust or reputation.
Compare every setup
Inspect recorded errors
Where review made a task worse
Scoring checks known test identifiers and structured claims against source records. Open-ended text and new obfuscations need additional review.
How the calculation works
Reviewed mistakes update a Bayesian probability model. Your cost ranges convert those probabilities into expected cost and tail-risk bounds. Simultaneous confidence bounds account for comparing several setups.
The estimates describe the selected test cases. Deployment projections also require representative traffic and cost assumptions.