HOW TO USE THIS
From an agent workflow to a risk estimate.
Upload your agent workflow and reviewed logs. Compare safeguards, estimate losses and see how much uncertainty remains.
1. Upload a description of your agents
List agents, tools, handoffs, trusted records and safety checks in the JSON template. Set a maximum loss per task before testing. RiskGraph draws the workflow.
The preview draws your workflow locally. It does not run uploaded code.
Run a custom simulation through an adapter to your own sandbox. Risk estimates require reviewed outcomes from those tests.
2. Run the workflow in a sandbox
Use test tools that record proposed actions without making bookings or payments. Supply tasks and trusted records, then score policy violations and losses independently.
| What to supply | What to record |
|---|---|
| Agents, prompts and allowed tools | A fixed system version and one ID per complete episode |
| Current policy, records and challenging tasks | Agent proposals, handoffs, memory retrievals and critic decisions |
| Loss model and a defensible exposure cap | Independently reviewed potential and final losses in cents |
| Baseline and optional safeguard variants | The same task under each setup, including corrections and regressions |
Use the built-in travel and refund examples, or run your own sandbox adapter and export reviewed episodes in the trace format below.
The public demo replays saved results. Live model evaluations require operator access.
3. Add reviewed activity logs
Open Import reviewed traces, choose your JSON file, and select “Risk estimates · whole trace set”. Start with a downloaded example to check the format.
- Use source: observed for real activity, simulation for sandbox outcomes, or illustration for hand-authored demos.
- Keep one row per complete business episode. Link agent steps with event IDs and parent IDs; steps are not additional independent samples.
- Review the outcome independently, add a review reference, and supply potential and executed losses in integer cents. Keep missing outcomes null.
- If comparing safeguards, include recorded matched outcomes for memory, critic and memory plus critic. The toggles compare these recorded outcomes.
- For observed data, in the same trace page choose Explore → Operator access and verify access before requesting analysis. “Save privately” is optional; keep its batch ID to reopen it.
Remove credentials and unnecessary personal data before import. Preview keeps trace text in the tab; estimation sends numeric labels and settings to RiskGraph. Private saving sends the full batch to the server and requires operator access.
4. Get a report for your deployment decision
| Output | What it answers |
|---|---|
| Failure and mistake probabilities | How often did a reviewed mistake or positive-loss proposal occur, with statistical uncertainty? |
| Chance of loss above your chosen amount | How likely is a loss above the dollar threshold you select? |
| Expected loss and bounds | What is the mean dollar exposure under the selected evidence, cap and sampling assumptions? |
| Matched safeguard comparisons | Which mistakes did memory or a critic prevent, and which did they introduce? |
| Agent topology and reviewed events | Where did the proposal, review, gate and outcome occur? |
| Untested traffic and missing failure types | How do traffic weights, real/simulated evidence and an explicit undiscovered-failure component change the projection? |
Bayesian methods update mistake probabilities from reviewed outcomes. Combining these probabilities with cost ranges gives expected loss and tail-risk estimates. The uncertainty ranges reflect sample size and missing outcomes.
Uploaded trace reports use reviewed loss labels. Registered experiments also show task-error categories. Real traces and simulations use separate models; source weights combine their estimates. An adjustable untested-traffic share covers failures outside the tested scenarios.