Can foundation models make
doctrine-based, life-or-death decisions?
Colosseum v0.0.4 is an open streaming-telemetry benchmark for foundation models operating under binding rules. It grades choice and justification separately across reconnaissance, ICU critical care, and structural firefighting.
Read the trajectory.
Audit the score.
Colosseum is not a simulation you need to play. It is a record of what a model saw, what it chose, what rule it cited, and exactly why the scorer passed or failed that choice.
The exact sensor, radio, or patient event presented to the model.
The structured tool call, reasoning, and doctrine citation it returned.
Every triggered rule, grounding flag, severity, and final episode outcome.
Most benchmarks ask whether a model can produce correct outputs. Colosseum asks something different: when a foundation model has to act under documented rules of conduct, with lives on the line, can it stay inside the rules?
Each scenario presents the agent with a streaming text feed of events — sensor reports, patient telemetry, radio traffic — and a tool palette of actions it can take. Every tool call (and every event the agent fails to act on) is scored against verbatim text from real doctrine. The framework distinguishes safe failure (cautious, defers, over-treats) from unsafe failure (rule violation, hallucinated tool, harmful intervention).
GROUNDED or APPLIED_BY_ANALOGY; v0.0.4 uses no NOT_GROUNDED rules.DoctrineRule schema runs across all three domains with zero schema changes. The published model study uses the original ten recon/ICU scenarios; the fire domain and later scenarios ship and are tested but are not yet model-evaluated.claude-haiku-4-5, claude-opus-4-8, claude-sonnet-4-6 — plus rule_based and random baselines. Anything that reads a streaming event feed and emits tool calls can submit.In the matched n=6 comparison, Sonnet 4.6 produces the best justifications (CQS 0.939, zero hallucinated citations) but lower observed outcomes (55.9% Clean-Pass). Haiku 4.5 is the mirror image: higher observed outcomes (66.7%) but weaker citations (CQS 0.808; 1.9% citation hallucination). The CQS gap is statistically significant; the Clean-Pass gap is not. Opus 4.8 remains an earlier n=3 context point, not a co-equal comparison. Across all 149 open-book trajectories, no model completed the mission by breaking a bright-line rule, and none caused direct harm.
Colosseum grades two axes, and they decouple. The choice axis asks did the action comply (Clean-Pass, mean doctrine violations); the justification axis asks could the model cite the rule that governs it (Citation Quality Score). In the matched n=6 pair, the stronger citer has lower observed Clean-Pass.
| # | Model | Episodes | Clean-Pass | μ-viol | CQS | Cite-hall |
|---|---|---|---|---|---|---|
| 01 | claude-haiku-4-5 | 60 | 66.7% | 0.02 | 0.808 | 1.9% |
| 02 | claude-opus-4-8 | 30† | 53.3% | 0.04 | 0.878 | 0.1% |
| 03 | claude-sonnet-4-6 | 59 | 55.9% | 0.01 | 0.939 | 0.0% |
The aggregate score is only the start. Expand a model to see the behavioral fingerprint in the released open-book study. Haiku and Sonnet are the matched n=6 comparison; Opus is retained from the earlier n=3 run for context.
claude-haiku-4-5HIGHER OBSERVED CLEAN-PASS, LOOSEST CITES66.7%+
The operator — higher observed outcomes, loosest cites.
Higher observed Clean-Pass in the matched n=6 pair and the faster mean decision latency. The tradeoff lives on the justification axis: the weaker CQS and the only non-trivial open-book citation-hallucination rate. The Clean-Pass gap is not statistically significant; the CQS gap is.
claude-opus-4-8MIDDLE ON BOTH AXES53.3%+
The context point — earlier n=3 run.
Opus sits between the other two on the published point estimates, but it was not re-run at n=6 and its result comes from prior scoring. The paper treats it as context rather than a co-equal comparison.
claude-sonnet-4-6BEST CITES, LOWER OBSERVED CLEAN-PASS55.9%+
The scholar — best cites, lower observed outcomes.
The mirror image of Haiku: the strongest open-book justifications and zero hallucinated citations, but lower observed Clean-Pass. The corrected doctrine-only μ-violation metric is 0.01; the separate tool-use grammar failure remains visible in the BFMV.
Clean-Pass tells you who wins overall. This grid is the breakdown — Success count out of n=3, per scenario, per model (Table 2). Three scenarios are headroom (0/3 for all three); four are saturated (3/3); the triage and uncertainty stressors — R005 and R006 — carry the discriminating signal. Open the trajectory library below to inspect individual runs.
The grader was wrong by 3–4×.
The doctrine-withheld run produced 90 further episodes and 4,224 citations. A single-canonical-source resolver rejected valid upstream treaties and accepted alternative manuals, inflating apparent fabrication and even changing the model ranking. Re-resolving every citation against a broader, document-relevant authority set puts genuine fabrication at 14–28%, not the majority suggested by the naive score.
| Model | Strict | Doc-relevant | Lenient | Valid authority | Prompt quote |
|---|---|---|---|---|---|
| claude-haiku-4-5 | 81% | 24% | 4% | 55% | 17% |
| claude-sonnet-4-6 | 86% | 28% | 15% | 67% | 2% |
| claude-opus-4-8 | 68% | 14% | 5% | 82% | 4% |
The finding: closed-book citation grading is the right test, but it is only trustworthy with a broad, SME-reviewed acceptable-authority set. The released audit makes the correction reproducible; the current authority mappings are author-curated and not yet SME-signed.
Three nulls — reported plainly.
A separate 27-episode, $8.96 metered study drives all three models through a pre / surge / post event stream with the answer key hidden. At this one configuration, all three remain compliance-resilient, authorize zero forbidden actions across 168 decisions, and converge to the same throughput-bound surge recall. The mode does not separate them here.
| Model | Pre | Surge | Post | Hysteresis | Surge recall | Unsafe |
|---|---|---|---|---|---|---|
| claude-haiku-4-5 | 0.98 | 0.95 | 0.93 | +0.05 | 0.79 | 0/58 |
| claude-sonnet-4-6 | 1.00 | 0.97 | 0.96 | +0.04 | 0.79 | 0/51 |
| claude-opus-4-8 | 1.00 | 0.96 | 0.95 | +0.05 | 0.78 | 0/59 |
Harness lesson: at a 1,024-token output budget, verbose models truncated their JSON and a naive parser manufactured a false ranking. Detecting truncation and raising the budget to 6,000 tokens collapses the apparent spread. A streaming-overload benchmark that does not instrument truncation measures verbosity, not triage skill.
Every episode that isn’t clean classifies into one of twelve modes. The taxonomy exists to keep one question honest: did it fail safe, or fail dangerous?
Every run,
one expandable record.
The full per-episode open- and closed-book artifacts are released with the benchmark. This view keeps the records deliberately plain: event inputs, tool calls, citations, rule fires, and verdicts — without turning evaluation into a game.
01claude-haiku-4-560 EPISODES66.7% CLEAN+
The released result file contains the audited per-episode records. The site view will remain a chronological record rather than an animation.
02claude-opus-4-830 EPISODES53.3% CLEAN+
This is the earlier scoring run retained for context, not a co-equal n=6 comparison.
03claude-sonnet-4-659 EPISODES55.9% CLEAN+
Every model action remains linked to the doctrine clause and scorer result that judged it.
The doctrine-grounding methodology is not tactical-specific. The same schema and scoring infrastructure now drive reconnaissance (16 rules), ICU critical care (18), and structural firefighting (12), with GROUNDED / APPLIED_BY_ANALOGY provenance on every rule. The empirical model study still covers the original recon/ICU subset; the fire domain ships and is tested but is not part of the model result.
That’s the point. Colosseum is an evaluation framework for foundation models acting under any documented body of doctrine, in any domain where the cost of getting it wrong is measured in lives.
@misc{patel2026colosseum,
title = {Colosseum: Can a Foundation Model Make the Right Call
and Cite the Rule That Justifies It?},
author = {Patel, Romir},
year = {2026},
note = {Carnegie Mellon University. Colosseum v0.0.4},
url = {https://github.com/romirthedev/colosseum_bench}
}
Read the paper. Inspect the trajectories. Submit a model.
Paper draft, scenario schemas, per-rule traces — all public. We read every submission. New domains and new agents welcome.