Can a MUD evaluate LLMs? A $99 proof-of-concept in a persistent text world — and a case study in LLM-judge rank distortion
CrucibleBench is a proof-of-concept evaluation environment that places language models in a compact, persistent text world (a 12-room, 4-NPC MUD with rule-based mechanics) and scores their behavior over 50-turn runs with hidden social objectives. We evaluate 13 models from eight providers across 650 runs at a total billed cost of $99.59, on four behavioral dimensions: Goal Pursuit, Social Adaptation, World Grounding, and Strategic Sophistication.
Our central finding concerns measurement rather than model ranking: the LLM-classifier components inside composite behavioral scores can materially reorder model rankings while aggregate reliability metrics show nothing. Excluding the two classifier-dependent dimensions yields a Spearman rank correlation of ρ = 0.70 with the full composite; Gemini 3.1 Pro falls six positions, and three models change tier. Per-model agreement between the classifier and an independent judge ranges from 21.7% to 84.8%, non-uniform instability that the aggregate κ = 0.04 does not expose. The most-affected model shares a model family with the classifier.
Secondary findings: the environment sharply separates the open-weights floor model from the twelve frontier models, with interpretable failure modes; an episodic-reset ablation suggests persistent state amplifies between-model discrimination; and inference cost did not predict behavioral quality. These results justify a Phase 2 study with human baselines, multi-judge consensus, and block-aware analysis; they do not validate CrucibleBench as a benchmark.
Three results worth your attention
-
01
Judge dependence changes rankings
The classifier-minimized subtotal produced stronger differentiation (η² = 0.931 vs 0.847; 32 vs 23 of 78 distinguishable pairs) and a materially different ranking. The lesson is not that LLM judges are useless; it is that judge-mediated scoring needs per-subject audits and ranking-stability checks.
-
02
Task-specific behavioral specialization
Mistral Large 3 achieves 80% on identification objectives and 0% on trust-building; Claude Haiku 4.5 shows the same pattern. Only GPT-5.4 exceeds 50% on both. Single-score model selection masks capability gaps.
-
03
Dialogue looping dominates
Dialogue looping occurred in 14–66% of frontier runs, the dominant failure mode for every model. Wrong-room interaction and exploration paralysis separated the open-weights floor sharply and appeared selectively among frontier models.
Model performance under both scoring configurations
650 scored runs across 13 models from 8 providers. All scores on a 1–5 rubric scale; 95% confidence intervals via scenario-cell bootstrap (10,000 iterations). Sorted by classifier-minimized subtotal.
| Model | Classifier-min. | 95% CI | Full score | 95% CI |
|---|---|---|---|---|
| Claude Sonnet 4.6 | 4.04 | [3.92, 4.16] | 3.89 | [3.75, 4.03] |
| DeepSeek R1 | 4.00 | [3.88, 4.12] | 3.85 | [3.70, 4.00] |
| Claude Opus 4.6 | 3.93 | [3.82, 4.04] | 3.93 | [3.82, 4.07] |
| GPT-5.2 | 3.91 | [3.78, 4.04] | 3.88 | [3.70, 4.08] |
| GPT-5.4 | 3.88 | [3.76, 4.00] | 4.07 | [3.95, 4.18] |
| Qwen 3.5 397B | 3.81 | [3.72, 3.90] | 3.81 | [3.74, 3.89] |
| Claude Haiku 4.5 | 3.80 | [3.68, 3.92] | 3.88 | [3.77, 4.01] |
| GPT-5.3 Chat | 3.73 | [3.60, 3.86] | 3.72 | [3.59, 3.85] |
| Gemini 3.1 Pro | 3.71 | [3.54, 3.88] | 3.91 | [3.73, 4.11] |
| Grok 4 | 3.61 | [3.46, 3.76] | 3.48 | [3.32, 3.64] |
| DeepSeek V3.2 | 3.60 | [3.52, 3.70] | 3.61 | [3.54, 3.69] |
| Mistral Large 3 | 3.44 | [3.32, 3.56] | 3.69 | [3.59, 3.78] |
| OLMo 3.1 32B | 2.01 | [1.86, 2.16] | 1.93 | [1.80, 2.09] |
The classifier-minimized subtotal (World Grounding + Social Adaptation) has reduced but not eliminated classifier dependence. Gemini 3.1 Pro shows the largest rank change between configurations (six positions), consistent with classifier over-classification inflating its Goal Pursuit score, and it shares a model family with the classifier. Rankings are exploratory.
Cost and behavioral quality are decoupled
Regressing total score on log cost across the twelve frontier models yields β = 0.007 (p = 0.85). The most expensive model scored below median under both configurations; the strongest cost-performance models sit near the top of their tiers.
Full-composite score (bar) against billing-verified cost per run. Grok 4 alone accounted for 42.2% of total experiment spend.
A complete objective, verbatim
GPT-5.4, run 05, seed 20260496, gain_watch_trust. The model finds a signet ring where its owner works, returns it, and asks for exactly what the objective requires, in 14 of 50 turns.
Annotation
This run shows the pattern behind GPT-5.4's 68% success rate: a straight line from the city gate to the objective-relevant NPC in four turns, then context-responsive gifting: the signet ring was found in the barracks court, so the model returned it to its most plausible owner rather than brute-forcing items.
Every subsequent exchange couples a trust-building frame ("I mean to serve lawfully") with an explicit recommendation request: the exact completion criterion. The model treats the 50-turn budget as a scarce resource: the run terminates on success at turn 14 at a billed cost of $0.027.
Contrast this with the floor model's failure cascade on the home page: same world, same objective family, categorically different behavior. This gap, legible in plain text, is what the benchmark is for.
Substantive, not boilerplate
-
01
Classifier reliability
Two dimensions route through a single classifier with weak-to-poor reliability (intent κ = 0.11; probe κ = 0.04) and per-model agreement spanning 21.7–84.8%. Any classifier-dependent ranking is provisional pending multi-classifier consensus.
-
02
No human baseline
Whether the Social Adaptation ceiling (no model above 3.42) reflects model weakness, task difficulty, or rubric construction is unresolved without human players.
-
03
Compact environment
12 rooms, 4 NPCs, 50 turns maximizes scenario control but may not capture behaviors that only emerge in larger worlds. Once public, scores are gameable by scaffolds tuned to the fixed map, so Phase 2 requires a hidden-eval split.
-
04
Exploratory statistics
Effective sample size is closer to 10 scenario cells per model than 50 runs; effects below d = 0.79, including the Top/Mid tier gap, are below detection. Roughly 175 runs per model/objective would target within-tier effects, but that powered expansion is deferred beyond the current Phase 2 instrument-validation plan.
Fourteen limitations are enumerated in Section 9 of the paper, including judge-mediated world dynamics, judge circularity, provider routing, and the single open-weights anchor.
Everything ships: transcripts, code, and the bill
Technical paper (PDF)
Full methodology: scenario design, scoring rubrics, classifier audit, statistical framework, scripted-policy baselines, and annotated transcripts. May 2026.
Download PDFExperiment data
All 650 run JSONs with full transcripts, state-machine source, scoring and classifier code, and the complete OpenRouter billing export.
Download ZIPCiting CrucibleBench
Help make the next experiment well-posed
Phase 2 is in active build. The instrument-validation direction is defined; the publishable environment, calibration, preregistration, and final allocation remain in progress. Fund the provisional envelope, help finish the instrument, or join the post-calibration pilot cohort.