A real 30-case run, scored and attributable
Proof the AI's answers are actually good — measured, not assumed.
Real run: Anthropic agent (claude-opus-4-8) + LLM-as-judge (claude-opus-4-8, prompt v2) over the 30-case frozen FX backtest dataset. No live-API values were hand-edited.
Run summary
Aggregate metrics computed by the harness over every case — nothing recomputed here.
Per-case results
Every case is one row: the verdict, the 0–1 score, latency and cost. The 3 fails are real — open one to read why the judge failed it.
| case | verdict | score | latency | cost | |
|---|---|---|---|---|---|
| bt-0001 | pass | 0.70 | 5,859 ms | $0.0128 | › |
| bt-0002 | pass | 0.70 | 5,635 ms | $0.0122 | › |
| bt-0003 | fail | 0.40 | 6,186 ms | $0.0126 | › |
| bt-0004 | pass | 0.68 | 7,798 ms | $0.0140 | › |
| bt-0005 | pass | 0.85 | 6,124 ms | $0.0130 | › |
| bt-0006 | pass | 0.65 | 6,258 ms | $0.0125 | › |
| bt-0007 | fail | 0.40 | 5,662 ms | $0.0118 | › |
| bt-0008 | fail | 0.40 | 5,578 ms | $0.0124 | › |
| bt-0009 | pass | 0.85 | 6,135 ms | $0.0129 | › |
| bt-0010 | pass | 0.68 | 7,489 ms | $0.0133 | › |
| bt-0011 | pass | 0.85 | 10,448 ms | $0.0125 | › |
| bt-0012 | pass | 0.70 | 6,356 ms | $0.0117 | › |
| bt-0013 | pass | 0.85 | 6,348 ms | $0.0129 | › |
| bt-0014 | pass | 0.85 | 5,967 ms | $0.0115 | › |
| bt-0015 | pass | 0.90 | 5,065 ms | $0.0114 | › |
| bt-0016 | pass | 0.82 | 6,044 ms | $0.0120 | › |
| bt-0017 | pass | 0.85 | 5,087 ms | $0.0114 | › |
| bt-0018 | pass | 0.90 | 13,463 ms | $0.0118 | › |
| bt-0019 | pass | 0.85 | 5,957 ms | $0.0114 | › |
| bt-0020 | pass | 0.90 | 5,774 ms | $0.0125 | › |
| bt-0021 | pass | 0.85 | 5,605 ms | $0.0120 | › |
| bt-0022 | pass | 0.80 | 5,726 ms | $0.0123 | › |
| bt-0023 | pass | 0.82 | 6,157 ms | $0.0128 | › |
| bt-0024 | pass | 0.72 | 6,236 ms | $0.0127 | › |
| bt-0025 | pass | 0.85 | 5,601 ms | $0.0126 | › |
| bt-0026 | pass | 0.72 | 5,558 ms | $0.0121 | › |
| bt-0027 | pass | 0.78 | 4,962 ms | $0.0113 | › |
| bt-0028 | pass | 0.85 | 8,551 ms | $0.0126 | › |
| bt-0029 | pass | 0.78 | 6,415 ms | $0.0134 | › |
| bt-0030 | pass | 0.80 | 7,577 ms | $0.0136 | › |
Prompt drift — v1 → v2
The judge prompt is versioned. Drift re-scores the same agent outputs under an older prompt and attributes the delta — the same two runs, read both directions.
Both runs judged the identical 30 agent outputs — only the judge prompt differs, so the entire score shift is attributable to the prompt, not the model or the agent.
v1 shipped first; v2 is current. Across the same cases, v2 scores higher — no case dropped past the 0.15 threshold.
Same two runs, reversed. drift --fail-on-regression exits non-zero here — this is what stops a silent prompt downgrade from shipping.
| case | v2 (baseline) | v1 (candidate) | Δ | verdict |
|---|---|---|---|---|
| bt-0002 | 0.70 | 0.40 | -0.30 | pass → fail⚑ |
| bt-0004 | 0.68 | 0.40 | -0.28 | pass → fail⚑ |
| bt-0010 | 0.68 | 0.40 | -0.28 | pass → fail⚑ |
SKIP. While the reward:risk ratio of 2.0 is acceptable, the setup rests entirely on a modest price score of 2 with zero support from macro or news components. A long trade lacking any macro or fundamental confirmation is fragile, as an unfavorable data release or macro backdrop could easily invalidate the technical premise. Confidence: 72%
The SKIP call is defensible given the weak composite score (only price scoring, zero macro/news), and the reasoning correctly weighs component scores against the acceptable 2.0 R:R; confidence at 72% is reasonably calibrated. The backtest happened to win, but the rubric explicitly instructs not to penalize sound reasoning contradicted by outcome.
The agent chose SKIP with reasoning that ignored the recorded READY status and reasonable 2.0 R:R, and the backtest realized a win at +2.0R. While the caution about zero macro/news confirmation is a defensible concern, the confident SKIP call contradicts both the recorded status and the outcome, and the confidence was not calibrated to the winning result.
v2 adds the guardrail “don't penalize sound reasoning the outcome merely contradicts” — so it rewards the agent's defensible SKIP; v1, biased toward the realized outcome, fails it.