JevBench

# JevBench

Combination experiments (confidence cascades, committees, and real-sample best-of-n) are
reported in [RESULTS-COMBINATIONS.md](RESULTS-COMBINATIONS.md). None changed the ranked board.

A benchmark for **Jev-class decision models**: you hand the model a piece of state
and a bounded rubric, and it hands back a typed answer, ideally with a probability for
every option. No prose, no parsing, no "as an AI language model".

JevBench is [Benchmark Heaven](https://benchmarkheaven.com)'s own benchmark. It is not
affiliated with or endorsed by TypeSafe AI, whose Jev model is one of the systems
measured here.

## v1.2.3: the JevBench Score (current)

**[Results -> `RESULTS-v1.2.md`](RESULTS-v1.2.md)** · artifact [`results/v1.2/jevbench-v1.2-results.json`](results/v1.2/jevbench-v1.2-results.json) ·
interactive: [benchmarkheaven.com/jev-models](https://benchmarkheaven.com/jev-models) · how the hard tier was made: [`datasets/HARD-TIER.md`](datasets/HARD-TIER.md)

**JevBench Score** = Intelligence, Calibration, Speed, Cost — 25 % each, geometric mean: a weak axis pulls the score down hard.

| Axis | Score 0-100 |
|---|---|
| **Intelligence** | 100 x weighted accuracy: hard 30 %, easy 14 %, standard 28 %, judge 28 % |
| **Calibration** | hard tier: ECE + fidelity to exact gold distributions (label-only systems: none, counts as 0) |
| **Speed** | mean of score(p50), score(p95); score(s) = 100 - 20 log10(s / 0.1 s): 0.1 s = 100, each 10x slower -20 |
| **Cost** | 100 - 30 log10($ per 1,000 decisions / $0.001): $0.001 = 100, each 10x more expensive -30 |

> **The Cost column is US dollars per 1,000 DECISIONS, not per 1,000 tokens.** One decision is a whole question: its
> state, its rubric and its options — hundreds to thousands of input tokens.
> Jev 1.13.0 reads 950 input to