jev-eval

# jev-eval

Benchmark [TypeSafe Jev](https://typesafe.ai) against any OpenRouter model or a local checkpoint on labelled classification data: accuracy, calibration, confidence distribution, latency, cost. Ships four tasks; the results below are those tasks against GPT-5.6 Terra and [Laya](https://huggingface.co/convaiinnovations/laya).

## Results

300 items per task, run 2026-09-17 and 2026-09-21. `jev-1.13.0`, `openai/gpt-5.6-terra` via OpenRouter, `convaiinnovations/laya` (421M, Apache 2.0) local on an M-series GPU.

| | intent (77 classes) | sentiment (5 levels) | polarity as `noul` | polarity as `choice` |
|---|---|---|---|---|
| **Accuracy** Jev / Laya / Terra | 0.780 / 0.370 / **0.847** | 0.570 / 0.310 / **0.593** | **0.970** / 0.507 / **0.970** | **0.967** / 0.947 / — |
| **ECE** (lower better) Jev / Laya / Terra | 0.110 / 0.520 / **0.081** | **0.200** / 0.317 / 0.303 | 0.042 / 0.496 / **0.020** | 0.022 / **0.020** / — |
| **AUROC** of confidence Jev / Laya / Terra | **0.831** / 0.696 / 0.807 | 0.611 / **0.711** / 0.636 | 0.935 / 0.897 / **0.964** | **0.909** / 0.885 / — |
| **p50 latency** Jev / Laya / Terra | 0.20 s / **0.10 s** / 1.04 s | 0.19 s / **0.04 s** / 1.06 s | 0.20 s / **0.08 s** / 1.04 s | 0.18 s / **0.08 s** / — |
| **Cost per 1k** Jev / Laya / Terra | $0.040 / **$0** / $2.02 | $0.014 / **$0** / $0.64 | $0.021 / **$0** / $0.85 | $0.021 / **$0** / — |

- **Laya is 2–5x faster than Jev here and free**, but the two numbers are not the same quantity: Laya is local compute, Jev and Terra include the network round trip. Laya's latency scales with option count, not just question count: 0.04 s at 5 levels, 0.08 s at 2 options, 0.10 s at 77.
- **Laya's `noul` collapses on this data.** It returns 0.0 on 298 of 300 reviews and scores 0.507, chance on a balanc