0353GitHub
jev-eval
Independent Jev versus GPT-5.6 Terra comparison on three labeled classification tasks, reporting accuracy, calibration, latency, and cost.
4esv/jev-evalREADME ↗
# jev-eval Benchmark [TypeSafe Jev](https://typesafe.ai) against any OpenRouter model or a local checkpoint on labelled classification data: accuracy, calibration, confidence distribution, latency, cost. Ships five tasks; the results below are Jev, GPT-5.6 Terra, and three open Jev-shaped models. ## Results 300 items per task. `jev-1.13.0`; `openai/gpt-5.6-terra` via OpenRouter; [Laya](https://huggingface.co/convaiinnovations/laya) 421M, [open-jev](https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large) 435M and [Kev-0.8B](https://huggingface.co/jaredpalmer/kev-0.8b) run locally on an M-series GPU. **Accuracy** | task | options | Jev | Terra | open-jev | Kev-0.8B | Laya | |---|---|---|---|---|---|---| | clinc (out-of-sample) | 151 | **0.897** | — | 0.610 | 0.643 | 0.497 | | intent (Banking77) | 77 | 0.780 | 0.847 | **0.873** | 0.770 | 0.370 | | sentiment5 (SST-5) | 5 | 0.570 | **0.593** | 0.560 | 0.510 | 0.310 | | polarity (IMDB) as `noul` | 2 | **0.970** | **0.970** | 0.957 | 0.953 | 0.507 | | polarity as `choice` | 2 | **0.967** | — | 0.963 | 0.950 | 0.947 | **Calibration (ECE, lower better) and p50 latency** | task | Jev | Terra | open-jev | Kev-0.8B | Laya | |---|---|---|---|---|---| | clinc | **0.039** / 0.17 s | — | 0.072 / 0.43 s | 0.260 / 0.87 s | 0.471 / **0.16 s** | | intent | 0.110 / 0.20 s | 0.081 / 1.04 s | **0.047** / 0.27 s | 0.146 / 0.67 s | 0.520 / **0.10 s** | | sentiment5 | 0.200 / 0.19 s | 0.303 / 1.06 s | **0.052** / 0.07 s | 0.110 / 0.13 s | 0.317 / **0.04 s** | | polarity | 0.042 / 0.20 s | **0.020** / 1.04 s | 0.036 / 0.17 s | 0.030 / 0.37 s | 0.496 / **0.08 s** | - **Training on the benchmark decides the winner.** open-jev lists `mteb/banking77` and `SetFit/sst5` as training data and beats Jev on Banking77, 0.873 against 0.780.