jev-benchmarks

# jev-benchmarks

[](https://github.com/AbdelStark/jev-benchmarks/actions/workflows/ci.yml)
[](https://www.python.org/downloads/)
[](LICENSE)

Probability-aware evaluation for typed decision models.

`jev-benchmarks` measures more than whether a model selects the right label. It evaluates whether the
reported probabilities are calibrated enough to support automation, how much work can be accepted at
a fixed error budget, what resources each decision uses, and how long it takes end to end.

The first study compares [TypeSafe Jev](https://typesafe.ai/) with
[`fastino/gliner2.5-multi-v1`](https://huggingface.co/fastino/gliner2.5-multi-v1) on their shared
capability: zero-shot, single-label text classification with per-label probabilities. It uses three
conditions from [BTZSC](https://huggingface.co/datasets/btzsc/btzsc), fixed model and dataset
revisions, identical examples and label descriptions, a uniform negative control, and paired
target-stratified bootstrap intervals.

## Pilot result

Three hundred held-out examples, 100 per condition:

| Dataset | Labels | Jev accuracy | GLiNER2.5 accuracy | Jev − GLiNER 95% CI | Jev coverage at ≤5% error | GLiNER coverage |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| AG News | 4 | **0.910** | 0.700 | **[+0.130, +0.290]** | **0.830** | 0.240 |
| Banking77/BTZSC | 72 | **0.870** | 0.610 | **[+0.220, +0.300]** | **0.860** | 0.270 |
| DAIR Emotion | 6 | 0.480 | 0.440 | [−0.070, +0.150] | 0.000 | **0.020** |

The result is deliberately mixed. Jev has a clear accuracy and Brier-score advantage on AG News and
Banking77/BTZSC. On DAIR Emotion, the accuracy difference is unresolved and Jev is substantially
worse calibrated: Brier `0.846` versus `0.668`, NLL `5.588` versus `1.381`, and zero probability on
the true label for 16% of e