0352GitHub

jev-benchmarks

Reproducible evaluation for calibration, selective risk, and latency.

AbdelStark/jev-benchmarksREADME ↗

# jev-benchmarks

[](https://github.com/AbdelStark/jev-benchmarks/actions/workflows/ci.yml)
[](https://www.python.org/downloads/)
[](LICENSE)

Probability-aware evaluation for typed decision models.

`jev-benchmarks` measures more than whether a model selects the right label. It evaluates whether the
reported probabilities are calibrated enough to support automation, how much work can be accepted at
a fixed error budget, what resources each decision uses, and how long it takes end to end.

The first study compares [TypeSafe Jev](https://typesafe.ai/) with
[`fastino/gliner2.5-multi-v1`](https://huggingface.co/fastino/gliner2.5-multi-v1) on their shared
capability: zero-shot, single-label text classification with per-label probabilities. It uses three
conditions from [BTZSC](https://huggingface.co/datasets/btzsc/btzsc), fixed model and dataset
revisions, identical examples and label descriptions, a uniform negative control, and paired
target-stratified bootstrap intervals.

## Pilot result

Three hundred held-out examples, 100 per condition:

| Dataset | Labels | Jev accuracy | GLiNER2.5 accuracy | Jev − GLiNER 95% CI | Jev coverage at ≤5% error | GLiNER coverage |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| AG News | 4 | **0.910** | 0.700 | **[+0.130, +0.290]** | **0.830** | 0.240 |
| Banking77/BTZSC | 72 | **0.870** | 0.610 | **[+0.220, +0.300]** | **0.860** | 0.270 |
| DAIR Emotion | 6 | 0.480 | 0.440 | [−0.070, +0.150] | 0.000 | **0.020** |

The result is deliberately mixed. Jev has a clear accuracy and Brier-score advantage on AG News and
Banking77/BTZSC. On DAIR Emotion, the accuracy difference is unresolved and Jev is substantially
worse calibrated: Brier `0.846` versus `0.668`, NLL `5.588` versus `1.381`, and zero probability on
the true label for 16% of e

Also filed under Research & data

0620

Jev Score: rubrics, scores and confidence

A worked guide to Jev's Score primitive: writing a request, defining rubric levels, reading recorded probabilities and the weighted-score math.

Jev Trader · Research & data

0614

Bot journey classification in WebDecoy

Sends a detected bot's last 48 request paths to Jev, which picks what it's after (prices, articles…) and how it crawls (pagination, IDs…), or unknown.

WebDecoy · Research & data

0610

Fake / Real: link fact-checker with Jev as judge

Paste an article or post link. Fake / Real extracts its claims, finds outside evidence, and has Jev judge whether the evidence supports or contradicts each one.

@DansiDanutz · Research & data

0609

AutoRubric: rubric-based evaluation

Combines rubric science and LLM-as-a-judge research to grade outputs with AI judges: LLMs, decision models like Jev, or both.

@deliprao · Research & data