jev-rerank-bench

# jev-rerank-bench

I gave [TypeSafe's Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) thirty search results and asked
it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. This repository contains the experiments, saved responses and scoring code.

The ranking average put **Jev's rubric at 0.692 and Cohere Pro at 0.691**, without establishing a winner. Giving
every query equal weight instead puts Cohere ahead. Jev did better on the negation test. An open-source Qwen recipe
improved substantially when I gave it one passage at a time, but still showed no clear gain over keyword ranking.

The main comparison covers eight English datasets. Five more BRIGHT subsets, NevIR negation pairs and MIRACL
French are reported separately. Models start with the same thirty BM25 candidates, cut to 2,000 characters each;
the duel variant only compares the first ten, and NevIR supplies its own two-passage pairs.

Measurements began September 16, 2026; the Qwen controls were added September 17. Jev calls used `jev-latest`,
reporting version 1.13.0. The [evidence viewer](https://anessbelbati.com/lab/jev-reranking/) includes the original
benchmark and the Qwen follow-up, with individual questions, scores and saved outputs.

## Headline (8 English datasets, 1,617 scored questions, each dataset counts once)

| Model | nDCG@10 | Top pick right | Time per query | $ per 1,000 queries | Spots "nothing here" (AUROC) |
|---|---|---|---|---|---|
| Jev 4-level rubric, 30 in one call | 0.692 | 74% | 422 ms | 0.45 | 0.75 |
| Cohere Rerank 4 Pro | 0.691 | 73% | 844 ms | 2.51 | 0.78 |
| Jev 30 yes/no in one call | 0.685 | 72% | 396 ms | 0.41 | 0.75 |
| Jev one Choice + none | 0.684 | 76% | 338 ms | 0.33 | 0.72 |
| Cohere Rerank 4 Fast | 0.684 | 72% | 726 ms