jev-rerank-bench

# jev-rerank-bench

**The write-up:** [Jev vs Cohere Rerank 4 Pro vs zerank-2: reranker benchmark](https://anessbelbati.com/blog/i-gave-jev-a-rerankers-job) walks through the results, the negation test and the Qwen follow-up.

I gave [TypeSafe's Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) thirty search results and asked
it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. This repository contains the experiments, saved responses and scoring code.

The ranking average put **Jev's rubric at 0.692 and Cohere Pro at 0.691**, without establishing a winner. Giving
every query equal weight instead puts Cohere ahead. Jev did better on the negation test than the rerankers; the open-weight Open-Jev 9B, run afterwards, reads negation better than Jev asked the same way (77% vs 71% of pairs). An open-source Qwen recipe
improved substantially when I gave it one passage at a time, but still showed no clear gain over keyword ranking.
Eight more open models, run September 25, all beat keyword ranking; the best of them, Qwen3-Reranker-4B, is within
noise of Jev asked one passage at a time (0.660 vs 0.670), trails Jev's rubric, and beats every Jev setup on French.

The main comparison covers eight English datasets. Five more BRIGHT subsets, NevIR negation pairs and MIRACL
French are reported separately. Models start with the same thirty BM25 candidates, cut to 2,000 characters each;
the duel variant only compares the first ten, and NevIR supplies its own two-passage pairs.

Measurements began September 16, 2026; the Qwen controls were added September 17. Jev calls used `jev-latest`,
reporting version 1.13.0. The [evidence viewer](https://anessbelbati.com/lab/jev-reranking/) includes the original
benchmark and the Qwen