jev-rag-benchmark

# jev-rag-benchmark

Reproducible, vendor-neutral experiments for measuring whether TypeSafe Jev improves a
small RAG system. “Jev wins” is not an assumption: quality, latency, and cost can improve,
stay flat, or get worse.

## Benchmark status — XQuAD-TR (2026-09-20)

The locked full benchmark contains **1,044 unique Turkish XQuAD questions** and a corpus of
240 passages. Published model and embedding calls use OpenRouter; no local LLM or local
embedding model is used. The first stage fuses BM25 with `baai/bge-m3`, exposes the same 20
candidates to every reranker, and gives the answer model the best five passages. The Jev
request was `typesafe/jev-1.13`; OpenRouter resolved it to
`typesafe/jev-1.13-20260917`.

### 1. Candidate retrieval ceiling

This table answers the first prerequisite question: *was the gold passage available to the
reranker at all?* A reranker cannot recover a passage outside its candidate pool.

| Candidate source | Candidate depth | Gold passage found | Candidate recall |
|---|---:|---:|---:|
| BM25 | top-5 | 979 / 1,044 | 93.774% |
| BM25 | top-20 | 1,009 / 1,044 | 96.648% |
| BM25 | top-50 | 1,020 / 1,044 | 97.701% |
| BM25 | top-100 | 1,028 / 1,044 | 98.467% |
| BM25 | all 240 | 1,044 / 1,044 | 100.000% |
| Hybrid BM25 + BGE-M3 | top-5 | 1,019 / 1,044 | 97.605% |
| Hybrid BM25 + BGE-M3 | top-20 | 1,039 / 1,044 | **99.521%** |
| Hybrid BM25 + BGE-M3 | top-50 | 1,044 / 1,044 | **100.000%** |

Full-depth hybrid fusion raises the top-20 ceiling by **2.873 percentage points** over BM25
alone, from 1,009 to 1,039 answer-bearing candidate sets.

### 2. Full reranker benchmark

Generation is disabled here, so the table isolates passage selection. All three rows use
the same 1,044 questions and the same frozen hybrid top-20 candidates.

| Reranking meth