0356GitHub
jev-rag-benchmark
Reproducible experiments on whether reranking with Jev improves a small RAG system, on a locked Turkish dataset, with quality, latency and cost reported together.
erendikmenn/jev-rag-benchmarkREADME ↗
# jev-rag-benchmark A small, auditable Turkish RAG experiment: does reranking improve passage selection, and how do four answer models perform on identical retrieved evidence? **Current evaluation: 29 October 2026 — 1,044 XQuAD-TR test questions, 240 passages.** This is a short-answer benchmark on a small corpus, not a general-purpose RAG leaderboard. ## What is held constant? - The candidate pool is the historical hybrid BM25 + BGE-M3 baseline's top 20, frozen once. - Jev, Cohere, Clef Flash and Luna Decisions receive identical candidate **texts and order** for every question. - All rerankers return five passages. The baseline retains the original hybrid order. - Clef Flash is reported separately for 20-question batch and one-passage-per-request scoring; Jev 1.13 is the original batch run, Jev 1.14 and Luna Decisions have both modes, and Cohere uses its native rerank endpoint. - Gemini, DeepSeek, Haiku and Luna receive the **same Jev 1.13 top-five context and exact prompt**, capped at 6,000 context characters. They must return a short answer span with a source citation. - All use nominal medium reasoning and a 4,096 output-token limit. Gemini/DeepSeek use temperature 0 and seed 20260919. Temperature is unsupported for Luna/Haiku; only Luna supports the seed. Equal effort labels do not imply equal reasoning compute. - Generation errors and truncated outputs receive zero EM/F1; they are not dropped. The published run contains 1,044 unique, matched questions and **zero candidate mismatches**. The corpus SHA-256 matches the historical corpus. The frozen pool, per-question outputs, prompt hashes and model identities are committed under [`results/fair-2026-10-29/`](results/fair-2026-10-29/). ## Passage selection Generation is excluded from this comparison. All