jev-search-rerank-eval

# jev-search-rerank-eval

**Does a TypeSafe Jev rerank beat embedding search? Measured, with the judge-bias removed.**

A graded relevance evaluation over the [Agent Skills Hub](https://agentskillshub.top) catalog
(33,047 skills, MCP servers and coding-agent tools; snapshot 2026-09-18) with 164 real
Chinese / English / mixed queries, 9,831 labelled (query, skill) pairs, and a bake-off of:

- **`ash-0.4.0`** — the keyword ranker that ships in the [`ash` CLI](https://www.npmjs.com/package/@agentskillshub/cli), run byte-for-byte via [`ash_ranker.mjs`](src/jse/rankers/ash_ranker.mjs)
- **`bm25`** — BM25 over the same metadata (latin words + CJK bigrams)
- **`bge-m3`** — dense retrieval with BAAI/bge-m3 (local) · **`text-embedding-3-small`** — dense retrieval via OpenRouter
- **`jev-score(X@30)`** — a [TypeSafe Jev](https://typesafe.ai) `score` rerank of system X's top 30, one API call per query
- **`rrf(…)`** — reciprocal-rank-fusion hybrids

Metric: **NDCG@10** on graded labels (0–3), plus MRR and P@3 (relevant = label ≥ 2), with
paired bootstrap 95 % confidence intervals over queries.

## TL;DR

| claim | evidence |
|---|---|
| **Jev as a standalone reranker does not beat a good embedding ranker.** | `jev-score(bge-m3@30)` − `bge-m3` = +0.012 NDCG@10, CI [−0.013, +0.037]. Judged by an unrelated model only, it is **−0.028** [−0.052, −0.004]. |
| **Jev fused with the embedding ranking is the best system, and that survives removing Jev from the judging.** | `rrf(bge-m3, jev@30)` − `bge-m3` = **+0.090** [+0.077, +0.104]; under LLM-only labels **+0.064** [+0.052, +0.077]. |
| **The apparent Jev advantage under Jev's own labels is judge circularity, and it is measurable.** | The same comparison reads +0.053 under Jev-only labels, +0.012 under merged labels, −0.028 under LLM-onl