jev-harness

# jev-harness

A minimal coding agent that filters every tool result through a small relevance
model ([Jev](https://vercel.com/ai-gateway/models/jev)) before the main model
sees it, plus an A/B harness that runs the same tasks with filtering on and off.

The question: can a clean, filtered context plus a cheap model match a
conventional harness at lower cost?



On 30 tasks (Terminal-Bench 2.1 + DeepSWE), one run per arm, with
`deepseek-flash` as the main model:

| | pass@1 | cost | cost per passing task |
|---|---|---|---|
| Jev on | **25 / 30** | $2.82 | **$0.113** |
| Jev off | 22 / 30 | $2.80 | $0.127 |

The gap is entirely on DeepSWE (6/9 vs 3/9); on Terminal-Bench the two arms tie
at 19/21.

Read at test granularity — DeepSWE's verifier reports how many of a task's new
tests pass, not just whether all of them do — **the filtered arm is never worse
on any of the 9 tasks: 6 wins, 0 losses, 3 ties** (sign test, p≈0.031). The
clearest case is `meriyah`: 49/49 tests with the filter, 0/49 without, where
the unfiltered agent was shown 1.46M characters of tool output and ran out of
time.



Plotted against the 12 harness configurations FrontierHarness Eval ran on the
same 30 tasks — a different model (Kimi K3) and 360 runs, so this places our
arms, it does not rank them:



See [RESULTS.md](RESULTS.md) for per-task numbers and caveats. One run per arm:
this is a prototype measurement, not a generalization claim. Figures are
regenerated with `uv run --with matplotlib scripts/plot_results.py`.

## How it works

- Main model over the OpenAI Responses API (`deepseek-flash` supports it natively).
- Three tools only: `bash`, `read`, `apply_patch`.
- Every tool result except `read` goes through Jev first: the output is split
  into ~2k-char chunks on line boundaries, and each ch