jev-harness

# jev-harness

A minimal coding agent that filters every tool result through a small relevance
model ([Jev](https://vercel.com/ai-gateway/models/jev)) before the main model
sees it, plus an A/B harness that runs the same tasks with filtering on and off.

The question: can a clean, filtered context plus a cheap model match a
conventional harness at lower cost?



## Latest outcome: TB4 follow-up, September 22, 2026

**The current integration did not demonstrate enough value to justify default
adoption or further large-scale runs. The experiment is closed.**

On 49 scored TB4 pairs, Jev on passed 2 tasks versus 5 for off, used 11.78%
more cumulative input tokens, and incurred 8.17% more estimated model cost.
Mean per-task test pass fractions were 44.27% versus 48.04% on 42 pairs with
test counts in both arms. Infrastructure costs are separate.

Trajectory inspection found unbounded-output failures, environment limitations,
and divergent execution paths. The result does not establish that Jev is
generally ineffective or that filtering caused the success-rate difference.
It does establish that this implementation has not demonstrated an end-to-end
benefit sufficient to justify its added complexity.

Read the [TB4 report and trajectory analysis](TB4-CONCLUSION-2026-09-22.md)
and [per-task metric snapshot](tb4-results-2026-09-22.json).
The experiment VM has been deleted and automatic reruns disabled.

## Earlier 30-task prototype

The following results belong to the earlier experiment. They are retained as
historical evidence and are not pooled with, or a substitute for, the TB4 follow-up.

On 30 tasks (Terminal-Bench 2.1 + DeepSWE), one run per arm, with
`deepseek-flash` as the main model:

| | pass@1 | cost | cost per passing task |
|---|---|---|---|
| Jev on | **25 / 30**