0050GitHub
jev-evaluation
Nine experiments and 28 predictions, all fixed before any data.
Will Kelly wrote down 28 predictions about Jev before collecting any data, then ran nine adversarial experiments against them. The run took 123,805 requests and $12.69, and the repository publishes what held and what did not.
willkelly/jev-evaluationREADME ↗
# Evaluating jev An adversarial evaluation of **jev**, the decision model sold by TypeSafe. It follows [a plan](jev-evaluation-plan.md) written before any request was sent. That plan fixed nine experiments, the sample size of each, and twenty-eight predictions, each stated with the result that would prove it wrong. Nothing reported here was chosen after seeing an outcome. One run produced every number below: **123,805 requests, 138 minutes, $12.69, five failures**, all answered by `jev-1.13.0`. - **[A prompting guide](PROMPTING.md)** — thirteen rules for using this model, each with the requests to write, the mistakes to avoid, and the measurement behind it. Start here if you are deciding how to call the API. - **[The report](https://willkelly.github.io/jev-evaluation/runs/full-20260919/report.html)** — findings, method and figures. Also available as [markdown](runs/full-20260919/report.md) or [a local file](runs/full-20260919/report.html). - **[The plan](jev-evaluation-plan.md)** — what was predicted, and why. ## What it found Twelve of twenty-five testable predictions held. Thirteen were wrong, which is the useful half. **Calibration holds where the model was built to work and fails completely outside it.** On support-ticket routing the expected calibration error is 0.075. On random 3-SAT the model answers *satisfiable* for every formula at every clause ratio: its average probability varies by 0.026 while the true satisfiable fraction varies by 1.000. A short program reading clause density beats it at 41 of 75 conditions. Given `x AND NOT x`, unsatisfiable by inspection, it returns P(satisfiable) = 0.38. **Asking many questions at once is genuinely free.** Accuracy at question 200 matches question 1, there is no decline up to 255 options, and latency is