jev-evaluation

# Evaluating jev

An adversarial evaluation of **jev**, the decision model sold by TypeSafe. It
follows [a plan](jev-evaluation-plan.md) written before any request was sent.
That plan fixed nine experiments, the sample size of each, and twenty-eight
predictions, each stated with the result that would prove it wrong. Nothing
reported here was chosen after seeing an outcome.

One run produced every number below: **123,805 requests, 138 minutes, $12.69,
five failures**, all answered by `jev-1.13.0`.

- **[A prompting guide](PROMPTING.md)** — thirteen rules for using this model, each
  with the requests to write, the mistakes to avoid, and the measurement behind it.
  Start here if you are deciding how to call the API.
- **[The report](https://willkelly.github.io/jev-evaluation/runs/full-20260919/report.html)**
  — findings, method and figures. Also available as
  [markdown](runs/full-20260919/report.md) or [a local file](runs/full-20260919/report.html).
- **[The plan](jev-evaluation-plan.md)** — what was predicted, and why.

## What it found

Twelve of twenty-five testable predictions held. Thirteen were wrong, which is
the useful half.

**Calibration holds where the model was built to work and fails completely
outside it.** On support-ticket routing the expected calibration error is 0.075.
On random 3-SAT the model answers *satisfiable* for every formula at every clause
ratio: its average probability varies by 0.026 while the true satisfiable
fraction varies by 1.000. A short program reading clause density beats it at 41
of 75 conditions. Given `x AND NOT x`, unsatisfiable by inspection, it returns
P(satisfiable) = 0.38.

**Asking many questions at once is genuinely free.** Accuracy at question 200
matches question 1, there is no decline up to 255 options, and latency is