jev-acento

# jev-acento

**Does Jev understand your accent?**

An independent, reproducible audit of [Jev](https://typesafe.ai) — TypeSafe AI's "System One"
evaluation model — on **Spanish**, plus a CLI that lets anyone run the same comparison on their
own labelled data.

**Run `20260921-es-v1`** — 19,200 calls, 3,200 paired items, model `jev-1.13.0` (version-pinned),
USD 0.58, 0 errors. Every number below comes from [`results.json`](results.json) via
`make reproduce`; none is typed by hand.

## Findings

**1. Spanish costs accuracy on every dataset.** Holding the instructions in English and swapping
only the `state` from English to Spanish (B − A), Jev is measurably worse on all four:

| Dataset | A (EN state) | B (ES state) | Δ accuracy | Verdict |
|---|---|---|---|---|
| XNLI | 0.850 | 0.786 | −6.4 pp `[−8.6, −4.3]` | **measurably worse** |
| PAWS-X | 0.834 | 0.772 | −6.2 pp `[−8.5, −3.6]` | **measurably worse** |
| MASSIVE | 0.845 | 0.808 | −3.7 pp `[−5.8, −1.5]` | **measurably worse** |
| Belebele | 0.982 | 0.952 | −3.0 pp `[−4.5, −1.7]` | **measurably worse** |

**2. It also costs calibration, on the two hardest tasks.** ECE roughly doubles on XNLI
(0.057 → 0.101) and PAWS-X (0.033 → 0.078) — *less calibrated* under the pre-registered rule.
On MASSIVE and Belebele the change is not detectable. Calibration matters more than accuracy
here: the operational consequence is that automating at `p_max ≥ 0.9` covers **72.2% of XNLI in
English but only 63.4% in Spanish**, and the items you do automate are *less* accurate
(0.938 → 0.904), not more.

**3. Writing the instructions in Spanish does not help.** This is the question nobody had
measured, and the answer is a clean null on three of four datasets (C − B):

| Dataset | Δ accuracy | Verdict |
|---|---|---|
| XNLI | −0.2 pp `[−1.1,