jevcal

# jevcal

**Stop guessing confidence thresholds.** jevcal measures a typed decision model on *your* data, picks the
threshold that meets *your* accuracy target, tells you how much traffic still needs an LLM, and fails CI
when a model update quietly breaks it.

Built for [Jev](https://typesafe.ai) (TypeSafe's System One model), and for anything else that returns
answers with probabilities.

```
$ jevcal demo        # bundled example, built-in simulator, no API key

question               threshold  handled  accepted acc  all acc     ECE  status
is_urgent                  0.994    21.2%        100.0%    94.5%    3.5%  ok
department                 0.954    81.3%         96.3%    93.8%    3.5%  ok
frustration                0.875    98.0%         95.4%    95.5%    1.6%  ok
(handled / accepted accuracy are measured on the held-out split)

rows that escalate: 85.4%   cascade $1.930 per 1k rows vs LLM-only $2.25 (14.4% saved)
a row escalates when any question is unsure; the bottleneck is is_urgent (handles 21.2% at a 97% target)
```





That is simulator output, not a Jev benchmark. TypeSafe's customer agreement restricts publishing performance
numbers for Jev, so this README contains none, on purpose. That is also the reason this tool exists: nobody can
hand you trustworthy numbers for your use case, so you measure it yourself, privately, in one command.

Read the output as the kind of answer you get: `department` can run
81% on the fast model at 96% accuracy, while the 97% target on `is_urgent` is so strict that it alone sends
most rows to the LLM. Loosen that one target, or reword that one question, and the bill moves.

## Why

Jev answers "is this fraud?" with a probability, not a paragraph. That is the whole appeal: you can act on
the confident answers and send the rest