0028GitHub
jeval
Works out what a Jev confidence score is actually worth.
The tool calibrates the classifier against labelled data and sets the hand-off line from the cost of a mistake, so a low-confidence case goes to a person on the numbers rather than on a guess.
rlaope/jevalREADME ↗
What is your classifier's confidence really worth?</h3>
<p align="center">
jeval measures how well a classifier's confidence matches reality, and turns what a mistake costs<br>
into the threshold where the machine should stop deciding and a human should start.
</p>
<p align="center">
<a href="https://github.com/rlaope/jeval/actions/workflows/ci.yml"><img src="https://github.com/rlaope/jeval/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://github.com/rlaope/jeval/releases"><img src="https://img.shields.io/github/v/release/rlaope/jeval?color=1f5fbf" alt="Release"></a>
<img src="https://img.shields.io/badge/License-Apache%202.0-1d7a4b.svg" alt="Apache-2.0">
<img src="https://img.shields.io/badge/python-3.10%2B-1f5fbf" alt="Python 3.10 or newer">
</p>
<p align="center">
<a href="#install"><b>Install</b></a> ·
<a href="#see-it"><b>See it</b></a> ·
<a href="#get-your-data-in"><b>Get your data in</b></a> ·
<a href="#read-the-report"><b>Read the report</b></a> ·
<a href="#catch-a-model-change-in-ci"><b>Drift in CI</b></a> ·
<a href="#commands"><b>Commands</b></a>
</p>
<br>
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/report-verdict-dark.png" />
<img src="docs/report-verdict.png" alt="The verdict of the example report: 'Your threshold is too low', a ruler from 0 to 1 with the line in use at 0.60 and the recommended line at 0.75 inside its 95% interval, and the three figures the verdict rests on" width="820" />
</picture>
</p>
---
Your classifier answers with a label and a confidence. Two questions follow, and jeval answers both:
1. **When it says 0.9, how often is it actually right?**
2. **Given what a mistake costs,