jeval

Find out what your classifier's confidence is really worth,<br>
  and where to hand off to a human, based on what a mistake costs.</strong>
</p>

<div align="center">

[](https://github.com/rlaope/jeval/actions/workflows/ci.yml)
[](https://github.com/rlaope/jeval/releases)
[](https://opensource.org/licenses/Apache-2.0)
[](https://www.python.org)
[](#install)

</div>

<table>
  <tr>
    <td width="50%" valign="top">
      <img src="docs/report-verdict.png" alt="The verdict card: 'Your threshold is too low', with the current threshold, the measured accuracy and the recommended threshold">
      <br><sub>The line is in the wrong place, and here is why.</sub>
    </td>
    <td width="50%" valign="top">
      <img src="docs/report-reliability.png" alt="Reliability chart: confidence along the bottom, how often the model was right up the side, with grey bands where the data is thin">
      <br><sub>What the model claimed, against how often it was right.</sub>
    </td>
  </tr>
  <tr>
    <td width="50%" valign="top">
      <img src="docs/report-impact.png" alt="Impact table comparing the threshold in use with the recommended one: auto rate, accuracy and cost per case">
      <br><sub>What moving the line changes, and what it costs.</sub>
    </td>
    <td width="50%" valign="top">
      <img src="docs/report-segments.png" alt="Cost per case by segment, each segment's own threshold drawn against the global one">
      <br><sub>Who does worse than average, and needs their own line.</sub>
    </td>
  </tr>
</table>

Your classifier answers with a label and a confidence. Two questions follow, and jeval answers both:

1. When it says 0.9, how often is it actually right?
2. Given what a mistake costs, where should the line sit between "the machine decides" and "a human
   decides"?