0056GitHub

jev-align

Turns human labels into calibrated Jev classifiers with GEPA.

Sutro’s library builds calibrated classifiers from human feedback. You label examples, GEPA optimizes the question, and what comes out is a Jev function with a known error rate. It is on PyPI.

Open source ↗ github.comcost-time-

sutro-sh/jev-alignREADME ↗

# jev-align

`jev-align` is an experimental CLI from [Sutro](https://sutro.sh/) for building
AI Functions with [TypeSafe's Jev](https://docs.typesafe.ai/introduction).

It finds uncertain examples, asks you to label them, and uses
[GEPA](https://gepa-ai.github.io/gepa/) to improve the function. Use it in your
application, keep learning from production examples, and share portable
functions through [ai-functions.dev](https://ai-functions.dev/).

## Demo

https://github.com/user-attachments/assets/81650587-e3f1-4655-8213-ed5f6e120e9a

## Quick start

Requires Python 3.11 or newer.

```shell
uv tool install jev-align
export TYPESAFE_API_KEY="..." # Or use Vercel or Cloudflare below
export OPENAI_API_KEY="..." # or ANTHROPIC_API_KEY / GEMINI_API_KEY
jeva
```

Start the CLI with either `jeva` or `jev-align`.

Use `pip install jev-align` if you do not use
[uv](https://docs.astral.sh/uv/). The guided setup discovers local CSV,
Parquet, and JSONL files and includes three ready-to-run examples.

## How it works

Each round:

1. Evaluates the configured dataset and measures uncertainty.
2. Selects ambiguous rows plus a random audit sample for you to label.
3. Uses your accumulated labels and optional rationales to run GEPA.
4. Shows the score, certainty change, and proposed definition diff.
5. Lets you accept, reject, rewind, or resume later.

The guided workflow asks you to review every label. Labels may be human-created,
synthetic, imported, or agent-assisted; review them to the standard your use
case requires. A higher training score never accepts a proposal automatically.

## Task types

| Type | Output |
| --- | --- |
| Binary | `True` or `False` |
| Multiclass | Exactly one fixed label |
| Multilabel | Zero or more fixed labels |
| Score | One level from an ordered rubric |

Also filed under Research & data

0620

Jev Score: rubrics, scores and confidence

A worked guide to Jev's Score primitive: writing a request, defining rubric levels, reading recorded probabilities and the weighted-score math.

Jev Trader · Research & data

0614

Bot journey classification in WebDecoy

Sends a detected bot's last 48 request paths to Jev, which picks what it's after (prices, articles…) and how it crawls (pagination, IDs…), or unknown.

WebDecoy · Research & data

0610

Fake / Real: link fact-checker with Jev as judge

Paste an article or post link. Fake / Real extracts its claims, finds outside evidence, and has Jev judge whether the evidence supports or contradicts each one.

@DansiDanutz · Research & data

0609

AutoRubric: rubric-based evaluation

Combines rubric science and LLM-as-a-judge research to grade outputs with AI judges: LLMs, decision models like Jev, or both.

@deliprao · Research & data