jev-curate

# `jev-curate`

**High-Throughput Synthetic & Pretraining Dataset Sifter Powered by TypeSafe AI (Jev)**

[](https://crates.io/crates/jev-curate)
[](https://pypi.org/project/jev-curate/)
[](LICENSE)
[](https://typesafe.ai)

By **[Akash Priyadarshi](https://github.com/AkashPriyadarshii)**

[Why jev-curate](#why-jev-curate) • [Quickstart](#quickstart) • [CLI Reference](#cli-reference) • [Python API](#python-api) • [Architecture](#architecture) • [Non-Goals](#non-goals) • [Ecosystem](#ecosystem)

</div>

---

## Why `jev-curate`?

Cleaning 10M to 1B rows of synthetic reasoning data, instruction tuning pairs, or web-scraped corpora is an economic and technical nightmare:
* **Generative LLMs are too slow and expensive:** Running Claude 3.5 Sonnet or GPT-4o to judge synthetic rows costs **$15,000–$50,000** per billion tokens and crawls at a painful 30–50 rows/sec.
* **Regex heuristics are blind to reasoning flaws:** Keyword and regex filters can check syntax, but fail to detect circular reasoning, hallucinated derivation steps, or robotic sycophancy.
* **Context rot from uncompressed inputs:** Naively feeding raw data into LLMs causes decision accuracy to crater while burning money on boilerplate text.

`jev-curate` solves this by piping Apache Arrow and Parquet streams through **TypeSafe AI's Jev model** (`jev-1.13.0`):
* **1,500+ rows/sec throughput:** Evaluates rows in multi-threaded batches using Jev's speculative parallel fan-out.
* **~$4.20 per 100M tokens:** Jev charges $0.042/Mtok for input with zero output token fees—over 100x cheaper than GPT-4o-mini and 700x cheaper than Claude 3.5 Sonnet.
* **Mathematical calibration:** Receives calibrated probabilities (`Noul`), ordinal rubrics (`Score` 1–5), and categorical choices (`Choice`), elimin