typesafe-ai-benchmark

# typesafe-ai-benchmark

**LLM-native structured output vs. TypeSafe Jev: latency, cost, and judgment quality.**

[Hackers in the Loop](https://hackersintheloop.org/)

How does a fast general-purpose LLM compare with a purpose-built judgment model
on the same application tasks? This benchmark runs **Qwen 3.8 27B on Cerebras**
and **TypeSafe Jev** side by side across seven synthetic workloads, with a
separate local **Needle 3** evaluation on the same contracts. It records
validated outputs, mistakes, request latency, token usage and estimated cost.

[](docs/media/theater-demo.mp4)

[Watch or download the demo (MP4)](docs/media/theater-demo.mp4) ·
[Static screenshot](docs/media/theater.png) ·
[Raw results and methodology](docs/benchmarks/README.md)

The GIF and 98-second MP4 illustrate the comparison UI. Published measurements
come from the separately captured browser exports linked below.

## What we compare

| Approach | Model / provider | One application decision |
| --- | --- | --- |
| **LLM-native structured output** | Qwen 3.8 27B / Cerebras | One schema-constrained LLM response, validated and decoded locally |
| **Local tool-calling model** | Needle 3 / Cactus native runtime | One tool call, strictly validated before any simulated action |
| **Native judgment API** | Jev / TypeSafe | One request batching native Choice/Noul questions, mapped to the same application output |

“LLM-native” means using the LLM provider’s structured-output capability. The
benchmark compiles application fields into compact numeric slots, asks Qwen for
one constrained response, then reconstructs the typed result. It does not ask
Qwen to emulate Jev probabilities in this comparison. Jev's native probabilities
remain available in the exports. All paths validate outputs before applying
simul