arbiter

# arbiter

Serve typed-decision models — Laya or your own — on your GPU or your Mac, with a Jev-compatible
API.

Read the story behind it: [My cat woke me at five on a Sunday, so I built a local Jev](https://blog.0xbakeer.com/my-cat-woke-me-at-five-on-a-sunday-so-i-built-a-local-jev-1350583cddc6?sharedUserId=0xbakeer)



What it does with a state and a set of questions, and where it sits next to an LLM.

## Why the name, and what this is

A typed-decision model answers questions about a piece of text in **one forward pass** — no
tokens generated, no sampling, no loop. You hand it a state and a set of questions, and it
returns an answer and a probability distribution for each of them at once. There is no prose to
read back and nothing to argue with: it arbitrates, and your code decides what to do with the
numbers.

This repository is the serving layer around that and deliberately nothing more — a small HTTP
server that speaks TypeSafe's **Jev** API, so a client written against Jev works against this by
changing the base URL; cross-request micro-batching; routing between checkpoints; a playground;
and the measurements that picked every default. The model is a plug: **Laya** today, with its
three checkpoints (English, multilingual, and a typed-decisions fine-tune) resident at once and
automatic routing between them, and whatever is trained here next behind the same interface
([engines/README.md](engines/README.md)). So is the machine: one recipe per accelerator under
[recipes/](recipes), NVIDIA and Apple Silicon both measured.

On one GB10 with the card to itself it answers a single question in **20.9 ms** and fifty
questions in one call in **152 ms**, measured end-to-end over HTTP; on an M2 Max Mac the same
calls take **30.3 ms** and **462 ms**. Realistic states — a suppo