jev-eval-agent

# jev-eval-agent

🇺🇸 English · 🇧🇷 [Leia em português](README.pt-BR.md)

A personal-assistant agent built with [eve](https://eve.dev) (Vercel), with **100 mocked tools**,
served through **OpenRouter**. The repository exists to answer one question: *how many steps does the
agent need to finish the same task when the LLM picks the tool itself vs. when
[Jev](https://docs.typesafe.ai) (TypeSafe's classifier) picks it?*

| `AGENT_MODE` | Who picks the tool | Model reasoning |
|---|---|---|
| `llm-direct` (default) | the LLM, seeing all 100 tools on every step | on (`DIRECT_REASONING`, preset default) |
| `jev-classifier` | Jev, called before **every** model step with the conversation state; only the chosen tool is exposed to the LLM, which just fills in the arguments | the minimum the endpoint accepts (`JEV_REASONING`) or off |

Everything lives on `main`; the mode is a single environment variable. Instructions, catalog, evals and UI are identical in both modes.

## How it works

```
agent/
  agent.ts              model (OpenRouter); reasoning and providerOptions depend on MODE
  mode.ts               reads AGENT_MODE (llm-direct | jev-classifier)
  instructions.md       shared system prompt
  instructions/mode.ts  extra guidance per mode (session.started)
  lib/models.ts         model presets (EVAL_MODEL)
  lib/catalog.ts        100 tools: name, description, zod schema and mock executor
  lib/world.ts          deterministic mock "world" (calendar, contacts, flights, expenses…)
  lib/jev-router.ts     builds the state from ctx.messages and asks Jev (choice with 101 options)
  lib/trace.ts          writes Jev's decisions to eval-results/jev-trace.jsonl
  tools/catalog.ts      defineDynamic: session.started → 100 tools | step.started → the 1 tool Jev picked
evals/
  prompts