Jev in Production: The Ops Guide the Launch Threads Skipped

Demos end at "it works"; production begins there. The directory is full of builds that crossed that line in week one, and this page collects what they converged on: the operational habits that separate a Jev verdict pipeline you trust at 3 a.m. from one you mute by Friday. Standing caveat first: rate limits, auth, SLAs, and anything contractual live at docs.typesafe.ai and change at launch-period speed; this page owns the patterns around the API, which outlive any particular limits.

Version everything that shapes a verdict

The question-craft rule graduates to law here: question text, boundary clauses, confidence thresholds, and routing policies are production config. Pin them in version control, tag every logged verdict with its question version, and treat a rewording as a breaking change with a diff and a rollout, because your metrics, your eval baselines, and your audit trail all silently fork the moment wording drifts. Teams that skip this spend month two doing archaeology on their own dashboards.

Cache aggressively, expire honestly

Verdicts on identical inputs are free money: exact-match cache first (an afternoon), semantic cache where inputs paraphrase (the reduce-costs page covers both), and domain-appropriate TTLs, since identity and category change slowly (entity resolution caches for weeks) while risk and safety verdicts should expire fast or never cache. Cache keys must include the question version, or a rewording serves stale rulings forever, the ops bug nobody enjoys finding.

Design for nondeterminism instead of denying it

The limitations page's warning becomes architecture: identical inputs can occasionally draw different verdicts, so anything requiring replayability (audits, datasets, billing-adjacent decisions) reads from logged verdicts, never from re-asking; confidence gates route the wobble to the escalation tier instead of into silent flip-flops; and retries distinguish transport failures (retry with backoff, idempotency keys on anything with side effects) from low-confidence verdicts (escalate, don't re-roll until you like the answer, which is p-hacking with extra steps).

Monitor the four signals that actually predict incidents

Latency and error rate, obviously, but the verdict-pipeline-specific two are where fires start: verdict-distribution drift (the spam-rate or category mix shifting without a world event usually means input drift, upstream breakage, or adversarial probing) and escalation-rate drift (rising unclears mean your questions and reality are diverging; falling to zero means your confidence gate broke, and both page-worthy). Wire the weekly human audit sample per labeling discipline as a standing job, not a launch ritual, and meter cost per pipeline per day, because the runaway-loop failure is invisible precisely when verdicts are cheap.

Batch the offline, gate the irreversible, rehearse the fallback

Backfills, labeling runs, and eval suites are batch jobs: run them as batches with checkpoints, not as a for-loop hammering the live path. Anything irreversible downstream keeps its verification gate regardless of upstream confidence. And decide now what each pipeline does when the API is unreachable: queue-and-wait suits labeling, fail-open suits nothing safety-adjacent, fail-closed-to-human suits triage; the wrong default chosen during an outage is how a five-minute blip becomes a week of cleanup.

Frequently asked questions

What's the most common production mistake with decision pipelines?

Unversioned questions: metrics silently forking under rewordings. Second place: retrying low-confidence verdicts until a preferred answer appears, which converts uncertainty into false confidence.

How should I handle rate limits?

Per the official docs' current numbers: client-side queues with backoff, batch offline work off the live path, and cache so repeat inputs never spend quota. Architecture outlives whatever the limits are this month.

Do I need a fallback model?

You need a fallback policy per pipeline (queue, degrade, or route-to-human); a second model is one implementation of it. Decide the policy before the first outage, not during.

What belongs in the weekly ops review?

Verdict-distribution and escalation-rate trends, the human-audit agreement number, cost per pipeline, and the incident-or-near-miss list feeding new eval cases and boundary clauses. Twenty minutes, standing agenda, no heroics.