Structured Outputs From LLMs: Stop Parsing Prose for a Living

There's a support group that meets in every AI company's Slack, and its topic is always the same: the model was asked for JSON and returned an essay about JSON. Structured output, getting a machine-readable answer instead of prose, is the unglamorous problem that decides whether LLMs are a demo or an ingredient, because software downstream can't if on a paragraph.

The ecosystem grew three answers. This page covers all three honestly, including where the newest one wins by refusing to play.

Answer one: schema-constrained generation

Frontier APIs now offer structured-output modes: hand over a JSON schema, and constrained decoding guarantees the response parses. This genuinely works and you should use it whenever you need a chat model's brains with a machine's manners. The residual failure isn't syntax anymore, it's semantics: the JSON always parses, but the model can still fill "category": "other" for everything when confused, hallucinate enum-adjacent values into free-text fields, or satisfy the schema while missing the point. Valid shape, wrong content, and no parser error to warn you.

Answer two: prompt-and-pray with a validator

Ask nicely for JSON, strip the markdown fences, try/catch the parse, retry on failure. Everyone's first version; fine for prototypes; a retry-loop cost multiplier and a 3 a.m. pager in production. If you're here, graduate to answer one this week.

Answer three: don't generate structure, decide it

Here's the reframe that the decision-model wave rode in on: a huge share of "I need structured output" cases were never generation problems. If your schema is really {category, urgency, spam: bool, score}, you don't need a model to write an object; you need four decisions, each from a closed set. A decision model like Jev only answers in that form: the label, the boolean, the score, natively, with nothing to parse and no essay to strip, and per ecosystem clients, with a probability attached to every choice. The structure problem doesn't get solved, it gets deleted.

The receipts are all over the directory: 500 emails classified for 3.5 cents with zero parsing code, a scorer asking 61 discrete questions per draft and getting 61 discrete answers, and the purest example, a PostgreSQL extension where WHERE jev(people, 'could work from home') returns booleans straight into a query plan, 129 rows in about a second (numbers as reported by build authors). SQL cannot be negotiated with; that build only exists because the output is structurally incapable of being prose.

The decision rule for choosing

Ask one question about your schema: do any fields require the model to compose free text? Summaries, rewritten sentences, extracted quotes: that's generation, use a chat model with schema mode, and add semantic validation (a judge question checking the content, not just the shape) if the stakes warrant. All fields closed-set: labels, booleans, scores, choices? That's decisions, and a decision model gives you native structure at decision-model prices. Mixed schema? Split it: decide the closed fields cheaply, generate the free-text fields with the big model, and enjoy discovering that the closed fields were 80 percent of the calls.

Frequently asked questions

What are structured outputs in LLMs?

Model responses in machine-readable form (JSON, labels, booleans) instead of prose, produced via schema-constrained generation on chat models or natively by decision models that only answer constrained questions.

Does JSON mode guarantee correct answers?

It guarantees parseable answers. Semantic errors (wrong category, evasive defaults, schema-valid nonsense) survive constrained decoding, which is why validation moved from syntax checks to judge-style content checks.

When should I use a decision model instead of JSON mode?

When every field in your schema comes from a closed set. Classification, routing, scoring, and flagging are decisions wearing a JSON costume; a decision model returns them directly, faster and cheaper, with no parsing layer to maintain.

How do decision models handle multi-field outputs?

As multiple questions: one per field. Reported builds run dozens of questions per item for fractions of a cent, so decomposition costs nothing and makes each field independently debuggable, which your future self will bill as the real feature.

What about extracting structured data from documents?

Extraction that copies spans is generation-shaped; use schema mode. Extraction that classifies what it finds ("is a date present?", "which clause type is this?") decomposes into decisions. Most document pipelines are a braid of both, split accordingly.