Structured Outputs: Reliable JSON from a Model

Why please fails

Theory

A model doesn't follow rules; it predicts the next token, and every token is a probability, so the most a prompt can honestly promise is usually. Failures come in flavors: chatty preamble, code fences, a trailing comma, a missing closing brace, a renamed key. Prompting still helps with what goes in the fields, but it can't guarantee the shape of the container.

Practical example: Sure! Here's your JSON

Saying it again in capital letters doesn't help, and patching each failure with a regex leaves you maintaining a tiny, haunted parser.

import json

reply = 'Sure! Here is your JSON: {"age": 30} Have a great day!'
try:
    json.loads(reply)
except ValueError as err:
    print("parser says:", err)  # the chatty preamble breaks it

JSON mode

Theory

Many providers offer a switch that promises the output is valid JSON, which fixes the chatter, the code fences and the missing braces. But valid JSON is not the same as your JSON: it checks the grammar, not what you meant (details vary by API). Fine for prototypes; for anything with a schema, climb higher.

Practical example: Valid, and useless

Ask for an order with an id and a total, and you might get order_ID and amount instead.

import json

reply = '{"order_ID": "A17", "amount": 42}'  # valid JSON, wrong keys
order = json.loads(reply)                     # parses fine
print("id" in order, "total" in order)        # False False: not your JSON

Constrained decoding

Theory

Schema-constrained decoding changes how the model writes: the schema is compiled into a grammar, a state machine that knows which tokens may come next, and every token that would break the schema is blocked. The first request with a new schema can be slower; later ones reuse the work. Types, required keys and enums then match your schema by construction, for the supported schema subset, with refusals and max-token cutoffs as exceptions.

Practical example: Pending is not on the menu

If the schema says status is open or closed and the model wants to say pending, the decoder won't let it. Hosted APIs and open-source engines both do this, though the schema features they allow differ.

Tool calling

Theory

You describe a function with a JSON schema and the model answers by calling it with arguments; even if you never run the function, those arguments are structured output. Use tools when the model should trigger something, and a response schema when you just want the answer in a shape. A strict setting makes tool arguments follow the schema (setting names and rules differ); without it, treat them as best effort.

Practical example: Parse, then trust

In some APIs the arguments arrive as a string. Parse them, validate them, and only then run anything, especially the delete button.

import json

arguments = '{"order_id": "A17"}'  # in some APIs, tool arguments arrive as a string
args = json.loads(arguments)       # parse...
if not isinstance(args.get("order_id"), str):  # ...then validate, then trust
    raise ValueError("order_id: expected string")
print("look up order", args["order_id"])

Validate and retry

Theory

Validate anyway: your code is the last line of defense. A schema validator (Pydantic in Python, Zod in TypeScript) gives you a typed object or a list of exactly what went wrong. Feed that error back to the model and ask again, capped at two or three attempts, and add your own rules, like dates in the future or totals that don't add up.

Practical example: A repair loop

The model returned age as the word thirty; the validator says age, expected integer; you send that back and the next try returns the number. Log every failure with the raw output: that's your next eval dataset.

import json
def call_model(prompt):  # stub: the second try fixes the named mistake
    return '{"age": 30}' if "expected integer" in prompt else '{"age": "thirty"}'
def validate(text):
    data = json.loads(text)
    if not isinstance(data.get("age"), int):
        return None, "age: expected integer"
    return data, None
prompt = "Extract the age as JSON."
for attempt in range(3):  # cap the repair loop
    data, error = validate(call_model(prompt))
    if data:
        break
    prompt += "\nYour last answer failed validation: " + error
print(data)

What it misses

Theory

Constrained decoding doesn't guarantee four things. Meaning: valid is not correct. Truncation: hit the max token limit mid-object and you get half a JSON. Refusals, which can take precedence over your schema. And the schema subset: providers support only part of JSON Schema, and unsupported keywords can be rejected. Free-text fields are still free text.

Practical example: Check before you parse

Check for a refusal first, then the stop reason (named differently per API), and only then parse. Test your real schema on your real model.

import json

def parse(stop_reason, text):  # stop reason names differ per API
    if stop_reason == "refusal":
        return None, "refused"  # check for a refusal first
    if stop_reason in ("max_tokens", "incomplete"):
        return None, "truncated"  # ran out of tokens: don't trust the shape
    return json.loads(text), None

print(parse("max_tokens", '{"category": "bil'))

Schema design

Theory

Since the schema steers the model, design it like an API. Use enums for fixed lists, mark the fields you need as required, write descriptions, keep it small and shallow, and let the model say I don't know with a nullable field. Huge schemas compile slower, can hit provider limits, and give the model more ways to go wrong.

Practical example: A small, explicit schema

A key called date is a mystery; invoice date, ISO format, is a spec. If every field is mandatory with no way out, the model will fill in a guess. Enum casing isn't guaranteed everywhere, so compare values case-insensitively.

ticket_schema = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": ["billing", "bug", "other"]},
        "invoice_date": {
            "type": ["string", "null"],
            "description": "Invoice date, ISO format. Null if unknown.",
        },
    },
    "required": ["category", "invoice_date"],
    "additionalProperties": False,
}

Fallbacks

Theory

Plan the fallbacks before the pager does. Level one: retry with the error fed back. Level two: change the setup with a simpler schema, shorter input or a bigger output limit. Level three: a cheap repair step with a tested library, not a midnight regex. Level four: fail gracefully with a default, a human queue, or an honest error.

Practical example: Measure the rate

Never let a half-parsed object into the database. Track parse failures, retries and refusals like any metric; if one call in ten needs a retry, that's a schema bug.