结构化输出：如何从 LLM 获得可信的 JSON

Originally published at AI Frontier Post. Malformed JSON from LLMs doesn't have to be a fact of life. Here's the full reliability stack — strict schemas, provider enforcement, constrained decoding, and validation-and-repair loops — that ends the bug for good. Every LLM app builder has the same scar. You wire a model into a pipeline, ask it to return JSON, and it works in the demo. Then production arrives: a trailing comma here, a missing quote there, an extra explanatory paragraph wrapped around your payload, and your json.loads() explodes at 2 a.m. This is a solved problem. Getting JSON you can trust from an LLM is a stack of four layers — schema design, provider enforcement, constrained decoding, and validation loops — and you need all four. Here's how they fit together. Why "just ask for JSON" fails Asking a model to "respond in valid JSON" works most of the time, which is the worst kind of reliability. The failure modes are varied and annoying: Malformed syntax — trailing commas, unescaped quotes, markdown fences around the payload. Wrong shape — valid JSON, but a field is missing, renamed, or the wrong type. JSON mode (OpenAI's older json_object, Mistral's json_object) only guarantees syntax, not adherence to your schema. Truncation — a long response hits the token limit mid-object and you get half a document. Refusals — a safety refusal doesn't follow your schema at all, so parsing it blindly gives you garbage. No single fix covers all of these. That's why the answer is a layered stack, not one trick. Layer 1: Write a schema the model can actually satisfy Everything starts with the schema itself. A sloppy schema enforced perfectly is still sloppy. Design rules that pay off: Define the schema once, in code. Write a Pydantic model (Python) or Zod schema (JavaScript) and derive the JSON Schema from it. Your request constraint and your response validator then can never drift apart — they share one source of truth. Be strict and explicit. On OpenAI, set strict: true, list every key in required, and set additionalProperties: false. This is the difference between "hopefully the model cooperates" and server-side enforcement. Use enums for closed choices. "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]} beats hoping the model doesn't invent "mostly fine". Describe every field. Field descriptions become part of the model's instructions — they're prompts, not decoration. Keep schemas modest. Deep nesting, recursion, and exotic constraints are where even good providers wobble. Flatten what you can; the flatter the schema, the higher the hit rate. A small but honest example: from pydantic import BaseModel, Field from typing import Literal class SupportSummary(BaseModel): issues: list[str] = Field(description="Top issues mentioned, verbatim short phrases") sentiment: Literal["positive", "neutral", "negative"] confidence: float = Field(ge=0.0, le=1.0) Enter fullscreen mode Exit fullscreen mode Layer 2: Let the provider enforce it The big labs all offer server-side structured output now, though the knob differs per provider: Provider Mechanism Schema enforcement Notes OpenAI text.format with type: "json_schema", strict: true (Responses API) or response_format (Chat Completions) Full SDK helpers: responses.parse() takes a Pydantic model and returns typed output Google Gemini response_schema + response_mime_type: "application/json" Full Returns strictly validated JSON when a schema is set Anthropic (Claude) Tool Use: define a tool with input_schema, force it with tool_choice Full (via tool args) No generic JSON-mode switch; read the tool's arguments as your object. Newer SDKs increasingly add parse helpers on top Mistral response_format: {"type": "json_object"} JSON only, not your schema Valid JSON guaranteed; validate against your schema client-side AWS Bedrock Converse API toolConfig with JSON schema, forced via toolChoice Full Model-agnostic across hosted models The OpenAI pattern is the cleanest illustration of where the industry has landed: from openai import OpenAI from pydantic import BaseModel client = OpenAI() class CalendarEvent(BaseModel): name: str date: str participants: list[str] response = client.responses.parse( model="gpt-5", input=[ {"role": "developer", "content": "Extract the event information."}, {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, ], text_format=CalendarEvent, ) event = response.output_parsed # a CalendarEvent, not a string Enter fullscreen mode Exit fullscreen mode Note what this buys you: no parsing code at all. The typed object comes out of the SDK. Two caveats the official docs are honest about. First, the first request with a new schema has extra latency while the API processes the schema — subsequent identical-schema calls don't. Second, enforcement isn't absolute: a refusal or a response cut off by the token limit can still come back schema-invalid, which is why Layer 4 exists. Layer 3: When you self-host, constrain the decoder If you run your own models — Llama, Qwen, Mistral weights on your own GPUs — there's no provider API to lean on. The equivalent machinery is constrained decoding (also called structured generation), and it's arguably the most reliable approach of all. Instead of validating output after generation, constrained decoding enforces structure during token generation. At every step, the sampler masks out tokens that would violate your schema: if the next valid token must be a digit, string-starting tokens are blocked outright. The model can only ever sample schema-valid continuations, so the final output matches by construction — no retries, no post-processing. The ecosystem is mature: Libraries: Outlines, Microsoft Guidance, XGrammar Inference servers: vLLM and SGLang have structured generation built in (vLLM accepts structured_outputs: {"json": schema} or {"regex": "..."} in the request) The regex option deserves a mention: for fixed-format strings — order codes, dates, IDs — a regex constraint is lighter than a full schema and just as airtight. Layer 4: Validate everything, then run the repair loop Here is the non-negotiable rule, even with native provider enforcement: never trust a raw parse. Always validate every response against your schema client-side, because constraints can still be interrupted by length caps and refusals. The loop is generate → validate → repair: from pydantic import BaseModel, ValidationError import json def get_structured(call_llm, user_text, schema_cls, max_attempts=3): messages = [{"role": "user", "content": user_text}] for _ in range(max_attempts): raw = call_llm(messages) # JSON-mode / schema-enforced call try: return schema_cls.model_validate(json.loads(raw)) except (ValueError, ValidationError) as e: # Feed the validator's error back so the model self-corrects messages.append({"role": "assistant", "content": raw}) messages.append({"role": "user", "content": f"That was invalid: {e}\nReturn corrected JSON only."}) raise RuntimeError("No valid output after retries") Enter fullscreen mode Exit fullscreen mode Why this works so well: the validator's error message is specific ("field 'confidence' missing", "expected float, got string"), and models are excellent at fixing precisely-described mistakes. One repair attempt resolves the vast majority of transient failures. Three discipline points make this production-safe: Cap the attempts. Two to four tries, then stop. An unbounded correction loop is a cost and latency hazard — a model that can't produce your schema in three tries has a schema problem, not a luck problem. Fail typed, not loud. When retries are exhausted, return a structured error object (or a safe default), not an unhandled exception that kills the request. Fix the schema before retrying forever. Repeated failures on the same field mean the schema is ambiguous or too complex. Flatten it, add an example, or split the extraction into smaller calls. Anti-patterns to retire Anti-pattern Why it breaks Replace with json.loads(resp) right after a completion No schema check; throws on any malformation Native schema output + validated parse Regex-stripping markdown fences Regex can't reliably parse nested/escaped JSON Ask for raw JSON; stop requesting fenced output Bare JSON mode with no schema Valid JSON, wrong shape Schema-constrained output, or JSON mode + validate + retry data["field"] straight off the parse KeyError when a field is missing or renamed Validate into a typed model, then access attributes The takeaway Think of it as a reliability ladder. Good schemas give the model a target it can hit. Provider enforcement (OpenAI structured outputs, Gemini response schemas, Claude tool-use schemas) makes the target a guarantee in most cases. Constrained decoding brings the same guarantee to self-hosted models. And the validate-and-repair loop catches everything the other layers miss — truncations, refusals, and the odd provider hiccup. Most teams can stop climbing at Layer 2 plus the repair loop: that's already a two-nines improvement over prompt-and-pray. But if you implement only one thing from this article, make it the loop. Define your schema once in code, validate every response against it, and feed the error back on failure. Malformed-JSON bugs don't survive that discipline.