Structured Outputs: Valid JSON From Any LLM (2026)

TL;DR: All three major providers now constrain generation to your JSON schema at decode time: OpenAI via response_format: {"type": "json_schema", ...} with strict: true (Responses API: text.format), Anthropic via output_config.format: {"type": "json_schema", "schema": ...} (the old output_format parameter is deprecated), and Gemini via responseMimeType: "application/json" plus responseSchema. Schema tokens bill as input at normal rates — there is no extra surcharge — and you must still validate on your side, because safety refusals and max_tokens truncation return HTTP 200 with schema-violating output. Parameter names verified against provider docs 19 October 2026.

Every production LLM integration eventually hits the same wall: the model returns the right information in the wrong format. A missing field, a string where the number should be, a helpful preamble before the JSON — any of them breaks json.loads and, downstream, your pipeline. For years the fixes were prompts ("respond ONLY with valid JSON"), regex extraction, and retry loops. They work until they don't.

Structured outputs move the guarantee from your code to the API itself. The provider compiles your JSON schema into a grammar and masks the token distribution at each decoding step, so the model physically cannot emit a token that would violate the schema. The response you get matches the schema, or the API tells you why it can't. This post covers the exact parameter for each provider, the strict-mode rules that trip people up, what breaks in practice, and what it costs.

How constrained decoding works#

A language model generates one token at a time by sampling from a probability distribution over its vocabulary. Constrained decoding inserts a mask before the sampler: at each step, an incremental parser checks which candidate tokens can legally continue the partial output under your schema, and every token that would break the schema gets its probability zeroed out.

Schema:       { "rating": integer, "verdict": "pass" | "fail" }
Output so far: {"rating": 4, "verdict": "
Candidate token     Legal under the schema?    Result
-----------------   -----------------------    ------
pass                yes                        kept
fail                yes                        kept
"passed"            no (not in the enum)       masked out
4                   no (string expected)       masked out
Here is             no (not valid JSON here)   masked out

The sampler then picks only from the kept tokens. Two practical consequences follow from this design. First, the guarantee is structural, not semantic: the output will be valid JSON matching your schema, but a field marked {"type": "number"} can still hold a wrong number — values remain the model's judgment. Second, compiling the schema into a grammar costs time on the first request; providers cache the compiled grammar, so the latency penalty hits once per schema, not once per call. Anthropic's docs note compiled grammars are cached for 24 hours from last use; OpenAI documents the same first-request latency for a new schema.

OpenAI: response_format with strict: true#

Structured Outputs launched with gpt-4o-2024-08-06 in 2024 and is now the standard path on the Chat Completions API. The request shape:

from openai import OpenAI

client = OpenAI(base_url="https://tokens.bd/v1", api_key="YOUR_TOKENS_KEY")

response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": "Extract: Riya Karim, 29, Dhaka — backend engineer"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"},
                    "city": {"type": "string"},
                    "role": {"type": "string"},
                },
                "required": ["name", "age", "city", "role"],
                "additionalProperties": False,
            },
        },
    },
)

On the newer Responses API the same feature lives under text.format instead of response_format — same schema, different parameter path. Strict mode has hard rules, and every one of them fails loudly (400) rather than silently:

• All fields must be in required. Want an optional field? Use a union type: "type": ["string", "null"] — and the field still goes in required.
• additionalProperties: false on every object, so the model can't invent extra keys.
• Root must be an object. No top-level anyOf, no bare arrays at the root — wrap them in an object.
• A JSON Schema subset. Supported: primitives, objects, arrays, enum, anyOf, string pattern/format, numeric minimum/maximum, array minItems/maxItems. Unsupported: allOf, not, if/then/else, and (as of this writing) recursive schemas. Limits to keep in mind: roughly 5,000 properties, 10 nesting levels, 1,000 enum values.

The same strict: true flag works on function/tool definitions — OpenAI's own docs recommend always enabling it. That means your tools array gets the same decode-time guarantee as response_format: tool names and arguments come back schema-valid, not "probably parseable."

Anthropic: output_config.format (and strict tools)#

Claude's structured outputs arrived as a public beta on the Claude Developer Platform and has since settled on its parameter shape. The current request shape:

from anthropic import Anthropic

client = Anthropic(base_url="https://tokens.bd", api_key="YOUR_TOKENS_KEY")

msg = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Extract: Invoice #4821, total 12,450 BDT"}],
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "invoice_number": {"type": "string"},
                    "total": {"type": "number"},
                    "currency": {"type": "string"},
                },
                "required": ["invoice_number", "total", "currency"],
                "additionalProperties": False,
            }
        }
    },
)

Two things to know that older tutorials get wrong. First, output_config.format replaces the deprecated output_format parameter — if you see output_format in sample code, update it; per Anthropic's docs the beta name no longer appears. Second, the Python and TypeScript SDKs rewrite your schema automatically: unsupported keywords (like numeric minimum/maximum or string maxLength) are stripped from the constraint and moved into the field's description as instructions, then validated client-side against your original schema. Your response is still schema-conformant; the bounds just aren't enforced at the token level.

Anthropic's schema subset is similar in spirit to OpenAI's: additionalProperties: false required on objects, no recursive schemas, no external $ref — but with its own explicit complexity caps (roughly: no more than 20 strict tools, 24 optional parameters, 16 union-typed parameters per request; oversized schemas get a 400 with "Schema is too complex for compilation"). It also plays well with streaming, and strict tool use ("strict": true on a tool definition) uses the same grammar machinery — you can constrain both the final response and the tool inputs in one request.

One honest caveat from Anthropic's own docs: enum casing is not guaranteed to match exactly. Compare enum values case-insensitively on your side.

Gemini: responseMimeType + responseSchema#

Google's approach is the simplest parameter shape of the three, and it's been stable the longest. REST request shape:

{
  "contents": [{"parts": [{"text": "Classify this review: 'Delivery took two weeks but the product is great.'"}]}],
  "generationConfig": {
    "responseMimeType": "application/json",
    "responseSchema": {
      "type": "OBJECT",
      "properties": {
        "sentiment": {"type": "STRING", "enum": ["POSITIVE", "NEGATIVE", "NEUTRAL"]},
        "churn_risk": {"type": "NUMBER"},
        "needs_human": {"type": "BOOLEAN"}
      },
      "required": ["sentiment", "churn_risk", "needs_human"]
    }
  }
}

Both fields are required — responseMimeType alone switches Gemini to JSON mode (valid JSON, but no shape guarantee); responseSchema adds the contract. In the newer JavaScript SDK the same capability is exposed as responseJsonSchema, and there's an interactions.create path with a response_format object ({type: "text", mime_type: "application/json", schema}), which pairs nicely with Zod: define the schema in Zod, convert with z.toJSONSchema, then schema.parse(JSON.parse(...)) on the way back.

Google's own documentation is explicit on one point the other providers imply: validate again on your side. Gemini implements a subset of JSON Schema, very large or deeply nested schemas can be rejected, and the schema contract is your public API boundary — keep it closed and bounded. And one production tip that shows up repeatedly in developer reports: enums are the highest-leverage constraint on Gemini. "IN_STOCK" | "OUT_OF_STOCK" | "BACKORDER" eliminates an entire class of "in stock" vs "In Stock" inconsistencies at the token level.

The comparison, in one table#

Parameter names and rules verified against provider and third-party docs 19 October 2026.

What it costs: nothing extra#

There is no surcharge for structured outputs on any provider. The schema you send counts as input tokens at the model's normal input rate; the JSON the model returns counts as output tokens at the normal output rate. The guarantee is free; the tokens are not.

That makes the cost arithmetic trivially estimable. Take a 200-token schema attached to 10,000 classification calls — 2,000,000 input tokens for the schema alone:

Two things worth noticing. First, on cheap models the schema overhead is noise — $0.20 per 10,000 calls on Luna. Second, because the schema repeats on every call, it is a textbook prompt-caching win: a stable schema prefix gets cache-hit rates on subsequent requests, which is where the real savings come from on cache-heavy workloads. See prompt caching explained and the full line-item breakdown for the math.

Same request through your gateway#

If you access models through tokens.bd, nothing about this post changes except three lines. The gateway exposes OpenAI- and Anthropic-compatible endpoints (see the docs), so the same response_format parameter travels with the request — point the SDK at https://tokens.bd/v1 (OpenAI-style) or https://tokens.bd (Anthropic-style), swap the model name for one in the catalog, and use your tokens.bd key. The pattern from using any LLM with the OpenAI SDK applies verbatim. Two gateway-specific bonuses: per-key spend caps (see spend caps for coding agents) put a ceiling on a runaway extraction loop, and provider failover means a 429 on one backend doesn't break your pipeline — relevant here, since extraction workloads are exactly the bursty, retry-heavy shape that trips rate limits (see the rate-limits guide).

What still breaks (and how to handle it)#

Structured outputs remove the parsing failure class. Four failure classes survive:

1. Safety refusals. On Anthropic, a refused request returns stop_reason: "refusal" with HTTP 200 — billed normally, output possibly not matching your schema. Detect it by checking stop_reason before you parse, not by try/excepting the parse. OpenAI surfaces refusals programmatically on the Responses API; inspect response.output rather than just output_text.
2. max_tokens truncation. The schema guarantee doesn't protect you from running out of output budget. Truncated JSON is invalid JSON. Size max_tokens for the largest plausible schema instance, and treat truncation as a retry-with-more-budget case.
3. First-request latency. The schema compile on a brand-new schema adds latency to that one request. In latency-sensitive paths, warm the schema once at deploy time, or reuse a stable schema so compiled grammars stay cached.
4. Right shape, wrong values. The schema constrains structure, not truth. A {"confidence": 0.97} that should have been 0.12 is schema-valid and wrong. For high-stakes fields, keep your own validation layer — range checks, enum membership, cross-field consistency — exactly as you would with any untrusted input.

The practical production pattern: schema-first design (write the schema before the prompt — it forces you to decide what you actually need), json.loads with no defensive regex, a client-side validation pass, and a bounded retry for the four failure classes above. That is a shorter, more honest pipeline than the prompt-and-pray it replaces.

New this month: OpenAI's Decisions API (beta)#

Worth a sidebar, because it pushes structured output one step further. OpenAI launched a Decisions API in public beta — a specialized endpoint for the most common structured-output workload: yes/no, category choice, and score answers. It runs roughly 10x faster than the Responses API for these judgments, supports text and images, and the pricing is the headline: $0.10 per million input tokens with no output-token charge at all — you pay only for input. It's currently backed by GPT-6 Luna, and carries zero-data-retention and HIPAA eligibility for US/Europe use. If your workload is "classify 10,000 items into 5 buckets," a decision-style endpoint with no output bill beats a general structured-output call on both speed and cost. Beta caveats apply — evaluate it on your own labeled set before committing.

Structured outputs FAQ#

Is structured output the same as JSON mode? No. JSON mode (e.g. {"type": "json_object"}) guarantees valid JSON but not your JSON — the model can return any shape, any keys. Structured outputs bind generation to your exact schema: specific keys, types, enums, required fields. Use JSON mode only when the shape genuinely doesn't matter.

Does structured output make the model smarter? No. It constrains the shape of the output, not the quality of the judgment inside it. A wrong classification wrapped in perfect JSON is still a wrong classification. Use it for reliability of format; evaluate accuracy separately.

Can I use structured outputs with function calling? Yes — on OpenAI set strict: true on the function definition, on Anthropic set strict: true on the tool. The arguments the model passes to your tools get the same decode-time guarantee as a direct response. Both can be used in the same request.

Why did my schema get rejected with a 400? The usual suspects, in order: a field missing from required, a missing additionalProperties: false, a root that isn't an object (OpenAI), a recursive or oversized schema (Anthropic), or a keyword outside the provider's supported subset. The error messages name the offending constraint — read them; they're among the more helpful 400s in API-land.

Should I still validate the response? Yes. Validate after every parse — at minimum against your original schema, ideally with your own semantic checks (ranges, enums, cross-field consistency). The provider guarantees structure at decode time; your validation layer owns everything else.

Parameter names and pricing checked 19 October 2026. Provider docs are the authority; the model catalog lists current prices on Tokens.

Sources: OpenAI structured outputs via Groq docs · OpenAI/Anthropic comparison table · Anthropic output_config.format guide · Taskade structured outputs guide (2026) · Anthropic structured outputs launch coverage · Anthropic strict tool-use schema subset · Gemini structured output tutorial · Gemini REST JSON extraction walkthrough · Gemini schema versioning guidance · GPT-6.1 Sol pricing · OpenAI Decisions API coverage · Decisions API vs TypeSafe Jev