Short answer: the moment a model returns data your application stores or acts on, that data needs the same treatment as any other untrusted input — a schema, validation, a bounded failure policy, and telemetry. Schema-first design means the contract is written before the prompt, every response is validated, and there is an explicit rule for what happens when validation fails. Anything less turns a probabilistic system into a source of corrupt records.
Schema validation is the gauge: nothing reaches the database until it fits the contract.
Why validation is the integration boundary
It is tempting to treat "structured output" as solved because providers offer JSON modes and tool calling. Those features improve the probability of well-formed output; they do not guarantee semantic correctness. A response can be valid JSON and still contain a category your system does not recognize, a date in the past that should be in the future, a total that does not equal the sum of its lines, or a tenant id that belongs to someone else.
Validation is where you decide what your system is willing to accept. It is also where you separate three different problems that teams often conflate:
- Syntax — is this parseable and shaped like the contract?
- Semantics — does it make sense given the request and the business rules?
- Authority — is this caller allowed to ask for this, and is the output allowed to be stored?
Provider features help with the first. Only your code handles the other two.
Define the contract before the prompt
Write the schema first and derive the prompt from it. The schema should be narrow: required fields, explicit types, enums instead of free text wherever the value space is known, and a deliberate choice about optional fields. Every optional field is a branch your code must handle; every untyped field is a validation gap.
A practical pattern is to keep two layers:
- Transport schema: what the model returns. Strict, minimal, versioned (
support-triage-v3). - Domain model: what your application stores. Validated, typed, and independent of the provider.
Map between them explicitly. If you find yourself storing the transport object directly, you have given a third-party system write access to your domain model.
The four outcomes of a response
Treat every model response as one of four outcomes, and give each one a defined action.
| Outcome | Detection | Action |
|---|---|---|
| Valid | Passes schema and business rules | Persist, act, log the bundle id |
| Repairable | Parseable, but fails a mechanical rule (unknown enum, wrong date format, missing optional) | Apply a deterministic repair, then re-validate; log the repair |
| Retryable | Malformed, truncated, or clearly incomplete | Retry once with the same bundle and a repair instruction, within a time budget |
| Unusable | Fails twice, or violates a hard rule (authority, safety, consistency) | Fall back to the manual path, escalate, or fail the request loudly |
The distinction matters because the costs differ. A deterministic repair is cheap and safe when the rule is mechanical. A retry doubles cost and latency, so it needs a budget. Falling back to a human path is the most expensive in the short term and the safest when the data is consequential.
A common mistake is unbounded retries. Two attempts is usually enough; a third rarely succeeds and multiplies your tail latency. If a task needs five attempts, the prompt, the schema, or the task decomposition is wrong.
Keep unvalidated data out of the database
This is the rule that prevents most incidents: nothing is persisted until it validates. Not "persist and fix later", not "store the raw response in a text column and reconcile weekly". Persisting invalid data creates work that grows: downstream jobs must defend against it, reports include it, and the eventual cleanup is a migration.
Three corollaries:
- No partial writes. If a record has five required fields and three validate, write nothing. A half-written record is worse than a failed one because it looks complete.
- Make the failure visible. The user should see that the automatic path did not complete and what to do next. Silent fallbacks destroy trust when they are discovered later.
- Keep the raw response for a bounded time. Store it in a diagnostic store with redaction and a short retention window, not in the domain table. It is the evidence you need to debug a failure, and it is a liability if it lives forever.
What to log per call
Telemetry is what turns a class of failures into a specific defect. At minimum, log per request:
- The bundle id: prompt version, model version, schema version, policy version.
- Input reference (hashed or redacted), not the full sensitive payload.
- Validation outcome, including which rule failed and which repair was applied.
- Attempt count, latency, token usage, and provider.
- The downstream action taken: persisted, queued, fell back, or rejected.
- A correlation id shared with the user-facing surface, so support can trace one complaint to one request.
Then aggregate. The metrics that matter are validation pass rate by schema version, repair rate, retry rate, fallback rate, and cost per successful record. A rising repair rate is an early warning that a model change shifted the shape of output; a rising fallback rate usually means the schema drifted away from the task.
Worked example: triaging support tickets
Consider an illustrative feature that classifies inbound support tickets into a queue, a priority, and a short summary. This is a composite scenario, not a client story.
The transport schema requires queue (enum of six known queues), priority (enum of three), summary (max 200 characters), and product_area (enum of twelve). The prompt is derived from the schema, with the enum values and a one-line definition of each.
Validation runs three layers. Syntax: parse and schema check. Semantics: the summary must be non-empty and must not contain a raw email address; the priority must be consistent with the queue for two hard combinations (a security report is never low priority). Authority: the classifier runs with a service identity that can write to the triage table only.
When the model returns an unknown queue, the deterministic repair maps near-miss strings to the closest enum value only if the edit distance is small, and otherwise marks the response repairable-but-unmapped, which routes the ticket to a general queue and flags it for review. When the response is truncated, one retry is allowed with an instruction to return a shorter summary. When both fail, the ticket goes to the general queue with a "classification pending" label and the raw response is stored in the diagnostic store for seven days.
After two weeks, the team has numbers: pass rate by queue, repair reasons, and retry rate. The repair log shows that one product area is consistently misclassified, which turns out to be a definition problem in the prompt, not a model problem. Fixing the definition raises the pass rate without touching the model — exactly the kind of fix the telemetry is supposed to make obvious.
Limitations and assumptions
This approach assumes you can define the output space well enough to write enums and required fields. For genuinely open-ended tasks — long-form drafting, exploratory analysis — a strict schema is the wrong tool; validate the properties you care about (length, presence of citations, absence of prohibited content) instead of the whole object. It assumes you control the persistence path, which is not true when the model writes directly to a third-party system through a connector; there, validation has to happen before the call, not after. It also assumes the cost of a fallback is acceptable. Where the manual path is very expensive, invest in the deterministic repair rules first and treat retries as the last resort. Finally, none of this replaces evaluation: validation catches structural and rule violations, not wrong-but-well-formed answers. Those need a labelled evaluation set, which is a separate discipline.
Working with CodeAustral
We integrate language models into existing products with the validation and failure paths designed first: versioned schemas, bounded retries, deterministic repairs, and telemetry that makes regressions attributable. If you are wiring a model into a system of record, read how we approach applied AI, look at the web engineering side, or send a short brief and we will review the contract with you. If the provider plumbing is still the open question, our OpenAI API integration guide covers that layer.
Frequently asked questions
Should we use the provider's JSON mode or write our own validation?
Use both. The provider mode raises the probability of parseable output; your validation decides what is acceptable. Provider modes change between model versions, and they cannot enforce business rules, so they are an optimization, not a control.
How many retries are reasonable for a structured output?
One, occasionally two, with a hard time budget. If a task routinely needs more, the task is too large or the schema is too strict. Splitting the extraction into two smaller calls is usually cheaper and more reliable than retrying one large one.
What should happen to the raw model response?
Keep it in a diagnostic store with redaction and a short retention window — days, not years. It is essential for debugging and it is a liability in your domain tables. Never let it become the system of record.
How do we handle a schema change without breaking production?
Version the schema, run both versions in parallel for a period, and log which version each request used. Change the transport schema first, keep the domain mapping stable, and only remove the old version when the repair and fallback logs show no traffic on it.