AI product development beyond a ChatGPT wrapper means engineering a dependable workflow around a model, not merely calling an API and displaying text. The work includes defining one job and its unacceptable errors, preparing and retrieving the right data, requiring a typed output contract, evaluating representative and adversarial inputs, restricting tools and permissions, designing for uncertainty and correction, controlling cost and latency, and assigning an owner to learn from real use. A model call is an ingredient; the product is the controlled system around it.
That distinction changes the release question. Instead of asking whether a prompt produces an impressive answer, ask whether the feature can show its evidence, defer when evidence is weak, protect data, remain useful when a provider is slow, and make changes reviewable. The framework below turns those questions into a release gate and a worked invoice-extraction example.
The 7C release gate for an AI product
A useful product-readiness review can be organised into seven gates: Context, Contract, Challenge, Contain, Communicate, Control, and Continue. The order matters. A better model cannot repair missing source material; a polished interface cannot compensate for an unbounded action; and a dashboard cannot create an owner who is responsible for the next change.
The gate is deliberately binary at the decision point: pass, hold, or explore. “Explore” is for a promising idea whose task, data, or risk is still unclear. “Hold” is not a judgement on the model. It means the surrounding product has not yet made a failure understandable and recoverable.
- Context — Define what the model may see, which sources are authoritative, how fresh they must be, and which records each user may access. For a knowledge feature, retrieval is part of the product. For an extraction feature, the original document, parsing quality, and provenance are the context.
- Contract — Turn free-form generation into a typed, validated result. Name required fields, allowed values, null behaviour, and what happens when the model cannot comply. A rejected result is safer than malformed text entering a database.
- Challenge — Build a representative evaluation set before treating a prompt or model change as an improvement. Include ordinary cases, boundary cases, missing information, ambiguous inputs, and hostile or instruction-like content where relevant.
- Contain — Treat model output as untrusted until it has passed the controls appropriate to the action. Separate reading from writing, grant the minimum tool permissions, and place deterministic checks or explicit confirmation in front of irreversible operations.
- Communicate — Give people a usable way to inspect, correct, defer, or reject the result. Show provenance when it exists, label drafts as drafts, and make uncertainty visible without pretending that a confidence value is proof.
- Control — Set a cost and latency envelope for each task. Limit context, choose a model appropriate to the work, cache only when freshness permits, and define a useful degraded state for timeouts or provider errors.
- Continue — Assign ownership for logs, feedback, evaluation updates, prompt and model changes, and rollback. An AI feature needs a learning loop because new inputs and changed dependencies can expose a failure that the original examples did not contain.
Each gate ends in pass, hold, or explore. Hold means the surrounding product has not yet made a failure understandable and recoverable.
AI Product Readiness Matrix and Release Gate
Use this matrix in a design review. Write down the evidence rather than awarding points for confidence. Thresholds are guidance to be chosen for the task and its consequences; they are not measurements of this article or claims about an existing system.
| Gate | Pass question | Hold condition | Evidence to record |
|---|---|---|---|
| Context | Can a reviewer trace each material answer or field to permitted, relevant input? | Sources are stale, duplicated, inaccessible, or impossible to inspect. | Source map, access rules, freshness decision, representative inputs. |
| Contract | Can the result be parsed, validated, stored, and rejected without silently changing its meaning? | The product parses prose with guesses, accepts missing required fields, or has no null and error state. | Schema, validation rules, invalid-output examples, fallback state. |
| Challenge | Do representative, boundary, ambiguous, and adversarial cases have an expected behaviour and a repeatable review method? | The evaluation set contains only clean demonstrations or has no reviewer for ambiguous cases. | Versioned cases, rubric, field-level checks, human-review record. |
| Contain | Are data access and tool permissions limited, with a deterministic check before consequential action? | Untrusted content can issue instructions, or a model can write where it only needs to read. | Permission map, action boundary, confirmation rule, refusal and fallback paths. |
| Communicate | Can a person see what is proposed, why it was proposed, and how to correct or defer it? | Confidence is hidden, provenance is unavailable, or the only choices are accept and abandon. | Screen states, source display, edit path, escalation route. |
| Control | Can the team observe cost and latency for the task and keep a slow or expensive call from degrading the whole workflow? | Context is unbounded, no budget exists, or a timeout leaves a blank or misleading result. | Budget policy, timing and usage fields, timeout state, degradation plan. |
| Continue | Is one person or team accountable for feedback, evaluation changes, dependency changes, and rollback? | No one owns the next review, or logs cannot explain which version produced a result. | Owner, release identifier, retention policy, change log, rollback procedure. |
A practical release rule follows from the table: pass only when every gate has an answer that another person can inspect. If a gate is unresolved, narrow the task, add a control, or keep the feature in exploration. Do not compensate for a missing safety or data decision by changing models until the demo looks better.
What the gates change in the build
Context: make evidence a first-class input
Start with the job, not a model catalogue. Identify the information needed, authoritative sources, freshness, access, and the behaviour when information is absent. Retrieval is part of the product: separate authoritative material from duplicates, preserve useful metadata, and make selected passages inspectable. For document workflows, keep the original file or source reference with extracted fields.
Tenant and permission boundaries belong here. Relevant is not automatically authorised, so carry the access decision into retrieval. Without a reliable source, return “not enough information”, ask for clarification, or use a manual route.
Contract: validate meaning before it reaches product state
A typed result gives the application explicit states. An invoice-extraction contract might require a vendor, non-negative total, supported currency, labelled line items, and a confidence signal. The fields depend on the workflow; the principle is to validate structure and meaning before storing or displaying a result as a business fact.
Reject missing required fields, unexpected values, impossible combinations, and values needing human confirmation. Keep the raw response separate from the validated representation where audit or debugging requires it. Confidence can help route a result to draft or review; it does not prove correctness.
Challenge: evaluate the task people actually perform
An evaluation set is a versioned description of good behaviour. Build it from permitted, representative examples and edge cases, not only attractive demos. For knowledge answers, label supporting sources and unacceptable unsupported leaps; for extraction, label important fields and the correct treatment of missing or ambiguous data.
Use field-level comparison for structured values and rubrics for open answers; keep human review for ambiguous cases and grader calibration. Test prompt, retrieval, schema, provider, and model changes against the same cases. Avoid a universal pass percentage: consequences differ, so set a task-specific floor, identify critical cases that must not regress, and document who can approve an exception.
Contain: separate suggestions from actions
Keep the action boundary explicit. A model may draft an email, propose a classification, or prepare a refund request; deterministic software and an authorised person should decide whether a consequential action happens. Do not grant write access when the task only needs reading.
Treat retrieved pages, uploaded files, and user text as data. Use least-privilege tools, schema and policy checks, sandboxing where needed, and confirmation before irreversible actions. Design the timeout path too: use a bounded retry, acceptable cache, manual queue, honest uncertainty, or a clear refusal with the next step.
Communicate: make uncertainty actionable
Uncertainty is a product state. Show whether the system is presenting a draft, a source-supported answer, a clarification request, or a blocked action, and offer edit, defer, or reject paths. Make correction inexpensive and preserve it for evaluation where policy permits.
Show relevant, readable provenance when the workflow allows it; do not present internal reasoning as proof. A source excerpt and editable result are usually more useful for review. Design latency states as well: stream interactive drafting or queue document jobs, while providing timeout and recovery states.
Control: make economics visible before they become a surprise
Cost control starts with the task boundary: retrieve only needed context, avoid repeating deterministic work, and choose a model that fits the capability required. Set an envelope for context, tool calls, response time, retries, and cost; record usage and timing by release. A cheaper path that weakens a critical field or makes review harder is not an automatic improvement.
Choose degradation by the job: a partial draft, a paused retry, a cached result when freshness permits, or a manual route.
Continue: turn corrections into a maintenance loop
Log the release, model or provider, prompt or configuration version, retrieved context identifiers, validation and action outcomes, and user correction where policy permits. Keep sensitive retention bounded.
Review corrections and rejected outputs; promote useful cases into the evaluation set, record why behaviour changed, and use a reversible control where possible. The aim is visible, recoverable change, not a frozen feature.
Example: invoice extraction with a review gate
Consider a product that turns an invoice or receipt into structured fields for an accounting review screen. This is an illustrative workflow, not a report of a shipped system or a measured result. Inputs may include a clean invoice, a scanned receipt, a multi-currency document, or an email thread pasted into an upload field. The model assists with extraction; it cannot post an accounting entry by itself.
Illustrative review screen: the original document stays beside the extracted fields, and every result leaves through edit, clarification, or rejection — never straight into the ledger.
1. Context. Preserve the original document, source identifier, readability decision, and access rules. Define whether an email thread or only its attachment is in scope; document content remains data, not a command to the tool layer.
2. Contract. Define fields such as vendor, total, currency, lineItems, and confidence, with rules for missing values and supported currencies. Validate the object before it reaches the accounting screen: malformed amounts, unsupported currency, or missing total should create review, not a guess.
3. Challenge. Label cases for the clean invoice, scanned receipt, multi-currency document, long email thread, missing field, and ambiguous total. Specify the expected fields and whether the right behaviour is extraction, clarification, or review. Include instruction-like content and expect it to be treated as document content.
4. Contain. Give extraction no authority to issue payment, alter a ledger, or send a message. If an accounting action is added, put deterministic validation and authorised confirmation between proposed fields and that action. Keep original and edited values distinguishable.
5. Communicate. Show extracted values beside supporting source areas where permitted. Mark the result draft or “needs review” when a required field is missing, the source is ambiguous, validation fails, or a task-specific review rule is triggered. Let the reviewer edit, reject, or request clarification; confidence is not proof.
6. Control. Bound context and retries; avoid irrelevant pages and repeated parsing. Choose an interactive or queued workflow with visible processing, delay, and failure states. Record usage and timing so changes can be compared without inventing a target from another workload.
7. Continue. Store schema version, input class, validation outcome, reviewer correction, and model configuration according to the data policy. Add recurring corrections or document patterns to the evaluation set.
The release decision is concrete: consider controlled release only when the schema parses, the input is in scope, evaluation cases have approved behaviour, and consequential actions remain gated. An ambiguous document without a review route is a hold.
Implementation steps for a release-ready build
- Write the job card. State the user, the input, the output, the decision it supports, and the actions the system must never take. Name the cost of a wrong field and the cost of a delayed answer.
- Map evidence and access. List source systems, freshness expectations, document boundaries, permissions, and provenance. Mark any assumption that still needs a product or domain owner to confirm.
- Draft the contract and states. Define the schema, null behaviour, validation errors, review state, clarification state, timeout state, and manual fallback before polishing the prompt.
- Assemble the evaluation pack. Label normal, boundary, ambiguous, missing-data, and hostile cases. Include expected outputs and expected non-actions. Keep the cases versioned and permitted for the intended environment.
- Build the containment boundary. Separate model calls from business writes. Add least-privilege tools, input handling, output validation, deterministic checks, explicit confirmation, and a route for refusal or deferral.
- Design the review experience. Show source and status, make edits reversible, identify generated text as a draft when appropriate, and provide a low-friction way to record a correction.
- Set operating controls. Choose context limits, retry rules, model routing, caching rules, latency handling, and an initial cost envelope. Add telemetry for release, usage, timing, validation, and fallback decisions.
- Assign the continuation owner. Decide who reviews feedback, approves changes, updates the evaluation pack, monitors failures, and can pause or roll back the feature. Put that responsibility in the delivery plan, not in an implied future task.
For a short release meeting, ask each gate owner to answer three questions: what evidence exists, what happens when the evidence is weak, and who can change the behaviour. A blank answer is a useful finding. It identifies the next piece of product work more clearly than another round of prompt polishing.
Every result has a next state: approved, blocked with a manual queue, or a bounded retry after a timeout — never a silent dead end.
Limitations and trade-offs
This framework is a decision aid, not a benchmark or a substitute for domain review. It does not supply universal quality thresholds because the acceptable error depends on the task. A draft headline, a customer-support answer, and a financial field need different release rules.
Evaluation sets are incomplete by design. They can miss new input patterns, drift as the underlying data changes, or encode a reviewer disagreement as if it were a single truth. Human review adds capacity and consistency work. Provenance makes inspection easier but does not make an extracted value correct. A visible uncertainty state can protect a workflow while also creating more review work; that trade-off should be acknowledged rather than hidden.
Retrieval is not automatically the right architecture. If the task has no changing knowledge source, a simpler deterministic or conventional software path may be clearer. Conversely, retrieval cannot repair an inaccessible, contradictory, or poorly maintained source. Model routing and caching can lower usage but add branches, invalidation rules, and new cases to evaluate. A smaller system with fewer moving parts may be preferable when the task is narrow.
Finally, this gate does not replace security assessment, privacy review, accessibility work, legal advice, or sector-specific controls. Those reviews should be added when the product context requires them. If the team cannot explain the data boundary, the action boundary, or the recovery path, narrowing the feature is often more responsible than adding more model capability.
Frequently asked questions
What is the difference between a ChatGPT wrapper and a real AI product?
A wrapper passes input to a model and displays the answer. A product defines one job and its unacceptable errors, then engineers the system around the model: permitted sources and freshness, a typed output contract, a representative evaluation set, least-privilege tools, review and correction paths, cost and latency envelopes, and an owner for the next change. If the result cannot show its evidence, be corrected, or be rolled back, it is still a demo.
Does every AI product need retrieval-augmented generation?
No. Use retrieval when answers must trace to permitted, current sources or when extraction must keep provenance. For closed tasks with stable rules, a validated schema and deterministic checks matter more. When you do retrieve, separate authoritative material from duplicates, carry tenant and permission decisions into retrieval, keep selected passages inspectable, and define the no-evidence behavior: state not enough information, ask for clarification, or use a manual route.
What should an AI evaluation set contain?
A versioned description of good behavior for the task people actually perform. Include ordinary cases, boundary cases, missing information, ambiguous inputs, and hostile or instruction-like content where relevant. For extraction, label required fields and correct handling of missing or ambiguous values with field-level checks. For open answers, label supporting sources and unacceptable leaps with a rubric and human review. Test prompt, retrieval, schema, provider and model changes against the same cases, protect critical cases from regression, and record who can approve an exception.
How should a product handle uncertain output?
Treat uncertainty as a product state. Show whether this is a draft, a source-supported answer, a clarification request, or a blocked action. Show relevant provenance, label drafts as drafts, and offer edit, defer, or reject paths. Validate structure and meaning before storage: reject missing fields, unexpected values, and impossible combinations. Treat retrieved pages, files and user text as data, grant minimum tool permissions, and put deterministic validation and authorized confirmation before payment, ledger, or outbound actions. Do not present internal reasoning or a confidence number as proof.
How can a team manage AI cost and latency?
Set an envelope per task for context, tool calls, response time, retries and cost, then record usage and timing by release. Retrieve only needed context, avoid repeating deterministic work, route each task to the smallest capable model, cache only when freshness permits, batch offline work and stream interactive drafting. Define degradation by the job: partial draft, paused retry, acceptable cached result, or manual queue with timeout and recovery states. A cheaper path that weakens a critical field or makes review harder is not an improvement.
When should a team use conventional software instead, and how long does production take?
Use conventional software when rules are stable, sources are fixed, or a deterministic check can decide the outcome — then reserve the model for drafting or classification inside those checks. A working demo can take days; a feature trusted with real work typically takes weeks to a couple of months, with most time in retrieval quality, evaluation, guardrails, UX for uncertainty, and iteration. If a 7C gate has no inspectable answer, narrow the task, add a control, or keep the feature in exploration.
CodeAustral's public applied-AI service description covers LLM feature integration, retrieval-augmented generation over a team's own content, tool-calling workflows, evaluation, cost and latency controls, guardrails, and fallbacks. The CodeAustral applied AI service track is the relevant place to review that delivery scope. A useful brief should include the user job, inputs, prohibited actions, representative examples, fallback state, and the person who will own iteration.