Short answer: treat every AI capability as reversible by design. Pin and version the things that change behavior (prompt, model, provider, retrieval index, tool permissions, guardrails), put each capability behind its own flag, define what the product does when the model is unavailable or wrong, and give one named person the authority to switch it off. If you cannot answer "who turns this off, and what happens next" in one sentence, the feature is not ready for production.
Every AI capability needs an owner and a switch that actually works under pressure.
Reversibility is a feature, not an operational afterthought
Traditional software fails loudly: a deploy breaks a page, the error rate spikes, you roll back the release. AI features fail quietly. Output quality drifts after a provider update, a prompt tweak makes one edge case worse, retrieval starts returning a stale document, or cost per task doubles because average output length grew. None of these produce a stack trace.
That asymmetry is why AI needs more rollback machinery than the rest of your stack, not less. The goal is not to prevent every failure. The goal is to make the blast radius small and the recovery boring: turn off the capability, keep the workflow usable, investigate with the evidence you kept.
Version everything that changes behavior
A deployed AI feature is not one artifact. It is a bundle of things that can change independently, often outside your release cycle. If any of them is unversioned, you cannot reason about a regression, because you cannot tell what changed.
| Component | Why it needs a version | Minimum practice |
|---|---|---|
| Prompt / instructions | A wording change can shift behavior across every request | Store prompts in version control, reference by id, log the id per request |
| Model | Providers update, deprecate, and silently re-tune models | Pin an explicit model version; treat upgrades as reviewed changes |
| Provider routing | A fallback provider may behave differently on the same input | Record which provider served each request |
| Retrieval index | Re-embedding or re-chunking changes what the model sees | Version indexes; make index id part of the request record |
| Tool permissions | An agent with a new tool has a new failure surface | Scope tools per capability; review permission changes like code |
| Guardrail / policy config | Blocklists and thresholds change what is allowed | Version the config; log which policy version was applied |
The practical test is simple: given one bad output from last Tuesday, can you reconstruct exactly which bundle produced it? If not, add the missing version to the request record before adding anything else.
Define degraded modes before you need them
Most teams design two states: working and broken. Production needs at least three: full capability, degraded capability, and off. Degraded is the state that saves the product, because it lets people keep working while the AI is unavailable or untrusted.
| Failure | Degraded behavior | What the user sees | Owner |
|---|---|---|---|
| Provider outage or timeout | Queue the request, or return a structured draft the human completes | "We could not reach the assistant; your draft is saved" | On-call engineer |
| Output fails schema validation twice | Fall back to a template or manual path | "We could not extract this automatically; review these fields" | Feature owner |
| Quality regression detected | Switch to the previous model/prompt bundle | No change for the user unless the older bundle is visibly different | Feature owner |
| Cost anomaly | Apply a stricter routing policy or lower quota for the capability | Possibly slower or lower-fidelity output, with a stated limit | Product owner |
| Unsafe or policy-violating output | Block the capability for the affected scope, log, and notify | "This action is unavailable right now" | Trust/security owner |
Write the degraded behavior down before launch, and test it. A degraded mode that has never been exercised is a diagram, not a mechanism.
Shadow evaluation and canaries
Two techniques buy most of the safety for a fraction of the effort.
Shadow mode runs the new bundle on real traffic without showing the output to anyone. You compare the new bundle against the current one on the same inputs and inspect the differences. This is the cheapest way to catch a regression before a user does, and it works for prompts, models, and retrieval changes alike. It requires that you can log inputs and outputs, which is a data-handling decision you should make deliberately — redact or hash anything sensitive, and keep retention short.
Canary exposure gives the new bundle to a small slice of traffic — an internal team, one tenant, one percent of sessions — with the same measurement you use for the current bundle. The canary is only useful if you have a comparison metric that is close to the product outcome: acceptance rate of suggestions, edit distance from the accepted draft, escalation rate to a human, or time to complete the task. Token cost alone will not tell you whether quality improved. The evaluation set behind those metrics is the subject of our AI agent release checklist.
The kill switch: one owner, one drill
The kill switch is not a flag in a dashboard nobody has permission to touch. It is a small, rehearsed procedure.
- One flag per capability, not one global "AI on/off". Turning off invoice extraction should not disable support reply drafting.
- A named owner per flag, with a named backup who is reachable in the on-call rotation.
- A documented blast radius: which workflows change, which users are affected, which integrations call the capability.
- A user-facing message that says what happened in plain language and what to do instead.
- A drill at least once per quarter. Flip the flag in staging (or in production during a quiet window), confirm the degraded path works, confirm the alert fires, and confirm the owner is notified.
- An investigation hook: turning the feature off should open an incident record with the request ids, prompt versions, and provider identifiers needed to debug.
If flipping the switch requires a deploy, you do not have a kill switch; you have a code change with extra steps.
Worked example: a document extraction feature
Consider an illustrative product that extracts line items from uploaded invoices and posts them to an accounting integration. This is a composite scenario, not a client story.
The bundle is versioned: prompt invoice-extract@14, model provider-a/large-2026-05, schema line-item-v3, extraction policy finance-default. Each request logs the bundle id and a hashed document reference.
Three modes exist before launch. Full mode extracts and pre-fills the review screen. Degraded mode runs when validation fails twice: the line items are left empty, the document is attached to a manual entry form, and the user is told the extraction needs review. Off mode removes the extraction button, keeps the upload and manual entry path, and shows a short notice.
The rollout is staged. Week one is shadow mode: extractions run but are only visible to the engineering team, compared against the previous bundle on the same documents. Week two exposes the new bundle to the finance team, with the acceptance rate per document type as the comparison metric. Week three exposes it to ten percent of tenants. The kill switch has one owner (the finance product owner) and one backup (the on-call engineer), and the drill is scheduled for the week after rollout, while attention is still high.
Limitations and assumptions
This playbook assumes you can deploy independently of provider changes, which requires a service layer between your product and the model. It assumes you can log inputs and outputs in a way that respects your data commitments; where you cannot, shadow evaluation has to run on synthetic or redacted data and will catch less. It assumes a real owner exists with the authority to make the call at 2 a.m. — organizations that route every such decision through a committee will not get the benefit. It also assumes that "off" is genuinely acceptable for the business, which is worth confirming before launch: if switching the feature off halts revenue operations, the degraded mode has to carry the load, and it must be tested harder than the happy path.
Working with CodeAustral
We build AI features with the rollback surface designed in: versioned bundles, per-capability flags, degraded modes, and an evaluation harness that tells you when to use them. If you are shipping an AI capability into a regulated or revenue-critical workflow, see how we approach applied AI or send us a brief and we will review the failure paths with you.
Frequently asked questions
Do we need a kill switch for an internal AI tool?
Yes, and usually more than for a customer-facing one. Internal tools sit close to operational decisions and rarely get the same review. A wrong extraction that feeds a finance system is more expensive than a wrong suggestion shown to a customer.
What is the difference between a feature flag and a kill switch?
A feature flag controls exposure; a kill switch is an operational procedure that uses a flag. The procedure includes the owner, the blast radius, the user-facing message, the alert, and the drill. Flags without a procedure tend to be discovered, not used.
How do we version prompts without slowing down iteration?
Reference prompts by id and version in code, and let the prompt text live in a store you can update quickly. The constraint is that every request records the version it used, so a regression is attributable. Fast iteration and traceability are compatible; what is not compatible is editing prompts in a dashboard with no record.
How often should we re-run evaluations?
Whenever the bundle changes, and on a schedule for the model and index you do not control — monthly is a reasonable default for a production capability. Also re-run after any incident, and treat the evaluation set as a living asset: every confirmed defect should become a case in it.