Control AI SaaS cost by routing requests within an entitlement boundary, rather than sending every request to whichever provider appears cheapest. Resolve the user's plan and capability access first; then check the task's quality, latency, context, data-handling, region, quota, and provider-health requirements. Only providers that satisfy those constraints should compete on cost. Premium, quality-sensitive work should have an approved equivalent route or a clear failure path, not a silent downgrade. Standard and low-risk work can use a lower-cost route when an evaluation set shows that it meets the same declared requirements.
Reserve quota before execution, keep the logical customer request separate from each provider attempt, and settle customer usage once the request reaches a defined final state. Record failed and retried attempts for infrastructure analysis even when the customer is charged or debited only once under the product policy. Classify errors before retrying, cap attempts and total exposure, and keep operator kill switches independent from ordinary plan logic. The result is a routing policy that makes cost a controlled trade-off instead of an accidental change to the premium experience.
The routing decision is part of product policy
An AI SaaS product may use multiple providers or models because different workloads have different capability, latency, availability, data-handling, geographic, or cost constraints. The hard part is not constructing another API call. It is deciding which request is eligible for which route, what happens when that route fails, and how the request affects the customer's entitlement and the business's infrastructure record.
Scattered rules such as “use provider A for paid users”, “switch when provider A is expensive”, “retry on any error”, or “count every attempt” hide policy. They can select unsuitable capability, degrade entitled traffic, double-count retries, or spend through an outage.
Treat routing as a small control plane with explicit inputs and outputs. Its inputs should include the resolved entitlement, operation, capability requirements, input and expected output bounds, data and region constraints, quota state, provider health, and temporary operator controls. Its outputs should include the selected route, an ordered fallback set, a reservation decision, a human-readable reason, and the accounting policy that applies when execution ends. A route decision that cannot be explained is difficult to support, audit, or change safely.
The central rule is: optimise among eligible routes, never by crossing an entitlement or data boundary. Provider names belong in configuration; product promises belong in policy.
The TRACE route card: an original decision framework
Use a route card for every request class. TRACE is a compact way to keep the important decisions in a stable order:
| Stage | Question to answer | Output to carry forward |
|---|---|---|
| Translate entitlement | What does this account and feature permit, and what must not be silently changed? | A normalised entitlement record |
| Record requirements | What capability, quality, latency, context, data, and region conditions does this operation need? | A constraint set |
| Allowlist candidates | Which configured routes satisfy both the entitlement and the request constraints? | An eligible route set |
| Check capacity | Which eligible routes are healthy, within quota, and not blocked by an operator control? | An ordered ready set |
| Execute and evidence | What route was chosen, what happened on each attempt, and how is the logical request settled? | A decision record and usage record |
The sequence matters. Do not compare cost before eligibility. A route that is cheaper but cannot satisfy the request is not a cost option; it is an invalid candidate. Likewise, a fallback should be selected during policy evaluation, not improvised inside an exception handler after the first provider has failed.
Store the route-card inputs and outputs with a request class, entitlement boundary, required output/data/latency constraints, approved allowlist, health/quota/control state, retry and exposure limits, accounting event, and reason code.
Keep this policy in one service or module rather than repeating it across controllers, background jobs, and provider adapters. The adapters should report normalised outcomes; the policy should decide what those outcomes mean.
Decision matrix: match the route to the promise
The following is a reusable policy matrix, not a claim about any particular provider. Replace the labels with the capabilities and commitments your product has actually documented.
| Request class | Minimum product contract | Preferred route | Fallback or downgrade rule | Cost and control boundary |
|---|---|---|---|---|
| Premium, quality-sensitive | The promised capability, data handling, and output requirements must remain intact. | The approved high-capability route for the entitlement. | Use an equivalent approved route; otherwise return a clear failure. Do not silently reduce capability. | Cost optimisation happens only among approved equivalents. A provider kill switch must not rewrite the entitlement. |
| Standard production task | The route must satisfy the operation's capability and output contract. | A balanced route selected from the approved set using quality, health, latency, and cost policy. | Use a pre-approved equivalent. A downgrade requires an explicit product rule and a user-visible or account-level policy. | Apply quota and budget rules before execution; preserve the request's data constraints. |
| Low-risk or asynchronous task | The task can tolerate the defined quality and latency envelope. | A lower-cost route that has passed the relevant evaluation. | Queue, defer, or use a bounded approved fallback. Do not turn an asynchronous job into an unlimited retry stream. | Use tighter token, concurrency, time, or unit limits where those controls are part of the product design. |
| Internal, preview, or test traffic | The audience and purpose permit a separate reliability and budget policy. | The configured test route or an approved internal route. | Fail clearly when the route is unavailable rather than leaking traffic into a customer route. | Keep reporting and budgets separate from customer traffic. |
This matrix prevents a common mistake: treating plan level as the whole routing decision. A premium account may submit a low-risk operation, while a standard account may submit a task with strict data or output requirements. Route on the request context as well as the account entitlement.
Quota reservation and successful-work settlement
Quotas are useful only when the product explains what they measure and the request path enforces the same definition. Depending on the product, the unit might be requests, tokens, generated artefacts, processing time, or a combination. Pick a unit that the customer can understand and the system can reconcile.
Separate three states:
- Entitlement: what the plan or account is permitted to request.
- Reservation: capacity temporarily held while a logical request is running.
- Usage: the work that the accounting policy says was actually consumed.
Reservation closes the concurrency gap: hold capacity before execution, then release or settle it at the final state. Make release, expiry, and settlement idempotent so delayed callbacks cannot debit a logical request twice.
“Successful-only accounting” needs a defined customer-consuming event. For example, settle only after a response reaches the customer and passes required structural checks; keep failed attempts in infrastructure records. Decide separately how to treat generated output discarded by an internal check.
| Event | Illustrative customer-usage treatment | Record that should remain available |
|---|---|---|
| Provider rejects the request before usable work is returned | Usually no customer usage under a successful-response policy; verify the product's chosen rule. | Logical request, provider response class, attempt identifier, and any known infrastructure cost. |
| Timeout with no usable response | No customer usage in this illustrative policy. | Timeout, elapsed time, retry state, and whether the external cost is known or unknown. |
| Valid response is returned | Settle usage according to the declared unit and entitlement. | Success event, route, attempt, output validation, and usage data. |
| A retry succeeds after an earlier failure | Settle the logical request once if that is the product rule; keep every attempt linked. | Parent request, each attempt, error class, route reason, and final settlement. |
| A response is generated but discarded by an internal check | Product-specific; do not silently classify it. | Discard reason, validation result, attempt data, and the chosen accounting outcome. |
Use one idempotency key for the logical customer request and a different attempt identifier for each provider call. That separation answers two distinct questions: what did the customer consume, and what did the infrastructure attempt? Do not erase attempt records merely because the customer sees one final response.
Reliability controls: bounded retries, kill switches, and evaluation
Retries should follow an error taxonomy, not a blanket exception handler. A transient network failure, a provider availability response, or a rate limit with a usable retry signal may be eligible for a retry. Invalid credentials, malformed input, an unsupported capability, a policy block, or a quota rejection normally needs a clear failure or a different product path instead. The exact classes depend on the provider contracts you have verified.
Set hard limits for the logical request:
- Maximum provider attempts.
- Maximum total retry time.
- Maximum cost exposure or unit exposure per request, when that limit is defined in the product policy.
- The providers, capabilities, and regions that a fallback may use.
- Whether a fallback may change output shape, data handling, or user-visible quality.
Use backoff and jitter where appropriate, and share the retry budget across gateways, adapters, queues, and workers; otherwise an outage can multiply traffic while hiding the cause.
Kill switches are operator controls, not substitutes for entitlement logic or observability. They may disable a provider or capability, stop retries, cap traffic, pause non-essential asynchronous work, force a tested fallback, or reject work above an exposure threshold. Give each an owner, scope, audit trail, safe default, and review condition; the request path must actually read it.
When a switch is activated, record who changed it, why, what route classes it affects, and when it should be reviewed. Keep the switch separate from normal business rules so removing a provider during an incident does not silently alter a customer's entitlement.
Evaluate cost changes on the tasks the product actually handles, including common cases, edge cases, structured outputs, long inputs, refusals, and relevant recovery paths. Compare routes on the same inputs across schema validity, task completion, supplied-context consistency, policy compliance, latency, failures, unit use, and human review where appropriate. Start with a bounded cohort or workload and retain route version, attempts, outcomes, and usage data.
Worked example: a standard-plan JSON summary request
Illustrative scenario, not production evidence: a standard-plan user submits a low-risk summarisation request. The product requires a valid JSON response, the input fits the operation's configured context boundary, the account has enough remaining quota, and the data may be sent only to routes approved for that handling requirement.
Translate entitlement. The plan permits the summary capability and the structured-output requirement. It does not automatically permit every provider or every model. The router creates a context containing the plan entitlement, operation, output schema, data rule, quota state, and current controls.
Record requirements. The request needs the approved summarisation capability and a valid JSON object. A candidate that cannot meet the schema or data constraint is removed before cost comparison. “Lower cost” cannot make an ineligible candidate acceptable.
Allowlist candidates. Suppose the approved set contains Route A, a balanced production route, and Route B, an equivalent fallback. Route C may be configured for lower-cost asynchronous work, but it is not in this request's eligible set because the policy has not approved it for this output or data requirement. These names and relationships are illustrative placeholders.
Check capacity. Route A is healthy, the reservation succeeds, and no operator control blocks it. The router records a reason such as standard_structured:eligible_balanced_route rather than only storing a provider name. The reservation is attached to the logical request before the provider attempt begins.
Execute and settle. Route A times out without returning a usable response. The error class is eligible for one bounded fallback attempt, and the request remains inside its time and exposure budget. Route B is selected because it satisfies the same capability and data requirements. If Route B returns valid JSON, the customer sees one result and the illustrative policy settles one logical usage event. Both provider attempts remain linked in infrastructure records. If Route B also fails, the system returns a clear failure, releases or closes the reservation according to policy, and does not silently send the request to Route C.
For a premium quality-sensitive entitlement, the eligible set would contain only an approved high-capability route and equivalent fallback. A cheaper lower-capability route is not permissible unless the product explicitly defines and communicates that change.
Implementation steps for a first controlled route
- Write the contract before the adapter. Document plan entitlements, request classes, permitted capability, output requirements, data handling, region rules, quota units, and any explicit downgrade behaviour.
- Normalise the route context. Resolve account and feature state once, then pass a typed context to the policy. Keep provider names out of customer-facing entitlement definitions.
- Build an allowlist, not a blacklist. For each request class, list routes that have been approved for the relevant capability and data requirements. Treat missing approval as ineligible.
- Centralise selection. Make one policy component choose the route, fallback order, reservation amount, and reason code. Provider adapters should translate external responses into your error taxonomy.
- Implement reservation and settlement separately. Reserve before execution, release or settle after the final state, and make repeated callbacks safe. Store logical request IDs and attempt IDs independently.
- Add a shared retry budget. Classify errors, set attempt/time/exposure limits, and prevent multiple infrastructure layers from spending their own uncoordinated retry budget.
- Instrument decisions. Log the policy version, request class, eligible candidates, chosen route, reason, control state, attempt outcomes, usage settlement, and known or unknown infrastructure cost. Avoid logging sensitive prompt or response content unless the product's data policy permits it.
- Exercise controls and evaluation before broad rollout. Confirm that kill switches affect the request path, run the representative quality set, compare cost and reliability dimensions together, and introduce a route change to a bounded workload before expanding it.
Pre-release checklist
- [ ] Entitlements are defined independently from provider names.
- [ ] Each request class has explicit capability, output, data, and latency requirements.
- [ ] Eligible routes are maintained as an allowlist.
- [ ] Premium fallback rules prohibit silent capability reduction unless an explicit product rule says otherwise.
- [ ] Quota entitlement, reservation, and usage are separate records or states.
- [ ] Successful-work accounting covers timeouts, invalid output, discarded output, and retries.
- [ ] Logical request IDs are separate from provider attempt IDs.
- [ ] Error classes determine whether a retry is allowed.
- [ ] Attempts, total time, and exposure have hard limits.
- [ ] Kill switches have owners, scope, audit history, and a review path.
- [ ] Route decisions include a human-readable reason.
- [ ] Quality, latency, failures, and infrastructure usage are reviewed together.
- [ ] Current provider contracts and data-handling requirements are verified before a route is approved.
Limitations and trade-offs
This framework does not identify a universally best provider or eliminate the need to verify current provider terms. Prices, capabilities, context limits, regional availability, data handling, error semantics, and service behaviour can change. Keep those details in versioned configuration and re-approve routes when a relevant external contract changes; do not encode an old assumption in a permanent plan rule.
Successful-only customer accounting is not the same as zero infrastructure exposure. The product may count one logical use while several provider attempts remain relevant to operational cost analysis. The exact treatment of an attempted request must be checked against the external provider agreement and the product's own billing language. If that policy is unclear, expose the ambiguity internally rather than presenting a neat but misleading quota number.
A fallback can preserve capability while changing latency, response style, or failure behaviour. If that difference matters to the customer promise, the route should be treated as a distinct product option rather than hidden behind a generic retry. Data residency, retention, security, and regulatory constraints can also reduce the eligible route set; routing logic cannot replace legal, security, or privacy review.
The route card adds operational complexity. A small product with one stable provider and limited concurrency may not benefit from a multi-route control plane yet. Conversely, a product with strict entitlements should prefer a clear failure over an unapproved downgrade when no equivalent route exists. Kill switches also require ownership and follow-up: leaving one active without review can quietly change service behaviour.
Evaluation sets cover only the tasks and constraints they contain. Refresh them when the product, data, output schema, provider, or model changes; a passing set does not replace monitoring or rollback.
FAQ
Should the cheapest provider be the default?
No. First filter candidates by entitlement, capability, output, data, region, and operational constraints. Compare cost only among routes that are eligible for the request class. For low-risk work, a lower-cost candidate may be appropriate after it has been evaluated for the actual task.
Does successful-only accounting make retries free?
No. It is a customer-usage rule, not a claim that every provider attempt has no infrastructure exposure. Settle the logical request according to the documented product policy and retain each attempt so operations can inspect what the retry path consumed.
How should premium users be handled during an outage?
Use an equivalent approved fallback if one exists. If it does not, return a clear failure or a deliberately designed degraded mode that the entitlement and product communication permit. Do not quietly substitute a lower-capability route merely because it is available.
What belongs in a route-decision log?
Record the logical request identifier, request class, entitlement decision, relevant non-sensitive constraints, eligible candidates, chosen route, policy version, reason code, operator-control state, attempt identifiers, outcome classes, settlement event, and known or unknown usage data. Avoid storing raw prompts or responses unless the data policy allows it.
Can quotas use tokens, requests, or generated artefacts?
Any of those units can be suitable for a particular product. Choose units that are understandable to users and enforceable in the request path. If more than one unit matters, document how reservation, exhaustion, reset, and settlement interact rather than presenting several unexplained limits.
When should a routing change be evaluated?
Evaluate before broad rollout and again when a provider, model, prompt contract, output schema, data rule, or fallback changes. Use the same representative inputs for the comparison, inspect quality and operational dimensions together, and retain a rollback route.
Should customers see provider names?
Not necessarily. Customers generally need a clear capability and entitlement description. Internally, stable request classes and route reasons make policy easier to operate; disclose provider changes when the product contract, data handling, or user experience makes that information material.
Final takeaway
Provider routing is the control surface where product promises, quota policy, reliability, and infrastructure spend meet. Start with the entitlement, translate the request into explicit constraints, allow only suitable routes, reserve capacity, retry within a shared budget, settle successful work consistently, record every attempt, and evaluate changes against real product tasks. That sequence makes a cost change reviewable instead of allowing it to become an accidental downgrade.
For implementation support, CodeAustral's AI integration services page describes evaluation, cost and latency controls, guardrails, and fallbacks. Define the problem with the route card and matrix before deciding whether outside engineering help fits.