Cut LLM costs by treating each request as a measurable task rather than trying to make the entire bill cheaper. Log input, cached-input and output tokens, model choice, retries, escalations, latency and a task-specific quality result. Calculate cost per successful task. Then test one intervention at a time: deterministic routing for easy work, stable-prefix caching, shorter and more relevant context, or asynchronous processing. Keep a change only when the quality gate still passes and the complete workflow remains acceptable. The target is the lowest verified cost for useful work, not the lowest token rate.
Use the current rate card for the account and date you are analysing. Do not copy a price from an old article into a calculator. Keep fresh input, cached input and output as separate fields, and count every retry or escalation according to the billing record. The framework below turns those inputs into a decision rather than a guess.
Measure the unit you are actually trying to make cheaper
A monthly invoice is an outcome, not a useful unit of analysis. Start with a task that has a stable definition of done: classify a support request, extract fields from an invoice, answer a document question, draft a bounded description or plan the next action in a workflow. Give each task its own quality bar. A short classification and a multi-step answer should not be judged or budgeted as if they were the same work.
For every model call, capture enough information to reconstruct the task trace:
- task name, tenant or product surface where appropriate, and prompt version;
- model identifier and the reason for the route, if routing is used;
- input tokens, cached-input tokens and output tokens;
- latency, timeout, retry count and escalation count;
- a quality signal tied to the task, such as schema validity, evidence support, reviewer acceptance or a downstream business rule;
- whether the overall task succeeded after all calls, not merely whether one call returned text.
Calculate a call's token charge from the rate fields supplied by the relevant provider or account:
fresh_input = input_tokens - cached_input_tokens
call_cost = (
fresh_input * fresh_input_rate
+ cached_input_tokens * cached_input_rate
+ output_tokens * output_rate
) / 1,000,000Calculate each call separately when retries or escalations use another model or another rate. Then aggregate:
period_cost = sum(call_cost for every call in the task traces)
cost_per_successful_task = period_cost / count(successful_task = true)The denominator is the discipline. A response that is cheap but rejected, retried or escalated has not completed the job. Define “successful” before comparing candidates; otherwise a team can make a cheaper system look better by accepting weaker outputs or by counting only first attempts. Report the quality result and latency beside the cost number, never beneath it as an afterthought.
The CRAFT framework for quality-gated cost work
Use this five-part framework to decide what to change and when to stop. It is deliberately ordered so that a lower-cost choice does not outrun the evidence needed to trust it.
- Classify the work. Separate tasks by job, risk and context shape. Fixed-schema extraction, routing and short transformations may be candidates for a smaller or simpler path. Ambiguous reasoning, long-context synthesis and actions with difficult-to-detect errors need a stricter gate. Do not classify only by prompt length; a short request can still carry high consequence.
- Record the whole trace. A gateway total cannot tell you which task caused the spend or whether a second call repaired the first one. Preserve prompt version, token buckets, route, retry, escalation and outcome at the call site. Make the record append-only enough to compare a baseline with a candidate after a prompt or model change.
- Assign a quality contract. Write down the must-pass conditions, the acceptable latency ceiling and the disqualifying errors. For a structured extraction task, that might include valid schema, required fields and a review sample. For a document answer, it might include support from retrieved material and an acceptable “not found” behaviour. A model's own confidence can be a signal, but it should not be the only gate for a task where an error is expensive or hard to notice.
- Fit the least disruptive intervention. Match the remedy to the cost concentration. A stable repeated prefix suggests a caching test; redundant context suggests prompt or retrieval trimming; non-interactive work suggests deferred processing; a mixed task population suggests explicit routing. If a single task dominates spend, fix that task before polishing small contributors.
- Test, then keep or roll back. Compare the baseline and candidate with the same task definition and a versioned evaluation sample. Examine cost per successful task, quality, latency, retries and escalations together. A candidate that lowers token cost but breaches the quality contract is not a successful optimisation. Record the decision and the rollback condition so the next prompt, model or provider change does not erase the lesson.
Build the success-adjusted cost worksheet
The worksheet is intentionally provider-neutral. Add one row per task and one row per model call, or store the same fields in an event table. These are the minimum columns:
| Field | What to enter | Why it matters |
|---|---|---|
task_name and prompt_version | Stable labels | Makes comparisons reproducible |
task_volume | Tasks in the analysis period | Shows scale without hiding task mix |
input_tokens | Total input tokens per call | Captures context sent |
cached_input_tokens | The portion treated as cached by the account | Keeps billing categories separate |
output_tokens | Output tokens per call | Exposes unbounded responses |
fresh_input_rate, cached_input_rate, output_rate | Current rate-card values | Prevents stale assumptions |
retry_calls, escalation_calls | Additional calls, by route | Captures repair and hard-tail cost |
successful_tasks | Tasks passing the prewritten gate | Supplies the denominator |
quality_result and latency | Task-level measurements | Stops a cheaper but unusable path being selected |
Keep rate cards in configuration with an effective date. If a provider changes a billing rule, recalculate the affected period rather than silently mixing old and new rates. If the account has minimums, request charges or other non-token charges, add them as explicit columns; do not bury them in a made-up token rate.
A useful companion is an intervention ledger. For each proposed change, record change_id, task, baseline version, candidate version, intervention, quality gate, baseline and candidate cost per successful task, retry or escalation rate, latency, decision and rollback trigger. This makes the worksheet a reusable operating tool instead of a one-off spreadsheet.
Example: a 1,000-task support-answer worksheet
The following is a hypothetical worked example. Every number is illustrative accounting-unit arithmetic, not a provider price, production measurement, customer result or recommendation about a particular model. Replace the rate fields and observations with your own current records.
Assume a task has 1,600 input tokens and 500 output tokens per call. The example rate card uses 8 units per million fresh input tokens, 2 units per million cached input tokens and 24 units per million output tokens. The baseline averages 1.12 calls per task after retries or escalations and passes the task gate for 92% of the 1,000 tasks. The candidate changes only the reusable prefix: 1,000 of the 1,600 input tokens are now classified as cached. It has the same illustrative call count and the same illustrative pass rate; those conditions must be measured, not assumed, in a real rollout.
| Measure | Baseline | Candidate with cached prefix |
|---|---|---|
| Input tokens per call | 1,600 fresh | 600 fresh + 1,000 cached |
| Output tokens per call | 500 | 500 |
| Calls per task, including repairs | 1.12 | 1.12 |
| Tasks passing the gate | 920 | 920 |
| Cost per call in example units | 0.0248 | 0.0188 |
| Period cost in example units | 27.776 | 21.056 |
| Cost per successful task | 0.0302 | 0.0229 |
The arithmetic is ((1,600 × 8) + (500 × 24)) / 1,000,000 = 0.0248 for the baseline call. For the candidate it is ((600 × 8) + (1,000 × 2) + (500 × 24)) / 1,000,000 = 0.0188. Multiplying by 1,000 tasks and 1.12 calls per task gives the period totals; dividing by 920 successful tasks gives the final row.
Now imagine a second candidate with the same token pattern and period cost but only 880 tasks passing the gate. Its arithmetic cost per accepted task could still look favourable, yet it fails the written quality requirement if the baseline gate is 92% or higher. Reject it, investigate the missing answers or evidence, and do not present the lower number as a usable result. The point of the worksheet is to expose that decision, not to manufacture a universal saving estimate.
Decision matrix: choose the intervention that fits the failure mode
| Observed condition | First intervention to test | Keep the change when | Stop or reverse when |
|---|---|---|---|
| Spend cannot be attributed to a task | Instrument call and task traces | Every material call has tokens, route, outcome and version | Totals remain incomplete or the success denominator is unclear |
| A long reusable prefix is stable and repeated | Prefix-aware caching, subject to current provider rules | Cache behaviour is observable, content is versioned, and the task gate holds | Stale context, cache misses or invalidation risk affect correctness |
| Prompts repeat instructions or retrieve irrelevant context | Remove duplication; tighten retrieval and output format | The same required information remains available and quality holds | Evidence is missing, answers become less complete, or reviewers need repairs |
| Work is scheduled and users do not need an immediate answer | Deferred or batch processing using the current service terms | The delay fits the service-level agreement and failed jobs can be reconciled | Users wait for it, partial failures cannot be replayed, or freshness is unacceptable |
| Task difficulty is mixed but categories are recognisable | Deterministic routing by task type, length or validated metadata | Each route clears its own gate and route decisions are explainable | Misroutes, escalations or support burden increase |
| A cheap first pass can be checked cheaply | A cascade with a validator and explicit escalation path | The first pass resolves enough work while the combined cost and latency stay within limits | Escalation is frequent, the second call dominates, or the validator misses errors |
| One call is doing triage, extraction and synthesis | Decompose into bounded stages | The saved context outweighs orchestration and every stage is observable | More steps create retries, state bugs or longer user-visible latency |
| One task owns most of the spend | Optimise that task before broad changes | Its baseline and candidate can be compared on the same gate | The change merely shifts cost to another call or hides failures |
Do not combine routing, prompt compression and a provider change in the same first experiment. If the result moves, you will not know which decision caused it. A small intervention ledger with a named baseline is usually more informative than a large redesign.
Implementation steps for a safe rollout
- Choose one task. Write its definition of done, risk level, acceptable latency and disqualifying errors. State who can approve a change to the gate.
- Instrument at the call boundary. Attach the task and prompt version before the request is sent. Record all attempts, including a failed first call and its escalation, under one trace identifier.
- Load the current rate card. Store fresh, cached and output rates with an effective date. Calculate from usage records rather than from a rounded monthly total.
- Establish a baseline. Keep the existing route and collect enough comparable work to see normal variation. Preserve the task mix and label any known traffic shift.
- Create the quality sample. Include ordinary cases, long-context cases, malformed inputs and known failure modes that matter to the task. Version the sample and the evaluator so a later run remains comparable.
- Select one row from the matrix. Start with the lever that addresses the largest measured cost component while changing the least behaviour. For a cache experiment, first prove which content is stable and how it will be invalidated.
- Compare the complete outcome. Report total cost, cost per successful task, quality gate result, retries, escalations, latency and operational work. Break the result down by task type; an aggregate can conceal a regression in a small but sensitive group.
- Release with a rollback trigger. Define the trigger before exposure: a quality breach, stale-context signal, latency ceiling, escalation ceiling or unexplained billing mismatch. Revert the candidate when the trigger fires, then keep the trace for diagnosis.
- Maintain the decision. Version prompts, route rules, cache keys and evaluation data. Revisit the worksheet when the task, traffic mix or provider billing changes. Optimisation is a control loop, not a single model swap.
Limitations
This method cannot tell you a rate that is not present in your current account data. Provider pricing, cache eligibility, batch terms, model availability and billing categories change; use the applicable rate card and terms at the time of the decision. The article intentionally gives no vendor discount, price or savings claim.
Token spend is not the whole operating cost. Queueing, storage, observability, evaluation, human review, support and the engineering effort to maintain a route can matter. Add those items when the decision changes architecture or staffing. Conversely, do not add unrelated platform costs to a token comparison unless the same scope is used for both baseline and candidate.
A success rate can be misleading if the gate is vague, if reviewers accept different standards, or if a retry quietly changes the denominator. Keep task traces intact and distinguish “a model returned text” from “the product accepted the work”. A cache can reduce repeated input only while the cached material is valid; version and invalidate it when policy or source content changes. Compression and narrower retrieval can remove a detail that an evaluator did not cover, so include edge cases.
Cascades and decomposed pipelines can add calls, state, latency and new failure modes. They are not automatically cheaper. A capable model may remain the appropriate choice for a high-consequence task even when a smaller route costs less. The framework is a way to make that trade-off explicit; it is not evidence that every workload should be moved to a cheaper path.
FAQ
What is the most useful cost metric?
Use cost per successful task, calculated from every call in the trace and divided by tasks that pass a prewritten quality gate. Keep total spend, pass rate, latency and retries beside it. Cost per call is still useful diagnostically, but it cannot show the cost of repairs or rejected work.
Should I start by changing the model?
Usually start with attribution and a baseline. Once the expensive task and its failure modes are visible, choose the least disruptive intervention: remove redundant context, make a stable prefix cacheable, route recognisable low-risk work, or defer work that is not interactive. A model change without a task-level gate is difficult to interpret.
How should the worksheet handle cached input?
Keep cached-input tokens separate from fresh input tokens and enter the current account-specific rate for each category. If the provider's feature has prefix, minimum-size, expiry or invalidation rules, encode those rules in the experiment notes and verify them in usage records. Do not reuse a discount assumption from another provider or another date.
Is a model-based router worth its extra call?
Treat the router as another cost and latency line. Test deterministic signals first when they are sufficient, then compare a model-based router with the extra call included. Keep it only if route quality, total cost per successful task and latency satisfy the task contract. A confident routing decision that sends difficult work to the wrong path is still a failed decision.
When is batching or deferred processing appropriate?
Use it for work that can wait and whose failure can be retried or reconciled: scheduled classification, enrichment, backfills or report preparation are possible examples. Check the current service terms and your own freshness requirement. Do not move an interactive task to a deferred path merely because its token calculation is lower.
How do I know when to stop optimising?
Stop when the next change does not improve the complete decision, or when its engineering, operational or quality risk is larger than the measured benefit. Keep the baseline, candidate and rollback evidence. A stable, explainable route that meets the task contract is more valuable than a fragile lower-cost path that nobody can audit.
If you want this worksheet wired into a production path, CodeAustral's public AI development service describes evaluation, cost and latency controls, guardrails and fallbacks among its applied-AI capabilities. Use that as a scoping route: bring the task definition, usage fields, quality gate and operational constraints, and ask for an explicit comparison rather than a blanket promise.