Short answer: AI estimates fail when the uncertainty is collapsed into a single number too early. Break the work into stages — data readiness, task definition, evaluation, integration, and operations — and attach an exit criterion and a spend boundary to each. The estimate that matters is not "how much will this cost" but "what will we know after the next stage, and what is the maximum we are willing to spend to find out". Price the outcome per successful task, not per token.
AI estimates improve when the uncertainty is measured in stages, not guessed in one number.
Why AI estimates break
An AI feature estimate is really five estimates wearing one coat, and each has a different uncertainty profile.
- Data readiness. Is the input data available, structured, permitted, and representative? This is where most projects actually fail, and it is usually discovered late.
- Task definition. Is the task narrow enough to measure? "Summarise customer conversations" is not a task; "produce a five-line summary of a support ticket that a reviewer accepts without editing" is.
- Model behavior. Can the current generation of models do this reliably, at the quality bar, on your data? This is empirical and cannot be settled by argument.
- Evaluation. Can you tell whether the output is good? Without a labelled evaluation set, quality is an opinion and every change is a coin flip.
- Integration and operations. Where does the output go, what happens when it is wrong, what does it cost to run, and who is on the hook when it degrades?
A single-number estimate implies all five are known. In practice, only the last is usually well understood.
Estimate in stages with exit criteria
The useful structure is a sequence of stages, each with a deliverable, a spend boundary, and a criterion that tells you whether to continue. This converts an unanswerable question into a series of answerable ones.
| Stage | The question it answers | Deliverable | Exit criterion |
|---|---|---|---|
| Data readiness | Can we use the data we have, legally and practically? | Data inventory, access notes, a sample set | A representative sample exists with permission to use it |
| Task definition | Is the task narrow and measurable? | One-paragraph task statement, output contract | A reviewer can agree on what "correct" means |
| Feasibility spike | Can models do this at the required bar? | Prototype, error analysis on the sample | Quality clears the bar on the sample, or the task is re-cut |
| Evaluation harness | Can we measure quality repeatably? | Labelled set, scoring rubric, baseline numbers | A regression is detectable before release |
| Integration | Does it fit the product and the data model? | Validated pipeline, failure and fallback paths | End-to-end run on staging with failure modes exercised |
| Operations | What does it cost and who owns it? | Cost model, telemetry, runbook, rollback drill | Cost per successful task is known and acceptable |
Two properties make this structure work. First, each stage can be stopped: the maximum loss is the sum of the stages you have already funded, not the whole project. Second, each stage produces an artifact that survives a stop decision — a data inventory, a task statement, and an evaluation set are all reusable even if the feature is shelved.
Price per successful task
Token cost is a supplier metric, not a product metric. The number that belongs in a business case is cost per successful task, which accounts for the retries, escalations, and human reviews that a real system incurs.
A worked calculation, using clearly illustrative numbers: if a task costs 0.02 in model spend per attempt, needs 1.3 attempts on average, and 15 percent of tasks are escalated to a human reviewer who spends three minutes, then the cost per successful task is the model spend plus the expected human cost — and the human cost usually dominates. That is why quality work has a direct financial return: raising the acceptance rate lowers the human share, and it is often the largest line in the model.
Track four numbers per capability: attempt cost, attempts per task, acceptance rate, and escalation cost. Then define the target: a feature is viable when cost per successful task is below the value of the work it replaces or the price you charge for it.
The worksheet
For each stage, record:
- The question, the deliverable, and the exit criterion.
- The spend boundary in engineer-weeks, not just money, because the constraint is usually the team.
- The decision owner — who says "stop" or "continue".
- The assumptions the stage rests on, and which one, if wrong, changes the plan most.
- The fallback: what the product does if this stage fails.
Fill this in with the client in the room. The value is not the document; it is the argument about exit criteria, which surfaces disagreement about the goal while it is still cheap to resolve.
Worked example: a support deflection assistant
Consider an illustrative project: a B2B SaaS company wants an assistant that answers common support questions from its own documentation, with the goal of deflecting tickets. This is a composite, not a client story.
Stage one, data readiness, finds 400 help articles, of which roughly a third are outdated or contradict each other. The exit criterion is not met, so the first funded work is a documentation cleanup, not an AI project. That is an important result: the honest answer at stage one was "the model is not your problem yet".
Stage two defines the task narrowly: answer questions covered by the current documentation, cite the article used, and refuse when the answer is not in the corpus. Stage three runs a spike on 120 real past tickets with a rubric that scores answer support, citation correctness, and refusal behavior. The spike shows strong results on the top five question types and weak results on billing questions, which depend on account-specific data the assistant cannot see.
Stage four builds the labelled set and the scoring harness, and the team decides to launch only the top five categories, with billing questions routed to humans. Stage five integrates with validation, refusal handling, and a fallback to the existing contact form. Stage six measures cost per deflected ticket, including the human review sample, and sets a monthly review of the acceptance metric.
The feature ships narrower than the original ambition, on a date close to the original estimate, with a measured cost per outcome. That is what staged estimation buys: not a better guess, but an early decision about scope.
What usually breaks the estimate
- Data access takes longer than expected. Legal review, exports, and permissions are the hidden critical path.
- The evaluation set does not exist and cannot be bought. Building it is real work, and it is the project's foundation.
- Quality is acceptable on average and unacceptable in a segment. Segment-level results drive scope, and they are only visible with a labelled set.
- The happy path ships and the failure path is not funded. Fallback behavior is where the operations cost lives.
- A provider change moves the baseline. Pin versions, and budget a small maintenance allowance for re-evaluation.
- Nobody owns the metric after launch. Without an owner, quality drifts until a customer complains.
Limitations and assumptions
This method assumes the buyer can tolerate staged funding and a stop decision, which is not true in every procurement process; where a fixed price is mandatory, the practical approach is to price a bounded discovery stage and defer the build estimate until it completes. It assumes the organisation has, or can name, someone who owns the outcome metric — without that, stage exits become political rather than technical. It assumes the value of the outcome can be estimated at least roughly; where it cannot, the decision reduces to a bet, which should be labelled as one. Finally, the illustrative numbers here are deliberately generic: any real business case must use the team's own measured rates, and no estimate should present illustrative figures as observed data.
Working with CodeAustral
We scope AI features as staged investments with explicit exit criteria, build the evaluation harness early, and hand over a cost model per successful task. If you need a defensible estimate rather than a range, see how we approach applied AI, send a brief with the workflow you have in mind, and compare it with how we scope conventional product work.
Frequently asked questions
How accurate can an AI estimate be before a spike?
Before a feasibility spike, the honest range is wide — often a factor of two or more on the build phase — because model capability on your specific data is unknown. The useful precision at that point is not in the total but in the cost of the next stage and the criterion that ends it.
Should we build an evaluation set before or after the prototype?
Before, or at least alongside. A prototype without an evaluation set produces a demo that cannot be compared with anything. The evaluation set is also the artifact that survives if the project stops, which makes it the highest-value early deliverable.
How do we budget the ongoing cost of an AI feature?
Budget three lines: inference cost per successful task, human review or escalation, and maintenance — re-evaluation after model or data changes. The third line is the one teams forget, and it is the one that determines whether quality holds over a year.
What is the most common reason AI projects overrun?
Data readiness discovered late. Teams estimate the model work and assume the data is usable, then spend the build phase doing data cleanup that should have been the first funded stage.