Here is a question that decides more AI projects than any technical choice: how will everyone know when it is finished?
On most projects the honest answer is that somebody will look at it and form a view. There will be a demo. Some examples will work. The room will nod, the invoice will go out, and three months later a different conversation will start about whether it was ever any good.
That conversation is unwinnable, because nobody wrote down what good meant.
Why “we’ll know it when we see it” fails here
Traditional software has a natural acceptance test. The button either saves the record or it does not. You can sit down with a list of features and tick them off, and two reasonable people will usually agree on the ticks.
AI systems break that, for a specific reason. They produce an output for every input, always, with the same confident tone whether the answer is right or wrong. There is no error message. There is no crash. A system that is wrong a fifth of the time looks exactly like a system that is wrong a hundredth of the time, until somebody counts.
So the demo tells you almost nothing. A demo is a handful of examples chosen by the person who built it, which is the worst possible sample. The question is not whether it answered those three questions correctly. It is what proportion of the questions your business actually asks it will answer correctly, and how it behaves on the ones it gets wrong.
That is a measurement, and measurements need to be agreed in advance or they turn into arguments.
What agreeing the number actually involves
It is less exotic than it sounds. Three things have to exist before the build starts.
A set of real questions with known-good answers. Drawn from the work the business actually does, including the awkward cases rather than the tidy ones. This is the part clients expect to be hard and usually is not, because the questions already exist. They are in the inbox, the ticket queue and the head of whoever currently does the job.
A statement of what is being measured. Accuracy alone is rarely the right target. For a system answering from documents, the useful measures are whether each claim traces to a real source passage, and whether the system declines when the answer is not in the material. A system that answers everything is not better than one that abstains sensibly. It is worse, and it hides the fact.
A threshold, written down before the run. This is the part that makes the rest honest. A score chosen after seeing the result is not a threshold, it is a rationalisation. We covered how accuracy actually gets proven in more detail separately.
The part clients find surprising
Refusal counts as a correct answer.
If the material does not contain the answer, the right output is to say so. A system judged only on how often it answers will learn to guess, and guessing is the failure mode that costs businesses money, because a confident wrong answer gets acted on.
So the target is not “answers everything correctly”. It is “answers correctly when it can, and declines when it cannot, and both of those are counted”.
This tends to land well with owners once it is explained, because it matches how they already think about the people who work for them. The person you trust is not the one who always has an answer. It is the one who says when they do not know.
What it changes commercially
This is where it stops being a technical practice and starts being a different kind of engagement.
Fixed scope becomes survivable. A fixed price on a vague outcome is a gamble for both sides. A fixed price against an agreed score is a defined piece of work with a defined finish line.
The argument moves to the front. Any disagreement about what the system should do surfaces while the test set is being built, which is the cheapest possible moment for it to surface. The alternative is finding out at handover.
Nobody is asked to judge on impressions. The client does not have to develop an opinion about retrieval quality. They have to agree that these two hundred questions represent their work, which is something only they can do and the thing they are expert in.
A failed run is visible instead of deniable. This is the uncomfortable one, and it is the point. If the number comes back under the threshold, the thing is not finished, whatever the demo looked like.
We run this against a research system of our own, so the awkward version is not hypothetical. Runs have come back worse than the run before them. A change that felt like an obvious improvement has scored lower and not shipped. That is the harness doing its job, and it is only useful because the threshold was fixed before the run rather than after it.
The objection
The reasonable pushback is that this front-loads work onto a client who came to you precisely because they did not want to think about it.
That is fair, and it is why the test set is built with them rather than handed to them as homework. Their part is domain judgement: is this a real question, is this a correct answer, would you accept this response from a member of staff. The rest is not their problem.
There is a second objection, which is that a test set can be gamed. It can. A supplier who builds the questions, sets the threshold and marks their own work has produced a number that means nothing. The protection is that the questions come from the business, and the business can add awkward ones at any point.
What to ask for
If you are buying an AI system, four questions will tell you most of what you need to know.
What is the test set, how big is it, and who wrote the answers? If the supplier wrote both the questions and the answers, the score measures agreement with themselves.
What threshold are you committing to, and when was it set? Before the build, or after the first result arrived.
What does the system do when it does not know? And is that counted, or ignored.
Do I get the test set and the harness at the end? If the measurement leaves with the supplier, you cannot tell next year whether anything has drifted.
A supplier who does this work will enjoy those questions, because they are the questions their own process answers weekly. One who does not will steer the conversation back to the demo.
Frequently asked questions
What is a definition of done for an AI project?
An agreed score, on an agreed set of test questions, with a threshold written down before the build starts. It replaces the usual acceptance test of a demo and a subjective judgement.
Why can’t I just judge an AI system by using it?
Because it produces a plausible output every time, so occasional wrong answers are invisible in casual use. A handful of examples cannot distinguish a system that is right 95% of the time from one that is right 80% of the time, and the difference between those two matters commercially.
Should refusal count as a failure?
No. If the answer is not in the source material, declining is the correct behaviour and should be scored as correct. A system rewarded only for answering will guess, and confident wrong answers are the expensive failure mode.
Who should write the test questions?
The business, with help on structure. The questions have to reflect the work that is actually done, including the difficult cases, and only the people doing that work know what those are.
What happens if the system misses the threshold?
It is not finished. That is the entire value of setting the number in advance, and it is why the threshold has to be fixed before anyone sees a result.
Flux Dynamics is a fractional CTO who builds. We agree the acceptance number with you before writing code, because a system nobody measured is indistinguishable from one that did not work. Tell us what you are considering.