Language models make impressive demos and unreliable products, and the gap between the two is measurement. Before we build an AI feature we agree on what it is replacing, how well that thing currently performs, and what score would justify the change.
Establish the baseline first
If a clerk currently extracts fifteen fields from a scanned form with roughly two percent error, that is the number to beat. Without it, any accuracy figure is unanchored — ninety-four percent sounds excellent until you learn the humans were at ninety-eight.
Build the evaluation set before the feature
We assemble a held-out set of real cases with verified correct answers, deliberately weighted towards the hard ones: poor scans, unusual formats, missing fields, adversarial inputs. A hundred well-chosen cases beat a thousand easy ones.
- Sample from production data, not from what is convenient to collect
- Over-represent edge cases relative to their natural frequency
- Have the ground truth verified by someone who does the job today
- Version the set and keep it out of any prompt or fine-tuning data
Constrain the output, validate the result
Structured output schemas remove an entire category of failure. Beyond schema validity, business rules should run before anything is persisted: a date of birth in the future, a total that does not equal the sum of its lines, a national ID with a bad checksum. Rejected extractions go to human review rather than into the database.
Route by confidence, keep humans in the loop
Where being wrong is expensive, the correct design is not full automation. Set a confidence threshold, auto-process what clears it, and queue the rest for a reviewer with the source region highlighted. You get most of the throughput gain and keep the error rate under control.
Finally, the evaluation suite runs in CI. Any prompt change, model upgrade or retrieval tweak is scored against the same set, and a regression fails the build. Without that gate, a model version bump silently degrades a system nobody is watching.