The useful cases are narrower and more boring than the marketing suggests, and they are also genuinely valuable. Telling them apart before spending money is the whole skill.
It involves reading something unstructured. It repeats often enough to matter. A person could write down what a correct answer looks like. And a wrong answer is caught by a person or a later step rather than going straight out of the door.
Reading inbound documents and filing them, extracting figures from supplier PDFs, classifying and routing enquiries, and searching a decade of contracts for a clause all fit that shape.
Anything where the right answer depends on context that exists only in somebody's head, and anything where being wrong once is expensive and undetectable.
Also anything already solved by a rule. If a process can be expressed as conditions on structured fields, a rule is cheaper, faster, and does not need evaluating every time a model changes underneath it.
Most pilots fail for one reason: nobody wrote down what success was before starting. A demonstration is judged on whether it looks impressive, which every model does, so the pilot ends in an argument about impressions.
The fix is to build the evaluation set first. Fifty real cases from your operation, with the correct answer recorded by the people who do the task today. Then any model, any approach, and any vendor can be scored against the same bar.
Model usage is metered per unit of text processed, so cost scales with volume rather than with seats. That makes it cheap to trial and easy to be surprised by at scale.
The controllable part is routing. Most cases in a real workload are straightforward and can go to a small, cheap model, with the difficult minority escalated. Built that way the bill is a fraction of sending everything to the largest model available.
That outputs cite the source they came from, so an answer can be checked. That low-confidence cases route to a person. That accuracy stays measured after launch rather than only at the pilot. And that your data is not used to train anybody's model, which is standard under enterprise terms and worth confirming in writing.
Checklist
Not under enterprise API terms from the major providers, which exclude training on submitted data. Where content is sensitive enough to warrant it, models can run inside your own environment instead. Confirm which applies before sending anything.
Almost never. Fine-tuning is rarely the lever people expect. Retrieval over your own content, with a general model reading it, covers most operational cases at a fraction of the effort.
The evaluation set gets rerun on a schedule. Models change underneath you, and a tool with no ongoing measurement is a tool whose accuracy you are guessing at.
Describe what is slowing the operation down. We will come back with the approach, the scope, and what it costs.
Get in touch