Language-model features fail differently from ordinary software. The same code path returns a different answer on each run, the inputs are open-ended, and a change to a prompt or a model version can quietly move the quality of every answer at once. The usual test suite does not catch this. An evaluation set does.
What an evaluation set is
A list of real inputs paired with what a good answer looks like, written by the people who know the domain. For a legal document assistant that meant forty questions from partners with the paragraphs they expected to be cited. For a support-ticket classifier it was two hundred historical tickets with the queue they actually belonged to.
The set runs in continuous integration like any other test. The output is a pass rate and a list of the cases that changed. We agree a threshold with the client before launch, and a deployment that drops below it does not go out.
What it catches
Model upgrades that help on average and hurt on your data. Prompt edits that fix one case and break three. Retrieval changes that start returning the wrong document for a whole category of query. Cost regressions, because the harness records tokens per case and the bill is part of the pass criteria.
What it costs
Usually two to four days of domain-expert time to write the first version, and an afternoon of engineering to wire it into CI. It is the cheapest insurance in the project, and it produces the one number a non-technical stakeholder can read before approving a launch.
Privacy comes first
The set itself contains real inputs, so it is handled with the same care as production data: redacted where required, stored in the client's own accounts, and never sent to a model provider that has not been approved for that data. For Canadian clients subject to PIPEDA or provincial health privacy law, this review is part of discovery, not an afterthought.