Evaluating LLM quality before it costs you
Cheaper models are only cheaper if quality holds. A small evaluation harness pays for itself the first time you switch models.
The hidden cost of a bad answer
A wrong answer does not show up on your model bill. It shows up as a support ticket, a churned customer or an engineer rerunning the same prompt four times. Any cost programme that ignores quality just moves spend somewhere less visible.
Build a small golden set
You do not need thousands of examples. Fifty to two hundred real prompts with known-good outputs, drawn from actual traffic, cover most regressions. Refresh them quarterly as usage shifts.
Score what matters to you
Pick two or three dimensions: factual correctness, format compliance, and tone or safety. Use a strong model as a judge for the subjective ones and exact matching for the structural ones. Consistency matters more than sophistication.
Then optimise cost safely
With a harness in place, model switching becomes an experiment instead of a gamble. Run the candidate model against the golden set, compare score and cost per task, and route only the traffic where quality holds. Most teams find they can serve 60-80% of requests on a cheaper model with no measurable quality loss.
Wire it into the pipeline
Run the evaluation on every prompt change and every model version bump. Quality regressions caught in CI cost minutes. Caught in production, they cost customers.