A new model ships every week. Trying one takes 60 seconds. Trusting one in production takes something else entirely - and most teams skip that part. This is the small tool I built to stop skipping it, and why I run it before I change the model behind anything real.
The problem
I run a media company, not a lab. When a provider ships a new model, the question that lands on my desk is never "how does it benchmark." It's "should we switch our feature to it." Those are not the same question, and only one of them reaches users.
Swapping the model behind a feature is a production change. Most teams treat it like changing a setting: the new version scores higher, someone updates the model ID, it goes out. Nobody checked whether the new model still returns valid JSON for the routing prompt, whether it got slower from our servers, or what it now costs at our volume. You find out from the people using the product.
The gap between "it can" and "it's safe in my production" is the whole risk. A benchmark chart tells you how a model does on someone else's tasks. It says nothing about whether it survives yours.
What I actually check
So before any swap I run the same local preflight. It answers three things about the candidate model, measured from the environment that will run it in production:
- Does it still pass my actual prompt - not a generic one, mine, validated against the contract the feature depends on?
- How fast is it from here, start to finish - first token and total, at the percentiles that matter, not a single lucky call?
- What does it cost for my workload, including the retries a real run pays for?
Everything stays local by default - configs, prompts, results, and the failed outputs. A fast, malformed answer counts as a failure, because in production it is one. That last rule is the one most benchmarks quietly drop, and it's the one that bites you.
The core workflow
The workflow is deliberately boring. Put the approved model and the candidate in one config and run a migration check: three representative cases, once each, no warmup. It answers the cheap questions first - did the API work, did each response meet the basic contract, how quickly did the provider start and finish. It's a compatibility check, not a statistical verdict, and it runs in a few minutes.
The whole thing is three commands:
pip install llm-preflight
llm-preflight --init
llm-preflight benchmark.json --migration-checkThe first run needs no API key at all - --init creates a mock benchmark so you see the full report and exit behavior before a single paid request. When I ran the live version this week - two flagship models, two custom prompts, interactive selection through to the final recommendation - the entire comparison cost half a cent. The capture below is that exact run, not a staged one.
When that passes, I run the task-specific checks that match the feature - exact intent routing, structured output, whatever the thing actually has to preserve - before anyone approves the switch. And there's a catalogue side that lets me review what a provider has released, probe an uncertain model with one explicit request, and promote only the models that pass into an approved set. Nothing enters that set on vibes.
What it deliberately isn't
It's not a hosted evaluation platform, a tracing system, a RAG framework, or a public leaderboard. Those exist, they're good, and I'm not rebuilding them. The narrow, defensible thing is fast, local, cross-provider validation for the decision immediately before a swap. The moment it tries to be everything, it stops being the thing you actually run under deadline.
How it helps me
I've mostly stopped reading benchmark charts to decide a switch. I run the preflight and I get an artifact - a report I can hand to the team that says, in plain terms, this candidate passed our contract at this speed and this cost, or it didn't. "Should we switch" stops being a debate and becomes a check with a result attached.
That's also why I build these at all. I lead AI adoption across 160 people, and the fastest way to understand where AI actually helps - and where it quietly adds risk - is to build the thing myself and watch what breaks. This one exists because I got tired of "it benchmarks higher" being treated as "it's safe."
The tool is called llm-preflight - open source, MIT, no dependencies beyond the Python standard library. It started life last week as llm-speed-bench; I renamed it once it was obvious the speed table was the smallest part of the value. The name follows the job now: the preflight before the swap. Eleven releases in, the thing I want most is failure reports: point it at your provider and your prompts, and when something breaks, file an issue. The repo is at github.com/feronovak/llm-preflight.
If you ship anything on top of a model you don't control, the check is worth the three minutes. The new model isn't safe because it's newer. It's safe when it passes your test.
