I read a good piece on Hacker News this week — a team's writeup of migrating a production AI agent to GPT-5.6. The headline numbers are the sort that get shared without much thought: 2.2x faster, 27% cheaper. Nice. But the interesting bit, the bit worth your time, is everything underneath those two figures.
I've been shipping software since 1986 and running model-dependent pipelines across the Masher tools for years now — RSSMasher, MarketMasher, BookMasher and the rest all lean on LLMs somewhere in the chain. So when I see "we swapped the model and everything got better," my first instinct isn't celebration. It's suspicion. Because in my experience, swapping the model is the easy 10% of the work. The other 90% is the bit nobody puts in the headline.
The bit everyone skips
What I liked about this writeup is that they didn't skip it. They talk about re-running their eval suite against the new model before trusting a single production call to it. They talk about prompt drift — the fact that a prompt tuned against one model's quirks doesn't automatically transfer to the next one, even from the same vendor, even a point release apart. And they talk about the failure modes that only show up under load, not in a demo.
That's the pattern I've learned to expect from every model migration I've done across the Masher suite. The demo always looks fine. Production is where the model's actual personality shows up — its verbosity, its tendency to hedge, its handling of edge-case inputs you didn't think to test. GPT-5.6 might genuinely be faster and cheaper. It's also, almost by definition, a slightly different animal to whatever came before it, and your pipeline was built around the old animal's behaviour.
How I actually evaluate a model swap
When a new model drops that looks tempting for one of my tools, I don't touch the production prompt straight away. I run a shadow pass first — same inputs, both models, side by side, no user ever sees the new one. I'm looking for three things:
Output shape. Does it still respect the structure I need — the headings, the length, the format my downstream code parses? A model that's "better" in general benchmarks can still break your regex.
Cost at your actual volume. Percentage savings sound great until you multiply them by your real token counts on your real content mix, not a benchmark's toy inputs. Sometimes the saving evaporates because the new model needs longer prompts to behave.
Failure behaviour, not just success rate. When it gets something wrong, how does it get it wrong? A model that fails loudly and predictably is far easier to build safety nets around than one that fails quietly and plausibly. That second kind is the one that costs you a client's trust, not just a support ticket.
Only once a migration passes all three, on a decent sample of real traffic, does it get anywhere near a live pipeline. And even then it goes out gradually — a slice of traffic, not the whole lot, with the old model kept warm as a fallback for at least a couple of weeks.
Speed and cost are real, but they're not the whole story
None of this is to talk down what that team achieved. 2.2x speed and 27% lower cost is genuinely good, and for anyone running high-volume automation — content generation, video scripting, whatever your pipeline touches — that kind of gain compounds fast. Cheaper, faster inference is one of the few honest tailwinds in this industry right now, and I'll take it.
But the lesson from their writeup, and from my own scars building the Masher tools, is that the win only counts once it's proven in your context, with your prompts, your users, your edge cases. Raw model benchmarks are somebody else's gold. Your job is turning that raw material into something that holds up in your own pipeline, under your own load, on a Tuesday afternoon when nobody's watching the dashboard.
The teams who get burned by model migrations aren't the ones who move slowly. They're the ones who trust the headline number and skip the shadow run. Test first, ship gradually, keep the old model on standby. It's not glamorous advice, but forty years in this trade has taught me that glamorous advice is usually the advice that gets people into trouble.
— Wayne