The model your app runs on will be retired
AI models get deprecated, providers go down, and the same prompt gives a different answer every time. Here's what we've built across our own products so the thing keeps working.
In February this year Anthropic retired Claude 3.5 Haiku. One of our products was calling it by the alias claude-3-5-haiku-latest, which sounds like the kind of name that looks after itself. It doesn’t. “Latest” means the latest 3.5 Haiku, and when there are no 3.5 Haikus left, the call fails. End of. The fix was three lines, but somebody had to know it was coming, and the product had been running happily for a year without anyone needing to think about it.
That’s the first thing to understand about building on AI: the thing you’re building on has a shelf life, and it’s shorter than anything else in your stack. Postgres from 2015 still runs. A model from 2025 mostly doesn’t.
Hard-coded model names are a time bomb
CrossChex, the booking verification tool Oliver and I built for TRIBE, had the model name claude-3-7-sonnet-20250219 written into the code in eleven places. We know because in April we had to find all eleven. Every one was a deploy waiting to happen, on the provider’s schedule rather than ours. They’re in one environment setting now, and the regression tests run the real model so we find out on our machines, not in production, when a swap changes the answers.
Obvious in hindsight. Most of the AI-built apps I see have the model name hard-coded in the same way, usually a model that’s already a generation old, and nobody will think about it until the morning it returns a 404.
It’s not only retirement
The subtler one is the way you call the model being retired while the model itself survives. Newer Claude models reject a request that sets a temperature, which every older model accepted, with a plain “temperature is deprecated for this model”. Gemini started ignoring sampling parameters and announced it would reject them. OpenAI’s reasoning models did the same. In each case our code was correct for the model we’d tested with, and wrong for the one we switched to, and the provider’s error message was the first anyone heard of it.
So in BrandScience, which is built around AI rather than merely using it, the provider adapters carry a list of which model families accept which parameters, and strip what a model won’t take. That list is maintained by hand, from release notes and from 400 errors. There isn’t a better way that I’ve found.
Primary and fallback, and what we got wrong
BrandScience makes fourteen different kinds of AI call, from analysing survey responses to writing the briefing a marketer reads on Monday morning. Every one has a primary provider and model, and a fallback provider and model, stored in the database so we can change them from an admin screen without a deploy. If the primary throws, the call goes to the fallback, the response is flagged as a fallback, and an event is logged so we can see it happened.
I’ll be straight about how crude that is. It’s one hop, not a chain. Any error triggers it, including a bad request that would fail on the fallback too. There’s no circuit breaker, so during an outage every call tries the dead primary first and waits for it to fail. And the prompt that was tuned for Claude goes to GPT unchanged, which usually works and occasionally reads like a different author.
For an outage of a few minutes or a few hours, crude is enough. The product kept answering. The part that wasn’t enough was quotas: a calibration run lost 322 of 456 samples in about two minutes when Google’s daily quota ran out and the “retry after” it sent back was fifteen hours. The fallback did its job for each call and the run still failed, because nothing stopped trying. That part of the system now classifies errors, honours the retry delay, and stops using a provider for the rest of the run when it’s out of credit. Honestly, I bet most AI-built apps don’t have a fallback at all, let alone one that knows the difference between “busy” and “gone”.
Drafts, versions, and a sandbox
The thing I’m most pleased with isn’t the fallback. It’s the versioning.
Every prompt in BrandScience, system and user, lives in the database, not the code. Each AI feature has one published version and at most one draft. Publishing is a transaction: the draft becomes live, the previous version is marked superseded, the caches clear, and the diff is logged. Rolling back doesn’t move a pointer, it publishes the old version again as a new one, so the history is honest. There’s a global snapshot across all fourteen features for the day a change to a shared fragment goes wrong.
Before publishing, you can run the sandbox: the published version and the draft, side by side, on the same frozen inputs for the same brand. Both outputs are stored with their tokens, cost and latency. The prompts are shown as a line diff. The outputs deliberately aren’t. A model’s output varies from run to run even with identical settings, even at temperature zero, so a diff of two outputs just paints everything red and green. You read them and decide.
Why does that matter? Because in July the briefing feature shipped a headline telling a customer they had a 43-point equity deficit when they were four points above benchmark. That’s not a hallucination problem, it’s an arithmetic problem, and at low temperature it’s near-deterministic: running it again gives the same wrong number. It blocked sales demos until it was fixed. The fix was a prompt change, the sandbox let us see it was fixed before anyone else did, and the model is now banned from doing arithmetic at all. The numbers are computed in code and the model narrates them.
If you’re the founder
None of this is exotic. It’s a config table, a try-catch, a draft flag and a comparison screen. But every piece of it exists because something broke first, and I’d rather you borrowed the list than the incidents:
- Put the model name in config, not code. Then search your codebase for the model name anyway, because it’s in more places than you think.
- Have a fallback, and log when it fires. One hop is fine. None is not.
- Treat “the way I call it” as deprecable, not just the model. Read the release notes.
- Keep prompts out of the code, version them, and compare before you publish. Don’t diff the outputs. Read them.
- Never let the model do the maths.
If your app calls a model and you’re not sure which of these it has, that’s one of the things we look at in a Production Readiness Review - US$1,950, fixed, within 48 hours.