The fungibility gradient

A year ago I argued for treating language models as commodities: put a gateway between your application and the providers, and switching models becomes a configuration change instead of a rewrite. That architecture advice aged well. Gateways went from a pattern to a product category, and today most serious deployments route through one.

But the argument succeeded a little too well, because a conclusion got smuggled in alongside it. If switching is a config change, models must be interchangeable. Procurement teams now treat model choice as a pricing negotiation. Strategy decks assume the application layer captures all the value because the model underneath can be swapped at will. All of it rests on one word: fungible.

What the gateway standardized is the transport layer: authentication, request shape, streaming, fallbacks. What it did not standardize is behavior. Those are different layers, and the distance between them is not fixed. It grows with the depth of the application.

What the prompts know

A prompt that has been in production for a year was not written in one sitting. It accumulated. Every “return only valid JSON,” every “do not explain your reasoning,” every carefully ordered block of instructions was added to fix something one specific model got wrong. The fixes that survived are the ones that worked on that model. Nobody wrote down which lines matter or why, because nobody knew at the time. The prompt slowly became a private dialect between one team and one model.

Hand that prompt to a different model and the dialect stops being understood. Research on cross-model prompt transfer keeps converging on the same result: a prompt engineered for one model often performs substantially worse on another, enough that researchers gave the failure a name, model drifting, and are building tools to work around it. The sensitivity runs deeper than content. Formatting alone, whitespace, delimiters, the order of instructions, can move benchmark accuracy by double digits. Provider conventions differ at the level of syntax: OpenAI’s models respond best to markdown-structured prompts, Anthropic’s to XML tags. None of this appears in any API contract. It is all unwritten, and a deep application depends on it in hundreds of places.

The same co-evolution happens one level up, in the harness. Tool definitions, retry logic, the decision about what goes in context and what gets summarized away: all of it gets tuned against the incumbent model’s failure modes. Benchmark researchers have stopped treating this as noise: an agent’s score is a property of the model embedded in a harness, not of the model alone. A harness that flatters one model can actively mislead another.

Depth is a multiplier

For a single-shot task, none of this is fatal. You swap the model, output quality dips, someone eyeballs twenty examples, you patch the prompt, done. The regression is visible at a glance because the whole task is one glance wide.

Agents take that away. A multi-step agent is a chain of dependent decisions, and chains turn small per-step differences into large end-to-end ones. The arithmetic is blunt: an agent that gets each step right 99% of the time completes a 100-step task about a third of the time. At 98% per step, about one in eight. One point of per-step difference, invisible in any demo and within noise on most benchmarks, is roughly a 3x difference in whether the work actually finishes.

So two models that look interchangeable on a leaderboard can be far apart at the end of a long chain, and the leaderboard will not tell you which way. It depends on which steps your workflows stress: the tool calls, the recovery from a malformed result, the fortieth decision, made with a context window full of the first thirty-nine. The only way to know is to run your own tasks, end to end, and grade the results.

The gateway made switching cheap. Evals make it safe. Most teams built the first and assumed the second.

The toll is paid in evals

Grading the results is the expensive part, and it is expensive in a specific way: it requires expertise, not just infrastructure.

A regression in a complex application is rarely a crash. It is a contract clause that reads as plausible but shifts liability, an analysis with the right shape and a wrong assumption, a migration plan that omits one dependency. Catching that requires evaluation sets built from your actual workflows, graded by people who can tell good from plausible. That is exactly the expensive judgment AI was supposed to save. For high-value domains there is no shortcut; the eval is only as good as the expert who defined it.

Most enterprises have not built this. Surveys of production AI teams keep finding that more than half still evaluate model output manually, which in practice means vibes: someone senior reads some outputs and nods. Vibes can catch a catastrophe. They cannot catch a model that got slightly worse at following instructions in long contexts while getting better at everything a demo shows.

The market behaves accordingly. Menlo’s enterprise survey found that even when switching costs are low, buyers mostly stay put and upgrade to their incumbent provider’s newest model. Watch what buyers do, not what the narrative says. Enterprises are not switching freely behind their gateways. They are behaving like the switch is risky, because for any application with depth, it is.

Here is the uncomfortable part: you cannot decline to pay. Providers deprecate models on their own schedule, so the migration you are avoiding is coming anyway, just not at a time of your choosing. And behavior moves even under a fixed name. The Stanford and Berkeley study of GPT-4 found that between March and June 2023, accuracy on one task fell from 84% to 51% with no version change at all. The evals are not the cost of switching models. They are the cost of operating on top of models, and switching is merely when the bill arrives.

A gradient, not a property

Fungibility, then, is not a property of models. It is a property of applications, and it runs on a gradient of depth.

At one end, classification, summarization, extraction, single-shot generation where “good enough” genuinely is good enough. There the commodity framing is simply true. Route to the cheapest thing that clears the bar, arbitrage prices, never look back. At the other end, multi-step agents doing high-value work, where the prompts and harness have co-evolved with one model and errors compound silently across the chain. There the model is a hard dependency, whatever the architecture diagram says.

Most strategic claims about AI right now quietly assume the whole gradient sits at the shallow end: the labs have no moat, the application layer captures everything, model choice is a procurement detail. The high-value applications sit at the other end, which is exactly where the assumption fails.

The practical reading is not to abandon the gateway. It is to notice what the gateway never claimed to give you. Mobility between models was always going to be bought with evals: task-level, expert-graded, run continuously rather than at migration time. Teams that have built that grading machine can treat models as commodities, benchmark a new one in a week, and negotiate with every provider knowing they can actually leave. Teams that haven’t are tied to their current model in ways nobody has written down, and no amount of routing infrastructure changes that.

Fungibility was never something you could buy at the infrastructure layer. It is something you earn at the evaluation layer, and so far, few have.