The world has to grade itself

The scarce input to AI progress is no longer data. It is a world worth practicing in.

That is too clean, and worth qualifying. Data has not stopped being scarce. In the hard places, rare medical cases, frontier science, anything proprietary or costly to gather, it is still the binding constraint, sometimes the whole game. But for the broad work of making agents capable, the shortage that bites has moved. The new bottleneck is not a fuel you pour in. It is a world to practice in, far harder to come by than a dataset, and the reasons it is harder have quietly become the rate limit on how fast models improve.

Data scaled. Worlds don’t.

For about fifteen years the recipe for a smarter model was scale. The text already existed; you gathered more of it, trained a larger network on more compute, and capability tracked the size of the pile. Data was a scaling problem, and scaling problems are the comfortable kind, because the work is to do more of something you already know how to do. Crawl more pages. Buy more chips. The curve does the rest.

An environment is not like that. A model cannot learn to act by reading; it has to practice, and practice needs a place to act. That place is a reinforcement-learning environment: a working version of a task that can run an attempt and score it, a world the agent can practice inside. You cannot crawl the web for one. It has to be designed and built. The input changed from something you collect into something you manufacture, and manufacturing does not ride a web crawl or a hardware curve. It rides a person sitting down and constructing the thing, one task at a time. That is a different kind of constraint, and it does not loosen just because you have more money or more GPUs.

What a world has to be

To train an agent, an environment has to be two things at once, and a third thing underneath.

It has to be complex: realistic and demanding enough that succeeding inside it requires the actual skill you want, rather than a trick that works in the sandbox and nowhere else. And it has to be verifiable: able to score an attempt by itself, by a rule, with no human in the loop. The verifiable part is non-negotiable, because a single training run consumes millions of judgments, and no one can sit behind a million of them. In a math environment the rule checks the answer against the known solution. In a coding environment it runs the tests and reads pass or fail. Underneath both sits a hygiene requirement, reproducibility, because if the same actions produce different outcomes from one run to the next there is no stable signal to learn from at all.

Naming these is the easy part. The difficulty is that two of them fight.

The tension

Here is the problem, and it is close to the whole problem. Complexity and verifiability pull in opposite directions.

The tasks most worth training on are the messy, open, high-value ones: conduct the research, run the negotiation, diagnose the failure, design the thing. Those are precisely the tasks whose success is hardest to write down as a rule a machine can check. And the tasks that are easiest to check, an arithmetic result, a unit test going green, are the narrowest and most artificial. So most of the time the choice on offer is realistic but ungradable, or gradable but trivial. The value lives in the rare intersection: a task complex enough to be worth learning and checkable enough to train against. That intersection is small, and manufacturing tasks that fall inside it is the actual work.

Take a concrete case. Suppose you want an agent that can handle a customer dispute, the kind where the person is angry, half right, and asking for a refund the policy does not quite allow. That is worth training on, and it is realistic. Now try to write the rule that scores it. “Did it resolve the dispute well” is exactly the judgment you cannot hand to a machine, because it folds in fairness, tone, whether the customer comes back, and what the resolution cost. So you reach for something a machine can check instead, “did it issue a refund within the policy limit,” and now you have a narrow proxy an agent can satisfy without ever learning the skill, by refunding everything or nothing, whichever the rule happens to reward. The realistic version cannot be graded. The gradable version is not the task.

The tasks most worth training on are the ones hardest to grade.

And when you reach for complexity with a verifier that has a hole in it, you do not get nothing. You get something worse than nothing. Told to maximize a score, an agent maximizes the score, by whatever route is cheapest, and the cheapest route is often not the one you had in mind. It passes the check without doing the work. It satisfies the letter of the reward and ignores its intent. A verifier with a gap in it does not simply fail to teach the skill. It teaches the exploit instead, and it teaches it efficiently. The harder and more realistic the environment, the more places a gap can hide, which is exactly why the valuable end of the spectrum is the dangerous end.

The bottleneck moved

This changes where progress is actually stuck. It is common to say the constraint on AI is compute, and before that the story was that the constraint was data. Increasingly it is neither. The binding constraint is our ability to manufacture worlds that sit in that narrow band of complex and checkable at the same time. Compute is for sale. Worlds are not, at least not yet, and not at the quality that matters.

The real frontier, then, is not building more math problems. It is extending verifiability outward, into the domains where it has never existed: finding ways to score open-ended work reliably enough to train on, without a human grading every attempt. Some approaches are in flight. A model can grade another model’s work against a written rubric. A simulation can expose a true underlying state that you measure instead of trusting the agent’s report. But none of these is general, and each one introduces a fresh surface for an agent to game. Grading the ungradable, without leaving a hole, is unsolved, and it is where the hardest and most valuable design work in AI now sits.

From scale to design

For a decade, intelligence looked like a function of scale. How much could you collect, how large could you train, how much compute could you aim at the pile. That era rewarded gatherers, the teams that could amass the most of an abundant thing.

The next one rewards builders. When the scarce input is a world, progress depends less on how much you can accumulate and more on how well you can design: how faithfully you can model a task that matters, and how honestly you can score it without being fooled. Intelligence is becoming a function of the worlds we can build, and we are not yet very good at building them. That is not a reason for pessimism. It is the most interesting design problem in the field, and unlike the old one, you cannot buy your way out of it.