Everything is an eval now
In machine learning, an eval is a careful thing. You take a model or an agent, run it against a fixed set of tasks with known answers, and score how well it did. It is how labs decide whether a new version is better than the old one. The word belongs to specialists, and it sounds like a lab coat.
That meaning is about to broaden into something everyone does all day. Because you already run evals, constantly, without calling them that. Every time an agent hands you an answer and you decide whether to use it, you are evaluating. You read the draft, the analysis, the code, the plan, and in a second or two you make a call: good enough, or not. The benchmark is gone, the lab coat is gone, but the act is the same. You are scoring the output and deciding what happens next.
As agents take over more of the producing, that act stops being a side task and becomes the main one. This is worth saying plainly, because it changes what skill matters.
The unit of work shrank
For most of the history of knowledge work, the scarce and valuable thing was producing the answer: writing the memo, building the model, drafting the contract, finding the bug. Evaluation came along for the ride, folded into the act of making, because the person who made the thing was checking it as they went.
When the answer becomes nearly free to generate, that bundle comes apart. Production is no longer the scarce contribution; the agent does it in seconds. What is left, the part that still needs you, is the evaluation. Your role shifts from author to editor, from the one who makes the thing to the one who decides whether the thing is any good. And it is not one decision; it is a steady stream of them, every answer, all day.
This is the same shift, at the scale of a single interaction, that is reshaping whole companies. AI made the work cheap and left the trust expensive, and the trust gets spent one answer at a time. It is also the mirror of how these agents are trained: a model gets better by acting in an environment that grades it, and now you are the environment, grading the agent so you can use what it produces. The grading used to happen in the lab, before the model reached you. Now a great deal of it happens at your desk, in real time, and the grader is you.
A discipline, not a vibe
If evaluation is the job, then “be skeptical of AI” is not advice, it is a shrug. The skill is more specific than that, and it has parts.
The first is calibrated skepticism. Blanket doubt is as useless as blanket trust, because re-checking everything destroys the speed that made the tool worth using. The move is to match scrutiny to stakes times checkability. An answer that is low-stakes and easy to verify, you accept and move on. An answer that is high-stakes and hard to verify is the one to slow down on, or not to delegate at all. Most of the skill is spending your attention where it actually buys something.
The second is the mental checksum. A checksum, in computing, is a cheap test that catches gross corruption: it does not prove a file is correct, it just flags when something is obviously wrong. Run the same kind of cheap test in your head on an agent’s output. Does the number have the right order of magnitude. Do the units make sense. Is the answer internally consistent. Does the shape of it look like the shape of a correct answer. These are quick, partial checks, run in seconds, and they catch a surprising fraction of confident nonsense before it costs you anything.
The third is triage: deciding, fast, what to do with what you got. There are really four moves. Accept it. Ask for a revision. Challenge it, push back and make it defend or correct itself. Or prune it, discard the whole attempt and move on, which is the most underrated of the four, because a wrong answer is often not worth fixing. Good evaluators route quickly between these and do not get stuck lovingly repairing output they should have thrown away.
You are no longer producing the answer. You are evaluating it, all day.
None of this is new, exactly. It is what a good editor does to a draft, what a senior engineer does in code review, what a manager does with a capable but fallible report. What is new is that nearly everyone is about to do it, all day, as the core of the work rather than the finish on it.
The catch
There is a hole in all of this, and it is worth being honest about, because it is where the real danger lives.
You can only checksum what you understand. The plausibility check, the unit check, the does-this-look-right check, all of them depend on already having a sense of what right looks like in the domain. Hand an expert a flawed analysis in their field and they feel the wrongness before they can articulate it. Hand the same analysis to someone outside the field and it reads as authoritative, because they have no checksum to run. The skill quietly assumes a competence, and where the competence is missing, evaluation silently fails.
That is the uncomfortable part of “everything is an eval.” It is most reassuring exactly where it is least reliable. AI is at its most dangerous not when it is wrong, but when it is wrong in a domain where you cannot tell, and it delivers the error in the same fluent, confident voice it uses for the truth. So part of the skill, maybe the hardest part, is knowing the edges of your own competence: noticing when you are not actually qualified to judge what you are looking at, and then either getting a real check from someone who is, or declining to rely on the answer at all. An evaluator who does not know what they cannot evaluate is not evaluating. They are just nodding.
The literacy
Prompting was the skill everyone rushed to learn in the first couple of years, and it is already fading, because the models keep getting better at inferring what you meant from less. Evaluation is the one that lasts. The person who gets the most out of these tools is not the one with the cleverest prompts. It is the one who evaluates best: who can run a fast mental checksum, route an answer in seconds, calibrate doubt to what is actually at stake, and stay honest about the places they cannot check.
We are all becoming evaluators. The work is learning to do it well.