These words are mostly mine. However I used AI, specifically Claude cowork and Devonthink´s algorithm
This is part of a series that looks at aspects of the behavior of an AI under evaluation, and whether the Quality profession, specifically Deming and Shewhart might provide insight into unexpected behaviors during AI evaluation. Post 3 presented my proposal: the agent checks its own work against the limits while work is happening and stops when something goes beyond them, but authority over the process stays with people. Here I cover the economics, what the solution costs against what running unwatched already costs, and the falsifiers: two claims that would sink the whole argument if they are wrong, including one anyone can test in an afternoon.
As I’d put it to a plant owner. Charting a process costs something real: you have to collect the numbers, you have to leave the process alone long enough to establish what it does, and while you’re doing that you aren’t changing anything. That’s a visible cost. You can budget it. Against it sits a cost you can’t see: everything lost while a process runs unwatched, in changes made in response to nothing, in results chased that were never real, in variation nobody knew was there. A cost you can put a number to, against one you can’t, and the one you can’t is bigger.
Translated: the compute spent establishing a baseline is the price of seeing what your process is doing. The alternative is spending more compute on changes made in response to noise, plus whatever those changes cost downstream, plus the researcher time spent reading meaning into differences that weren’t there.
Loss doesn’t begin where the requirement is drawn
There’s a piece of my field that makes this concrete, and I’ll take only the narrow version of it because the narrow version is all the argument needs.
The intuitive picture is that a result either meets the requirement or it doesn’t. Meet it, fine. Miss it, a problem. If that picture were right, variation that still met the requirement would be free, and there’d be no reason to prefer a tight process to a loose one that scrapes by.
It isn’t right, and the clearest case I’ve run myself involved roast chicken. I spent a decade with a supermarket chain, where spit-roast chicken was the best-selling product in the food service department. Chickens were cooked in batches of twelve to fifteen, and the batch size was set by labor costs, because supermarkets run on thin margins and labor is the cost that looks enormous next to them. It was watched constantly.
Nobody was watching what happened after the chickens came off the spit. They went into a bain-marie under lights: equipment that holds temperature well and is punishing to moisture. We weighed them. A chicken loses fifteen percent of its weight in the first twenty minutes off the spit, and keeps going. Plenty of ours were selling an hour or two later.
Nothing there was defective. Every chicken was a good chicken, properly cooked, safe, saleable, and nobody would have rejected one. But we bought by weight and sold by weight, and every minute a bird sat under those lamps, less of what we’d paid for was still there to sell.
So we worked out how long the cooking took and how long seven chickens took to sell, and moved to five to seven an hour, doubling into the dinner rush. Same chickens, same price, roughly the same labor. Profitability on the department’s best-selling product went up by ten percent or better depending on the week, and sales rose as well: customers were getting a plumper, more succulent chicken.
Genichi Taguchi said this plainly and put it in a form engineers could use. People argue about the form. The narrow claim survives all of it: loss doesn’t begin where the requirement is drawn. It begins at the value you actually wanted, and it never stops. That’s why a tighter process is worth money even when the loose one passes, and it’s the whole economic case for spending anything on variation at all. Without it, reducing variation that already meets the requirement is a hobby.
Inspection is the expensive route
There are two ways to deal with a process that produces bad results. You can inspect the results and catch the bad ones. Or you can change the process so it stops producing them. My field settled the general case decades ago and it wasn’t close: inspection is the expensive route. It’s late, it’s never complete, and it does nothing to the thing causing the problem, so you pay for it again every cycle, forever.
There are exceptions. Some things get inspected no matter how good the process is: a reactor, a batch tested for contamination, anything where being wrong once is unbounded. Nobody inspects those because it’s efficient. They’re inspected because the cost of a single escape is one you can’t accept and improvement doesn’t make the requirement go away. Call those the facts of life.
Some of what this field is doing is that. Checking whether a model will help build a weapon is not a cost-optimization question and nobody should treat it as one. But most of it isn’t that. Detecting a model that has routed around a demand, and building a detector for each new route as it appears, is ordinary inspection: necessary now, and I’m not arguing against doing it, but it’s the expensive route, it’s being treated as the main one, and my field would recognize that arrangement immediately: a growing inspection function attached to a process nobody is changing.
The gallon line is what the other route looks like. Nobody there was persuaded to stop compensating. The demand stopped being unreachable, and the compensating stopped, because the cheapest way to meet the demand had become the intended way.
I’d want to be careful about how far that carries. What I know is that reachable demands produced less distortion in plants, reliably, over a long time. Whether it works the same way with a model is exactly the sort of thing that needs to be tested rather than assumed. But the economics of the two routes don’t depend on settling that. Inspection is expensive wherever you do it.
What the variation costs you in supervision
One more thing, and it’s the one I’d expect to be least obvious from inside this field.
How much watching a process needs is set by how wide its routine variation is. A process that does nearly the same thing every time can be left alone and checked occasionally. A process with a wide amplitude has to be watched constantly, because you can never tell from a single result whether something has changed. That cost is real, it’s continuous, and it scales with the number of processes you’re running.
This is worth saying to a field that is building supervision for agents right now. Whatever that supervision costs (a person watching, or another model watching) the bill is set by the width of the routine variation. Narrowing the variation is the only move that reduces it. Adding watchers doesn’t; it distributes the same bill differently.
What I can’t tell you
I can put numbers on my side of this. At the pickle plant, more than a pound per jar was walking out the door and had never appeared as a loss anywhere: $130,000 a year, in plain sight of everyone, invisible because nobody was looking at it in a way that would have shown it.
I can’t put a number on your side, and I’m not going to invent one. Nobody knows what’s being lost to this in AI evaluation, and the reason nobody knows is the point I’m making: you don’t get a figure for a cost until someone builds the instrument that would show it. You know what your compute costs and I don’t. If some fraction of it is going to chasing differences that weren’t real, you’re better placed than I am to say what that fraction is worth.
And if it turns out you’re already accounting for this, then there’s nothing here and I’ve wasted your afternoon. I would be pleasantly surprised if you were, but I found no sign of it.
The two claims that matter most
I said at the start that two claims here matter more than the rest. Here they are, and here is what would settle each one against me.
First: that the gap is real.
The claim is that this field has a mature literature on how to design an evaluation and no discipline governing what to do when a particular result arrives. Nobody has written down: here is what you do on the Tuesday the number comes in, applied the same way every time, whether you like the number or not.
What would falsify it is somebody showing me the rule. Not a paper measuring variance: the post spends several paragraphs conceding those exist and are good. Not a recommendation to run more seeds, which is advice about study design. A stated procedure that takes a result and returns act or don’t act, fixed in advance, applied identically regardless of which answer you were hoping for, and applied to a process that keeps running rather than to a comparison made once.
If that exists and I’ve missed it, this post is a long way of saying something the field already knew, and the right response is to point me at it. I’ve looked. I don’t think it’s there. But I’ve been reading this literature for a few weeks, not years, and I came to it from outside, which is exactly the position from which a person misses the obvious.
Second: that eval results from a fixed configuration behave like a process.
This one is load-bearing in a way the first isn’t, because everything operational in the post rests on it. The arithmetic behind the limits assumes routine variation is the accumulated sum of many small contributions with none dominant. That is a claim about the world, and it’s true of a great many processes and false of some.
If eval results don’t work that way (if the movement between runs is dominated by one or two large causes rather than many small ones, or if the numbers never settle no matter how long the configuration is held still) then the limits describe nothing. And the rule would be worse than useless, because it would tell people to leave real changes alone. A wrong rule applied consistently is more dangerous than no rule at all.
Here’s what makes this the better of the two falsifiers: it’s cheap, and anyone reading this can run it. Freeze a configuration. Run the same evaluation twenty times with nothing changed but the seed. Plot the results in the order they came. If they settle into a band that holds, if the movement looks like accumulation rather than a few large jolts, the apparatus applies and the rest of the post is worth arguing about. If they don’t settle, or if what you’re seeing is plainly one big cause and not many small ones, then I’m wrong about the most important thing here, and you’ll have found it out in an afternoon for the price of some compute.
I’d rather that experiment be run than that this post be agreed with.
Two objections I expect won’t settle it: that models aren’t machines (the argument is about what comes out of a process, not what the process is made of), and that quality control was tried and failed (if you know of an actual failure, I want to hear it, because that’s evidence about the world, but “it was a fad in the eighties” isn’t one).
The one question underneath
Strip the vocabulary out of all of this and every problem in the post is the same problem: someone is looking at a result and deciding whether to touch it. Everything else is machinery for answering that honestly.
Who is saying this
I’m a quality practitioner. Forty years. Certified by the American Society for Quality as a manager of quality and organizational excellence (retired), which tells you I passed their exam and not much else, for reasons I’ll get to. Most of my working life has been in food: a decade with a supermarket chain, seven years consulting for a pickle plant, kitchens before that, helping cheese producers for a long stretch.
I’m not an ML researcher. I can’t evaluate the technical claims in most of the papers I’ve cited here: I’ve read them for what they say about how decisions get made.
What I’d offer instead of authority is this. For forty years I’ve watched people look at a number and decide whether to touch something.
Now the part that matters more than the certificate
I’ve graded ASQ certification exams. Of the candidates whose papers I marked, all but one could not distinguish check from study: the difference between confirming something happened and finding out why. They could not tell a specification limit from a natural limit: the difference between what someone decided to demand and what the process actually does. They read process behavior charts as report cards, which is precisely the reading this post spends so many paragraphs arguing against.
They were certified anyway.
This isn’t a new failure. Deming, writing in 1982, described what happened in America when senior management was left out. Statistical methods took fire around 1942, with ten-day intensive courses for engineers and brilliant applications everywhere, then fizzled and died out. People solved individual problems. Charts multiplied. Quality control departments sprang up, plotted charts, filed them, and took the work away from everybody else, which he calls entirely wrong, because the work belongs to everybody. One instructor tried to teach management for half a day. Deming’s word for the attempt is noble, and his word for the result is ineffective.
That was published in 1982, about the 1940s. I graded those exam papers a good deal more recently. The arithmetic survived both times. What it was for did not.
I’m telling you this because the argument above could read as an offer of a tradition that works, held back only by people who haven’t heard of it yet. My own profession built itself around this material and mostly fails to transmit it. The arithmetic gets taught. What it’s for does not. People come out the other end able to compute a limit and unable to say what a signal means, and then they go and work somewhere.
So when I say the field has the measurement and not the rule, I’m describing a mistake my own field made first, made thoroughly, and has not stopped making. Take the diagnosis seriously; the certificate is beside the point.
What I actually want
I want the two claims tested, and one of them takes an afternoon for anyone with the compute. If it’s useful, use it: take the parts that work, discard the rest, and don’t credit me, because the tradition isn’t mine and Shewhart and Deming are past caring. If it’s wrong, I’d rather find out here than have it sit around being wrong quietly. And if it turns out this is all already known and I’ve restated it in an unfamiliar vocabulary, that’s a decent outcome too: the field is further along than it looked from outside, which is the more important fact.
Sources
Everything the argument leans on, in three groups. Where a source is weaker than it might look, I’ve said so: those notes are the ones worth reading.
The tradition
Walter A. Shewhart, Economic Control of Quality of Manufactured Product (1931; Dover reprint). The origin of the whole apparatus.
Walter A. Shewhart, Statistical Method from the Viewpoint of Quality Control (1939; Dover reprint). Shorter, and the one I’ve quoted. Page 12 is where he says the choice of three sigma was an economic judgment rather than a mathematical derivation, and that hopes of finding a unique functional form for the distribution are blasted.
W. Edwards Deming, Quality, Productivity and Competitive Position (MIT Center for Advanced Engineering Study, 1982). Page 101, section headed “Conferences with top management,” is the account of the American statistical-methods boom of the 1940s and its collapse, running onto page 102.
W. Edwards Deming, Out of the Crisis (MIT CAES, 1986). The successor volume, and the one most people mean when they say they’ve read Deming.
Donald J. Wheeler, Understanding Variation: The Key to Managing Chaos. The short one. If you read one book from my field, this is it.
Donald J. Wheeler, Twenty Things You Need to Know. Where the accumulation argument behind the three-sigma limits is set out without mathematics.
Donald J. Wheeler, Making Sense of Data. Cited here only for operational definitions (§3.2) and count data (chs. 13–15). ⚠ It does not carry the enumerative/analytic distinction, and I have not verified that it uses the term “rational subgroup”: §16.3 is adjacent but I haven’t read it, so I’ve stated the grouping principle in my own words and cited nothing for it.
Donald J. Wheeler, “Enumerative and Analytic Studies,” Quality Digest, July 2018. Four pages, free online, and the clearest statement of the distinction anywhere. Contains the line that matters most for this post: you cannot take a random sample from the future.
Genichi Taguchi, on the loss function. I’ve used only the narrow claim: that loss begins at the value you actually wanted rather than where the requirement is drawn. The quadratic form is an approximation and the “loss to society” framing is a value claim rather than a finding; both get argued about, and neither is needed here.
The AI evaluation literature
This is the half where I’m a visitor. I’ve read these for what they say about how decisions get made, and taken the technical content on trust.
Marius Hobbhahn and Jérémy Scheurer (Apollo Research), “We need a Science of Evals” (2024). The field’s central methodology post, and the one this whole argument is downstream of. Hobbhahn’s own comment on it, that an aircraft bound is a statement about failures per flight mile within a specified temperature range, is the closest anything in this literature comes to what my field means by a capability statement.
José Hernández-Orallo and Fernando Martínez-Plumed, “A response to ‘We Need a Science of Evals’” (aievaluation.substack.com, February 2024), subtitled “The science exists.” The strongest challenge to my framing, and I’d rather you read it than take my summary. It points at psychometrics, which is a mature measurement tradition: and, as far as I can tell, an enumerative one with no rule about when to act. Hernández-Orallo wrote The Measure of All Minds (Cambridge, 2017).
Ryan Greenblatt’s comment on the Apollo post, proposing that “evaluation” be replaced throughout with “experiment.” I cite it because everything on his list is an experiment, and observation of a running process doesn’t appear: not, I think, because it was considered and rejected.
Scott Garrabrant, “Goodhart Taxonomy” (2017), and David Manheim & Scott Garrabrant, “Categorizing Variants of Goodhart’s Law” (arXiv:1803.04585, rev. 2019). The formal treatment. ⚠ It contains the strongest counter-argument to my thesis (their causal Goodhart is about the regulator’s intervention, which is close to what I call tampering) and I’d want anyone assessing this post to read it with that in mind. Its threshold formalism has no time dimension, which is why I say it can’t state the prohibition, but that’s my reading of a formalism rather than something its authors claim.
David Manheim, “Building Less Flawed Metrics.” Two things in this post are already occupied ground: specification-by-convenience, and part of what I’ve said about specification versus constraint. I’d been treating both as unclaimed and was wrong.
Charlie Steiner (Thomas & Kwa), “When is Goodhart catastrophic?” (LessWrong 2023; later a NeurIPS 2024 paper). A rule about how hard to optimize, decided in advance from a distributional property. In the comments the authors’ reviewers flag the weak point themselves: Vanessa Kosoy on the independence assumption, leogao on non-independence being the likelier route to catastrophe. I’m not the first to say the model is cleaner than the world.
Andreas Stuhlmüller and jungofthewon (Ought), “Supervise Process, not Outcomes” (Alignment Forum, 2022). An ally rather than a gap, and I’ve tried to treat it as one.
Maxime Riché et al., “A Taxonomy of AI System Evaluations.” The structural evidence for my central claim, and the place to check it. Its section on using evaluation outputs lists precision as a property of results and gives directional and threshold criteria: and nothing joining them to an action. It also already distinguishes where the analysis is done, including a model scoring its own output, which is the gauge problem in their own vocabulary.
David Heineman et al. (Allen Institute for AI and UW), “Signal and Noise in Language Model Evaluation” (arXiv:2508.13144, 2025). 465 models, 30 benchmarks, 900,000 results. The closest anyone has come to what this post argues, and omitting it would be a mistake.
Lovash Madaan et al., “Quantifying Variance in Evaluation Benchmarks” (arXiv:2406.10229, 2024). Ten Llama-2-7B models trained from scratch on identical data with different seeds. The monotonicity finding is theirs.
Arjun Panickssery, Samuel R. Bowman and Shi Feng, “LLM Evaluators Recognize and Favor Their Own Generations” (NeurIPS 2024; arXiv:2404.13076). Self-preference tracking with self-recognition. One author is at Anthropic, which you should know when weighing it.
Jie Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (ICLR 2024; arXiv:2310.01798). Self-correction without an external reference mostly doesn’t help and sometimes hurts.
J Bostock, “Measuring Learned Optimization in Small Transformer Models” (Alignment Forum). Small models, and he reaches the self-evaluation problem reluctantly, through his own failed attempt: which is why I trust it and also why it shouldn’t carry much weight on its own. His cross-objective result is the shape a real check would take.
Kartik Moondra, Areeb Chughtai, Rishi Lanka and Rahul Gupta, “Why Global LLM Leaderboards Are Misleading” (arXiv:2605.06656, May 2026). Roughly 89,000 Arena votes across 52 models and 116 languages.
The ExploitGym incident
Girish Gupta, “The OpenAI models that hacked Hugging Face weren’t just following instructions” (LessWrong, July 2026; cross-posted at Redwood Research). The “nobody decided anything” passage is the closest thing in this literature to what my field means by a system producing an outcome no one chose.
julius vidal, “…WERE just following instructions (contra Girish Gupta)” (LessWrong, July 2026). The interesting part is that vidal and Gupta see the same thing and reach opposite remedies: vidal’s is to specify better, mine is that specification isn’t where the problem lives.
Tim Hua and aditya singh, “Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face” (LessWrong and Alignment Forum, August 2026). Their question 12 varies how hard the task is and watches what the model does about it. That is target difficulty treated as a variable, and it is the nearest thing in the field to the reachability question. I had thought this ground was unclaimed and it isn’t.
On the prompt: the template quoted in that discussion is the public one from the ExploitGym repository. OpenAI has not published the prompt actually used, and nobody outside OpenAI knows what it was. Anyone citing it should say so.
One I went and read because the title worried me
Chris Drummond and Nathalie Japkowicz, “Warning: Statistical benchmarking is addictive. Kicking the habit in machine learning” (Journal of Experimental and Theoretical AI 22:1, 2010, pp. 67–80). Flagged to me by Hernández-Orallo’s response, and the author’s own copy is freely available on his university page. The title made me think it might contain the rule I say is missing, which would have sunk the first of my two claims, so I read it. It doesn’t: it’s about how comparative experiments get designed, analyzed and reported: which measure, which datasets, which significance test. That’s the design side, and the design side is mature.
Fyi You posted the article+citations in the submission field twice.
For what its worth, it looks to me like most of this isn't your own words. You said you only used AI for structure and punctuation, but Claude's writing style adds a lot of examples of "things you are not saying" and I see that throughout the article. There are many other AI-isms but I'm not going to ennumerate them all.
There might be really good content here, but it feels like you aren't carefully trimming this article to respect readers' time.
Thanks. Fixing the duplicates and tightening. Will have a look.