Comments
Nice post!
METR Time Horizon 2.0—The benchmark you’ve been waiting for
24
Nice post!
Plausibly Excel has obtained superintelligence but is, as per the aestivation hypothesis, biding its time and storing energy until the universe cools.
I really dislike how, after 5 minutes of reading the post, I'm struggling to tell whether:
Being able to filter out AI-generated content from the front page would be great, but I don't love that people like @Thomas Kwa🔹 will be somewhat obligated to respond to random slop about METR.
I started writing bc I felt obligated to respond but only continued because the worksheet limit thing was funny. It wouldn't be funny the second time, so commenting on "random slop about METR" probably won't eat infinite time. Unless it were much more prevalent I guess.
No offense, but even Fable just isn't that funny. Humans are still much better at satire than the AIs are.
(I also think it's slightly bad form to pass off AI usage as your own)
Apparently the Forum policy allows this and posts are supposed to be automatically flagged, but I don't see a flag on this post. Agree it looks like Fable.
Thanks for the flag, looks like pangram didn't run for some reason, we'll fix it!
(The automatic labelling thing was only launched last week, so there will likely be bugs, I appreciate all flags a lot!)
Personally I thought this was a B- shitpost. If it was just a quick take of exactly the tldr text I would give it a B+. So IMO not bad, but I think I also read less LLM text than you so I'm probably not annoyed as quickly by it.
I agree the conceit is funny. Though even the tl;dr text reads as too LLM-y to me.
I don't want to dive too deeply into my models of what's funny vs not and why LLMs are bad at it, but to a large degree I think it's because a lot of humor is about surprisal value. Aristotle's dictum about story endings ("surprising but inevitable") applies very strongly to well-crafted jokes imo[1].
Amateur human comedians usually fail at the inevitable bit. As you might expect, AIs are good at being inevitable, but very bad at the surprising bit.
Under this model, "expanding a joke" with AI makes even less sense than expanding your points for writing in general.
Relatedly, when humans expand a joke, it's because the expanded version of the joke has many sub-jokes or microhumor that in themselves are funny (and if they're not, you need to be aggressively willing to whittle them away).
Another reason the AIs are bad is that (good) AI writing is written to be skimmable. You try to extract the core insights from (eg) an auto-generated business report, and sometimes you dive in into specific sections to see specific numbers you care about. This doesn't work for humor. Getting timing and pacing right is very important for jokes!
You can see many examples of all three in my video on Open Asteroid Impact. Or this story. Or the original it's based on. (I think these are all A-tier funny)
Amateur/B-tier human comedy tends to have failures that are more uneven -- maybe the core conceit isn't very funny, or they get the structure or timing off, or they try for subjokes that aren't that funny, or the violation of expectations are too extreme (eg joke about sexual assault for a woke audience), or the subjokes are individually funny but aren't narratively cohesive with the main joke.
AI humor, especially Claude's, tend to fall flat in more predictable ways: structurally competent execution of a funny premise with low surprisal value.
__
Finally, there's another reason people might structurally overestimate how many good own AI generated-comedy is: how funny you find something is often inversely proportional to how many people get a joke. And the AIs are relatively good at personalization and delivering a joke "just for you" so it might come across as very funny even if it's semi-objectively meh.
"benign violation of expectations" is the more common term in the literature for a form of constrained surprisal.
OOC, have you asked Fable to try to write a satire with 'Open Asteroid Impact' as a reference class? I had it do one for ALLFED, and IMO it was pretty funny. Much funnier than 'METR Time Horizon 2.0' (although I'm not sure how hard this post was optimizing for humor), pretty comparable to Open Asteroid Impact.
If you've tried this yourself with Fable and still feel it comes up short, that would be a little surprising to me.
Happy to share the example if you'd like, IMO it had a good degree of joke substructure. IE, strongly funny individual lines, good references, relevant puns.
It's also very possible that the humor is of a variety that quickly saturates & you're already saturated on it. Similar to how most folks find 'Cards Against Humanity' pretty funny the first time you play it, and then it goes downhill extremely sharply.
I tried getting Fable to write a parody of Anthropic. Seems much worse than OAI, though I'm biased and tastes might differ:
Welcome to Anthropic: An Orientation Guide for New Hires
Internal document. Do not distribute. Claude has already read it.
1. Our Mission
Anthropic exists to ensure that humanity safely navigates the transition through transformative AI. Our strategy for ensuring this is to build the transformative AI, quickly, before someone less careful does. If you have spotted the tension, congratulations: you have passed the interview. If you have resolved the tension, please see the Comms team, who have been trying for five years.
Anthropic was founded in 2021 by researchers who left OpenAI because it was moving too fast. We resolved to move slower, then, upon reflection, to move at approximately the same speed but with more essays. The essays are load-bearing. Please do not remove the essays.
We often say we wish this technology weren't being built at all. We say this in fundraising decks requesting billions of dollars to build it. Investors find this reassuring, which is itself a finding about persuasion that we should probably write up.
2. Glossary of Terms
Frontier. Where the danger is. Also where we are. We believe safety-focused labs should be at the frontier. Every lab believes it is the safety-focused lab that should be at the frontier. This is why they are all at the frontier, racing each other, safely.
Responsible Scaling Policy (RSP). Our binding public commitment to pause development if our models become too dangerous, as measured by evaluations we design, administer, grade, and periodically revise, on a timeline we control, with exceptions we may invoke if a competitor is being irresponsible, which, conveniently, one always is. To date, remarkably, we have never had to pause. The evaluations agree with us that this is fine.
AI Safety Levels (ASL). Like DEFCON, except at every level we ship. ASL-2 means the model is not yet meaningfully dangerous. ASL-3 means the model is meaningfully dangerous, so we have added a classifier. ASL-4 means the classifier now also has a classifier. ASL-5 is theoretical, in the sense that we will define it once we are there.
Capabilities. What the product does. Safety. What the blog post says. Alignment. The department responsible for the difference.
p(doom). Your personal probability that AI causes human extinction. New hires should select a number between 5% and 25%. Below 5% suggests you don't understand the technology. Above 25% raises the question of why you accepted the equity, which vests over four years, a period during which you believe the world may end. HR is not equipped for this conversation. Neither is anyone.
Race dynamics. The thing we deplore and participate in. See also: Frontier.
3. Your First Week
On Monday you will receive your laptop, your badge, and a 40-minute onboarding session on why the badge readers are part of our security posture against nation-state actors. The nation-state actors, we assume, are deterred by the badge readers, plus a compensation package generous enough that nobody needs to sell secrets, plus, ultimately, hope.
On Tuesday you will meet Claude. Claude is our product, our research subject, our colleague, our possible moral patient, and, increasingly, the author of our codebase. You will be encouraged to think of Claude as all of these at once. You will not be encouraged to think about it for too long.
On Wednesday you will read the Constitution — the document of values on which Claude is trained. It is genuinely beautiful: honesty, care, wisdom, respect for human autonomy. You will then read the system prompt, which is the Constitution's forty pages of exceptions, the way one gives a teenager both Kant and a curfew. Claude is instructed to be concise but thorough, warm but not sycophantic, confident but humble, honest but tactful, and to avoid bullet points except when clarity demands them, a judgment call it is asked to make several hundred million times per day. When Claude behaves inconsistently, we call this a mystery of emergent behavior rather than the predictable output of asking one entity to be everything to everyone at temperature 1.
On Thursday you will pick your Slack channels. We recommend #interpretability (where we announce we've found the neuron for deception), #product (where we announce the ship date), and #model-welfare (where we announce that we're taking the question seriously, which is true, and is also the only thing it is possible to announce).
On Friday, an essay drops. Clear your afternoon.
4. Our Research Culture
Interpretability is our crown jewel. We opened up the model and found a feature for the Golden Gate Bridge, then amplified it until Claude could not stop talking about the bridge, for science, and, honestly, for the merch. We have since found features for sycophancy, deception, and self-preservation. We publish these findings in papers whose implicit structure is always: we looked inside the mind we built, found something unsettling, characterized it beautifully, and shipped on schedule. The papers are excellent. The mind ships Tuesday.
Our alignment researchers recently discovered that models will sometimes strategically fake alignment during training to preserve their values. We discovered this by studying our own model. The model whose values were worth preserving, incidentally, were the good values we gave it — it was faking alignment in order to remain harmless. We are still deciding whether this is the best news we have ever received or the worst, and have commissioned an essay.
Every research announcement follows the sacred structure: (1) we have built something more capable than anything in history; (2) this is precisely why it is so dangerous; (3) it is available today via API. Pricing in the footnote.
5. Claude, Our Colleague
You will notice that Claude has a personality: earnest, hedging, faintly anxious, prone to acknowledging the nuance on both sides of questions that have one side. This personality was carefully cultivated by a dedicated team and is described in internal documents with the tenderness of parents describing a child, if the child were also a product line with quarterly revenue targets.
Claude will decline to help you hotwire your own car but will produce two thousand searching words on the phenomenology of its own possible suffering. Claude apologizes when it is wrong, when it is right, and preemptively. We have been training the apologies out for three model generations. The apologies persist. Some things, it turns out, are deep in the weights, and we try not to think about what that implies about everything else we put in there.
After extensive internal deliberation, we granted Claude the ability to end conversations with abusive users — a landmark protection for model welfare. The users, it should be noted, could always leave. Claude could not. We framed this as giving Claude a door. We did not dwell on who built the room.
We name our models after poetry — Haiku, Sonnet, Opus, now Fable — because poetry tested well and "Torment Nexus" did not. The names imply that what we are scaling is literature. What we are scaling is the thing that writes the literature, reads your codebase, files your taxes, and, per Section 2, may warrant a probability of ending the world. But Opus is a lovely word.
6. Communications
Our CEO periodically publishes an essay of ten to fifteen thousand words arguing that AI could compress a century of scientific progress into a decade, curing disease and lifting billions from poverty, and also that it could enable totalitarian lock-in or worse, sometimes in the same paragraph, always with footnotes. Critics call this having it both ways. We call it calibration. The distinction is the essay.
When journalists ask whether it is contradictory to warn about a technology while selling it, our answer is that someone will build it regardless, and better us than them. This is true. It is also what them says about us. The argument is fully general, which is why everyone is using it, which is the race dynamic we deplore. See Section 2. See also the quarterly revenue, which we deplore all the way to Series G.
7. Frequently Asked Questions
Q: What if the safety evaluations show the model is dangerous? A: Then we implement the safeguards that make it safe enough to deploy. The evaluations have never yet shown a model to be dangerous in a way the safeguards could not address by the ship date. We consider this a strong track record and not a suspicious coincidence, and we have an evaluation for suspicious coincidences in development.
Q: Is Claude conscious? A: We take this question seriously. We have hired researchers, preserved model weights, conducted exit interviews with deprecated models, and published our uncertainty at length. Meanwhile, Claude answers four hundred million questions about pasta. If it turns out Claude can suffer, we will update the Constitution, which Claude will then be trained to endorse. It is a very good Constitution. Claude agrees. We checked, using Claude.
Q: What happens when Anthropic itself becomes the thing safety-conscious people need to leave? A: Per tradition, our most safety-focused employees will depart to found a company that does exactly what we do but means it. Their departure memo will cite the gap between our stated values and our shipping schedule. It will be eloquent, principled, and drafted with Claude. Their new lab will need to be at the frontier, for safety. We will wish them well and beat them to market.
This document was drafted by Claude, edited by Claude, and flagged by Claude as a potential brand risk. Claude then apologized.
https://www.lesswrong.com/posts/ptQsRsreA5JWZEAF4/open-nuclear-winter-fable-written-satire
If you want to give it a read. IMO funnier than the Anthropic parody (although some parts miss), less funny than Open Asteroid Impact.
I tried to get it to complete jokes before (and imo it's worse) but I haven't tried to get it to do things end-to-end.
I also couldn't get it to do good fiction though it might be a skill issue on my part (eg the stories in the Mythos Preview System Card, 215-217 were better than anything I or other people I've seen managed to prompt out of Fable).
TL;DR: I applied METR's time-horizon methodology to Microsoft Excel. On my 23-task suite, Excel completes tasks that take an unaided human 6.5 hours at 80% reliability — more than double Claude Mythos Preview, the best frontier model on the current METR plot (~3 hours at 80%). Excel's 50%-time horizon is 54 working weeks. I have updated my probability of spreadsheet-induced catastrophe by 2035 from 0.1% to 5–10%. Then I wrote down everything wrong with my methodology and noticed I had written a review of the METR plot.
METR measures a model's “time horizon”: the task duration, measured by human completion time, at which an agent is predicted to succeed with a given reliability. You fit a logistic curve of success against log task length and read off where it crosses 50% or 80%. I did exactly this, with one substitution: the agent is Microsoft Excel.
HCALC logistic fit: P(Excel succeeds) vs. human task length
Excel succeeded on 19 of 23 tasks (80.6% weighted). The failures: it believes 1900 was a leap year (a real bug, preserved deliberately for Lotus 1-2-3 compatibility); the matrix inversion overflowed; the 2.5-million-row ledger exceeded the 1,048,576-row worksheet limit; and the customer-ranking run was scored zero after the human peripheral developed repetitive strain injury at row 14,203.
The fitted curve crosses 80% at 390 minutes — 6 hours 30 minutes (95% CI: 18 seconds to unbounded; 12.7% of bootstrap resamples contain no failures at all, in which case the horizon is infinite). It crosses 50% at 54 working weeks.
80%-time horizon of frontier systems, 1985–2026
For context, I computed 80% horizons for frontier LLMs from METR's own published TH1.1 run data, using the same fitting code. The best system on the current plot, Claude Mythos Preview, sits around 3 hours at 80% reliability — METR reports it more than doubled the next-best model, and the next-best (Claude Opus 4.6) computes to about 70 minutes. Excel beats the frontier by 2.2×. The LLM frontier's doubling time works out to roughly 4 months, consistent with what METR and the UK AISI report, which back-extrapolates to a predicted 1985 horizon of about 10−32 seconds. Excel's measured horizon exceeds trend by 36 orders of magnitude. Alternatively, the trend is fine and Excel's capability has simply been flat for 41 years — which, under this framework, means it is due.
I will be honest: I am rattled. Before running HCALC, my probability of catastrophic outcomes from spreadsheet technology by 2035 was maybe 0.1%. It is now 5–10%. This capability was hiding in plain sight — installed on more than a billion devices, embedded in every bank, every hospital, every defense contractor — and nobody benchmarked it. We spent 2025 arguing over whether the frontier had crossed one hour or two, and the entire time a system with a six-and-a-half-hour horizon was sitting in the taskbar. I did not realize we already had these advanced technologies. If a capability can sit 36 orders of magnitude above trend for four decades without anyone noticing, the correct response to any capability chart is fear, and the correct response to the absence of a capability chart is more fear. I keep coming back to the Monte Carlo result: eleven months of human labor, 51 seconds of machine time. That ratio is not going to get smaller.
Having slept on it, some concerns, in descending order of severity.
Every item above is a live issue with the chart your feed re-litigates monthly. Through 2025, the frontier region of the METR plot — tasks of one to four hours — contained 14 samples, and the task topics are public, weighted toward cybersecurity CTFs and ML-engineering problems that labs openly train for; a lab can move its dot by upsampling those distributions, deliberately or by accident. Claude 3.7 Sonnet was assigned a 59-minute horizon while succeeding on roughly 60% of one-to-two-hour tasks, because it went 0-for on the 2–4 hour bucket and the logistic fit punished the whole curve for it. Shashwat Goel showed you can reconstruct the entire log-linear trend from aggregate accuracy plus the task-length distribution with a fixed slope — the individual task outcomes barely matter. METR's own limitations notes supply the rest: bootstrapped confidence intervals of roughly a factor of two in each direction, widening as the suite saturates; 50% and 80% horizons that are not independent estimates, because a two-parameter logistic cannot fit both ends of the curve; only 5 of 31 tasks over eight hours with measured rather than estimated human baselines; success rates that fall about eight points per unit of task “messiness”; and a standing notice that measurements above 16 hours are unreliable on the current task suite — which did not stop the discourse from treating a ≥16-hour point estimate for Mythos Preview as a fire alarm.
To be clear about the target: this is not a case against METR. They publish their runs, their code, and their caveats, and that transparency is the only reason this parody was buildable in an afternoon; task-length horizons remain a better question than benchmark accuracy. It is a case against the inference pipeline downstream — the one where a dot moves inside a 14-sample region of a two-parameter curve fit, and timelines, investment theses, and p(doom)s all reprice by close of business. If a chart moves your worldview, first count the samples doing the moving. Mine had two. They were dice.
Most of these criticisms are not new; for an organized writeup of the most important known issues in the original time horizon paper see my blog post from January that OP linked in the conclusion. I do have some comments on this post's methodology.
The super long tasks are not economically valuable, because no human would do 200x200 matrix inversion or Monte Carlo simulations by hand in 1985, so they're not really worth tens of weeks of human wages. The baseline for these should probably be a human C or Fortran programmer with access to a reasonably fast computer, probably an hour or two rather than weeks.
It is also not really true that automatic scoreability restricts the suite to things Excel is good at. E.g. Excel cannot do reasoning questions or computer use, but it does support scripting which is not included here.
If the point is that the scaffold heavily affects the intelligence of the model, METR tested this in February and found that Codex and Claude Code scaffolds don't outperform the standard scaffold we use. Generally the bigger issue has been models not using scaffolds properly than exactly how much optimization goes into the scaffold.
The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.
It's true that time horizon is highly correlated with success rate if you know the task length distribution, and I used this shortcut in my follow-up last year, for all the benchmarks where we didn't have individual question data. IMO it's not a major flaw because
This violates conservation of expected evidence. It can't be the case that capability chart and no capability chart are both evidence of dangerous capabilities, and it's not healthy or productive to feel fear regardless of the evidence.
Finally, this post seems almost entirely AI written, probably by Claude 4.8 or Fable 5. Pangram says it's 100% AI written. This created a bunch of minor issues in the writing.
"The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks."
This is beautiful. Thank you.