The hummingbird should clearly have solved decision theory, then acausally traded with infinitely powerful beings from elsewhere in the multiverse who could create entire fireproof universes.
I started writing bc I felt obligated to respond but only continued because the worksheet limit thing was funny. It wouldn't be funny the second time, so commenting on "random slop about METR" probably won't eat infinite time. Unless it were much more prevalent I guess.
Apparently the Forum policy allows this and posts are supposed to be automatically flagged, but I don't see a flag on this post. Agree it looks like Fable.
Most of these criticisms are not new; for an organized writeup of the most important known issues in the original time horizon paper see my blog post from January that OP linked in the conclusion. I do have some comments on this post's methodology.
Task suite (HCALC, n=23). Human-Calibrated Arithmetic and Ledger Computations: quantitative tasks screened for objective, automatic scorability — grocery receipts, payroll, a 360-payment amortization schedule, descriptive statistics on 500 blood-pressure readings, a 200×200 matrix inversion, and two Monte Carlo simulations. Screening for automatic scorability conveniently restricts the suite to things Excel can attempt.
Excel finished the retirement simulation in 51 seconds, a 135,294× speedup
The super long tasks are not economically valuable, because no human would do 200x200 matrix inversion or Monte Carlo simulations by hand in 1985, so they're not really worth tens of weeks of human wages. The baseline for these should probably be a human C or Fortran programmer with access to a reasonably fast computer, probably an hour or two rather than weeks.
It is also not really true that automatic scoreability restricts the suite to things Excel is good at. E.g. Excel cannot do reasoning questions or computer use, but it does support scripting which is not included here.
The agent. Excel cannot type, so it is scaffolded with a human peripheral who keys in values at a measured 0.3 seconds per entry (n=1, mildly caffeinated) and contributes no cognition.
If the point is that the scaffold heavily affects the intelligence of the model, METR tested this in February and found that Codex and Claude Code scaffolds don't outperform the standard scaffold we use. Generally the bigger issue has been models not using scaffolds properly than exactly how much optimization goes into the scaffold.
80%-time horizon of frontier systems, 1985–2026
The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.
The logistic does the work. Two parameters, fit through mostly-ones and a few zeros. Given only my aggregate success rate and the task-length distribution, you could recover the horizon without knowing which tasks Excel passed.
Shashwat Goel showed you can reconstruct the entire log-linear trend from aggregate accuracy plus the task-length distribution with a fixed slope — the individual task outcomes barely matter.
It's true that time horizon is highly correlated with success rate if you know the task length distribution, and I used this shortcut in my follow-up last year, for all the benchmarks where we didn't have individual question data. IMO it's not a major flaw because  is known to differ wildly by task distribution and you can't estimate it just from aggregate accuracy; there could be some weird distribution on which even  isn't enough.
If a capability can sit 36 orders of magnitude above trend for four decades without anyone noticing, the correct response to any capability chart is fear, and the correct response to the absence of a capability chart is more fear.
This violates conservation of expected evidence. It can't be the case that capability chart and no capability chart are both evidence of dangerous capabilities, and it's not healthy or productive to feel fear regardless of the evidence.
Finally, this post seems almost entirely AI written, probably by Claude 4.8 or Fable 5. Pangram says it's 100% AI written. This created a bunch of minor issues in the writing.
This was actually a deliberate choice. People are more agreeable on EA Forum than LW, so the simplest model that fits the data is that everyone who agrees with a comment on EAF will disagree on LW. The button placement just facilitates this!
A counterfactual donation isn't just my act being contingent on your act, it means AMF getting a net extra $50 is contingent on your act. So if part of the $50 were a donation match that would be filled regardless, or some (especially low op cost) funder would make up the difference, it wouldn't fully count.
Counterfactual is also used to mean the second choice opportunity that defines an opportunity cost. "Should I take job A? Well my counterfactual is job B, where I would do X..."
My guess is something like: Many organizations have quarterly caps on the number of false claims published. Their employees often want to make false claims, but towards the end of the quarter they're at the cap, so they delay the post to the first day of the next quarter.
Okay, but why only April 1? Well, on Jan 1 everyone is on holiday, and on July 1 everyone is out enjoying the good weather. Oct 1 coincides with national holidays in populous countries like China and Nigeria, and in the US people are hung over from fiscal New Year's Eve. So we only really see the effect on April 1.
I would strongly predict that a false claims spike also happens in places with bad weather on July 1. Unfortunately, most places are in the Northern Hemisphere where it's warm, and Australia has good weather all year, so I think this is only testable when it snows in New Zealand.
I'd love to sign up, but due to adverse selection concerns I'd prefer to be matched with an EA picked uniformly at random (whether they signed up or not). Is this possible?
Some people likely will have traditional lifestyles that include animals, which necessitates some amount of suffering even if it is smaller than today
The hummingbird should clearly have solved decision theory, then acausally traded with infinitely powerful beings from elsewhere in the multiverse who could create entire fireproof universes.
I started writing bc I felt obligated to respond but only continued because the worksheet limit thing was funny. It wouldn't be funny the second time, so commenting on "random slop about METR" probably won't eat infinite time. Unless it were much more prevalent I guess.
Apparently the Forum policy allows this and posts are supposed to be automatically flagged, but I don't see a flag on this post. Agree it looks like Fable.
Most of these criticisms are not new; for an organized writeup of the most important known issues in the original time horizon paper see my blog post from January that OP linked in the conclusion. I do have some comments on this post's methodology.
The super long tasks are not economically valuable, because no human would do 200x200 matrix inversion or Monte Carlo simulations by hand in 1985, so they're not really worth tens of weeks of human wages. The baseline for these should probably be a human C or Fortran programmer with access to a reasonably fast computer, probably an hour or two rather than weeks.
It is also not really true that automatic scoreability restricts the suite to things Excel is good at. E.g. Excel cannot do reasoning questions or computer use, but it does support scripting which is not included here.
If the point is that the scaffold heavily affects the intelligence of the model, METR tested this in February and found that Codex and Claude Code scaffolds don't outperform the standard scaffold we use. Generally the bigger issue has been models not using scaffolds properly than exactly how much optimization goes into the scaffold.
The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.
It's true that time horizon is highly correlated with success rate if you know the task length distribution, and I used this shortcut in my follow-up last year, for all the benchmarks where we didn't have individual question data. IMO it's not a major flaw becauseÂ
This violates conservation of expected evidence. It can't be the case that capability chart and no capability chart are both evidence of dangerous capabilities, and it's not healthy or productive to feel fear regardless of the evidence.
Finally, this post seems almost entirely AI written, probably by Claude 4.8 or Fable 5. Pangram says it's 100% AI written. This created a bunch of minor issues in the writing.
This was actually a deliberate choice. People are more agreeable on EA Forum than LW, so the simplest model that fits the data is that everyone who agrees with a comment on EAF will disagree on LW. The button placement just facilitates this!
It seems possible to vibe code a chrome extension to do this in under 2 hours, maybe under an hour if you have codex or claude code set up already.
A counterfactual donation isn't just my act being contingent on your act, it means AMF getting a net extra $50 is contingent on your act. So if part of the $50 were a donation match that would be filled regardless, or some (especially low op cost) funder would make up the difference, it wouldn't fully count.
Counterfactual is also used to mean the second choice opportunity that defines an opportunity cost. "Should I take job A? Well my counterfactual is job B, where I would do X..."
My guess is something like: Many organizations have quarterly caps on the number of false claims published. Their employees often want to make false claims, but towards the end of the quarter they're at the cap, so they delay the post to the first day of the next quarter.
Okay, but why only April 1? Well, on Jan 1 everyone is on holiday, and on July 1 everyone is out enjoying the good weather. Oct 1 coincides with national holidays in populous countries like China and Nigeria, and in the US people are hung over from fiscal New Year's Eve. So we only really see the effect on April 1.
I would strongly predict that a false claims spike also happens in places with bad weather on July 1. Unfortunately, most places are in the Northern Hemisphere where it's warm, and Australia has good weather all year, so I think this is only testable when it snows in New Zealand.
I'd love to sign up, but due to adverse selection concerns I'd prefer to be matched with an EA picked uniformly at random (whether they signed up or not). Is this possible?