At many points now, I've been asked in private for a critique of EA / EA's history / EA's impact and I have ad-libbed statements that I feel guilty about because they have not been subjected to EA critique and refutation. I need to write up my take and let you all try to shoot it down.
Before I can or should try to write up that take, I need to fact-check one of my take-central beliefs about how the last couple of decades have gone down. My belief is that the Open Philanthropy Project, EA generally, and Oxford EA particularly, had bad AI timelines and bad ASI ruin conditional probabilities; and that these invalidly arrived-at beliefs were in control of funding, and were explicitly publicly promoted at the expense of saner beliefs.
An exemplar of OpenPhil / Oxford EA reasoning about timelines is that, as late as 2020, their position on timelines seemed to center on Ajeya Cotra's "Biological Timelines" estimate which put median timelines to AGI at 30 years later. Leadership dissent from this viewpoint, as I recall, generally centered on having longer rather than shorter median timelines.
An exemplar of poor positioning on AI ruin is Joe Carlsmith's "Is Power-Seeking AI an Existential Risk?" which enacted a blatant Multiple Stage Fallacy in order to conclude this risk was ~5%.
I recall being told verbally in person by OpenPhil personnel that Cotra and Carlsmith were representative of the OpenPhil view and would be the sort of worldview that controlled MIRI's chances of getting funding from OpenPhil, i.e., we should expect funding decisions to be premised on roughly these views and try to address ourselves to those premises if we wanted funding.
In recent personal conversations in which I exposited my current fault analysis of EA, I've heard people object, "But this wasn't an official OpenPhil view! Why, some people inside OpenPhil discussed different views!" I think they are failing to appreciate the extent to which mere tolerance of dissenting discussion is not central, in an organizational-psychology analysis of what a large faction actually does. But also, EAs have consistently reacted with surprised dismay when I presented my view that these bad beliefs were in effective control. They may have better information than I did; I was an outsider and did not much engage with what I estimated to then be a lost cause. I want to know the true facts of OpenPhil's organizational history whatever they may be.
I therefore throw open to EAs / OpenPhil personnel / the Oxford EAs, the question of whether they have strong or weak evidence that any dissenting views from "AI in median >30 years" and "utter AI ruin <10%" (as expressed in the correct directions of shorter timelines and worse ruin chances; and as said before the ChatGPT moment), were permitted to exercise decision-making power over the flow of substantial amounts of funding; or if the weight of reputation and publicity of OpenPhil was at any point put behind promoting those dissenting viewpoints (in the correct direction, before the ChatGPT moment).
This to me is the crux in whether the takes I have been giving in private were fair to OpenPhil. Tolerance of verbal discussion of dissenting views inside OpenPhil is not a crux. EA forum posts are not a crux even if the bylines include mid-level OpenPhil employees.
Public statements saying "But I do concede 10% AGI probability by 2036", or "conditional on ASI at all, I do assign substantial probability to this broader class of outcomes that includes having a lot of human uploads around and biological humans thereby being sidelined", is not something I see as exculpatory; it is rather a clear instance of what I see as a larger problem for EA and a primary way it did damage.
(Eg, imagine that your steamship is sinking after hitting an iceberg, and you are yelling for all passengers to get to the lifeboats. As it seems like a few passengers might be starting to pay some little attention, somebody wearing a much more expensive and serious-looking suit than you can afford, stands up and begins declaiming about how their own expert analysis does suggest a 10% chance that the ship takes on enough water to sink as early as the next week; and that they think this has a 25% chance of producing a broad class of genuinely attention-worthy harms, like many passengers needing to swim to the ship's next destination.)
I have already asked the shoggoths to search for me, and it would probably represent a duplication of effort on your part if you all went off and asked LLMs to search for you independently. I want to know if insiders have contrary evidence that I as an outsider did not know about. If my current take is wrong and unfair, I want to know it; that is not the same as promising to be easy to convince, but I do want to know.
I repeat: You should understand my take to be that of an organizational-psychology cynic who is not per se impressed by the apparent tolerance of dissenting views, people invited to give dissenting talks, dissenters still being invited to parties, et cetera. None of that will surprise me. I do not view it as sufficient to organizational best practices. I will only be surprised by the demonstrated past pragmatic power to control the disposition of funding and public promotion of ideas, contrary to "AGI median in 30 years or longer" and "utter ruin at 10% or lower", before the ChatGPT moment.
(If you doubt my ability to ever concede to evidence about this sort of topic, observe this past case on Twitter where I immediately and without argument concede that OpenPhil was right and I was wrong, the moment that the evidence appeared to be decisive. (The choice of example may seem snarky but is not actually snark; it is not easy for me to find other cases where, according to my own view, clear concrete evidence came out that I was definitely wrong and OpenPhil definitely right; and I did in that case immediately concede.))
Re "Oxford EAs" - Toby Ord is presumably a paradigm of that. In the Great AI Timelines Scare of 2017, I spent some time looking into timelines. His median, then, was 15 years, which has held up pretty well. (And his x-risk probability, as stated in the Precipice, was 10%.)
I think I was wrong in my views on timelines then. But people shouldn't assume I'm a stand-in for the views of "Oxford EAs".
I ran a timelines exercise in 2017 with many well known FHI staff (though not including Nick) where the point was to elicit one's current beliefs for AGI by plotting CDFs. Looking at them now, I can tell you our median dates were: 2024, 2032, 2034, 2034, 2034, 2035, 2054, and 2079. So the median of our medians was (robustly) 2034 (i.e. 17 more years time). I was one of the people who had that date, though people didn't see each others' CDFs during the exercise.
I think these have held up well.
So I don't think Eliezer's "Oxford EAs" point is correct.
What's the Great AI Timelines Scare of 2017?
In my memory, the main impetus was a couple of leading AI safety ML researchers started making the case for 5-year timelines. They were broadly qualitatively correct and remarkably insightful (promoting the scaling-first worldview), but obviously quantitatively too aggressive. And AlphaGo and AlphaZero had freaked people out, too.
A lot of other people at the time (including close advisers to OP folks) had 10-20yr timelines. My subjective impression was that people in the OP orbit generally had more aggressive timelines than Ajeya's report did.
Wow - @Toby_Ord then why did you have such a high existential risk for climate? Did you have large likelihoods that AGI would take 100 or 200 years despite a median date of 2032?
Toby Ord had a x-risk probability of 10% from AI and about 7% from other causes back then for a total of about 1/6.
Reading this, I thought Toby Ord had a total all-cause x-risk probabilitity of 10% back then at first and checked it. Thought this might be helpful since Eliezer specifically mentioned <10% x-risk from AI as very unreasonable.