Here are my rules of thumb for improving communication on the EA Forum and in similar spaces online:
Say what you mean, as plainly as possible.
Try to use words and expressions that a general audience would understand.
Be more casual and less formal if you think that means more people are more likely to understand what you're trying to say.
To illustrate abstract concepts, give examples.
Where possible, try to let go of minor details that aren't important to the main point someone is trying to make. Everyone slightly misspeaks (or mis... writes?) all the time. Attempts to correct minor details often turn into time-consuming debates that ultimately have little importance. If you really want to correct a minor detail, do so politely, and acknowledge that you're engaging in nitpicking.
When you don't understand what someone is trying to say, just say that. (And be polite.)
Don't engage in passive-aggressiveness or code insults in jargon or formal language. If someone's behaviour is annoying you, tell them it's annoying you. (If you don't want to do that, then you probably shouldn't try to communicate the same idea in a coded or passive-aggressive way, either.)
I'm curious on why you can't (as opposed to don't want to or will not) discuss these on the EA Forum.
In my view, there are many points in the articles that could be interesting for discussion.
5
Yarrow Bouchard 🔸
Can't because I've received an indefinite, possibly permanent soft ban (severe rate limit of 2 comments per week) since January from the EA Forum mods with a warning that it may escalate to a full ban.
It's not really about something I can change — or at least that I'm willing to change — because it's ultimately an editorial decision by the mods about what kind of conversations, content, and disagreement they want on the EA Forum. In my understanding (which could be wrong), they want the EA Forum to be more of a collaborative space where contributors are uplifted, and not have too much emphasis on debate or criticism. And/or they just fundamentally disagree with what I think and the way I think (e.g., they think that what I take to be the cruxes of disagreements in many cases is not what's really the crux), to such an extent they don't think it's productive to discuss on the EA Forum. So, an editorial decision.[1]
Anyway, for people who do want to engage in these sorts of discussions, Substack is a great place for it. And if not Substack, where else do people have these sort of discussions these days? Uhh, Reddit?
1. ^
(Really long footnote for people who want a longer explanation.) Also, in many cases the users agree with the mods on this, and that's part of why the mods made this decision. A lot of users don't want to engage with too much criticism or debate, and tend to downvote posts or comments that are negative or critical rather than collaborative and uplifting. Which is a perspective I can certainly understand and empathize with. (Although, in my opinion, it also contributes to a filter bubble/echo chamber-type effect.)
This is all in my understanding and partially me reading between the lines, since I did not fully understand the explanation the mods gave me and found parts of it really confusing. This is me trying my best to reconstruct what they meant by what they said. I could be getting it wrong. I don't really know.
The
9
TFD
Although I think you and I would have several disagreements on the AI topic, I will put my vote in, to the extent anyone cares, that a ban was not justified in this case. There are things you've written that annoy me or that I'd have said differently, but in general I don't think these are anywhere close to warranting a ban (or frankly the level of downvotes some of your comments have gotten). I also think in several discussions you've been involved in that went unproductively, the people you were responding to or who responded to you are at least equally to blame and sometimes have behaved worse, including clearly uncivil behave (e.g. clearly intentionally insulting phrasing, unreasonable accusations of bad faith etc.).
5
Vasco Grilo🔸
Thanks for sharing, Yarrow. @Toby Tremlett🔹, have you considered being transparent about the reasons for soft and hard bans? I think the EA Forum used to have a thread where they were announced. I believe transparency is important to keep the moderation team accountable.
I have limited context. However, based on the above, I get the impression a soft ban is not justified. I do not think people should be banned due to disagreements with their thinking.
It's likely we should invest in public mod comments again, as this case underlines, it's pretty unfair to Yarrow to have to explain it to their interlocutors.
As you might guess I strongly disagree with pretty much every aspect of Yarrow's framing of the rate-limit we applied to their account. I'll talk to other mods about whether we want to reinstate the public comments, and if so I'll share more detail on the Yarrow decision in said public comment.
A quick version is that we made this decision because of the way that Yarrow disagrees with people, not the fact that they do. Primarily because of repeatedly sneer-y and snarky comments[1], but also because of generally unproductive and attritional disagreements, for example, see this description I sent to Yarrow at the time of the rate-limit:
"There is a clear pattern in your comments — you seem to have particularly unproductive disagreements with other users, generally due to an overly literal interpretation on your part, or excess defensiveness. This is no great sin of yours, but it isn’t great for the quality of Forum discussion, especially when you are naturally so prolific."
This wasn't a decision taken lightly... (read more)
Weighing in as a comment I made has been cited below: thank you for your moderation efforts.
I'm an admittedly infrequent poster/commenter on the Forum, in no small part because I don't have a lot of time on my hands. As such I'm often discouraged from engaging with certain topics or (potentially) productive disagreements because I simply cannot afford to engage with the kind of [charitable ? detailed : gish galloping] posts and comments exemplified in this thread.
Just wanted to provide something to counterbalance any adverse selection in responses.
7
JP Addison🔸
Hey, just stepping into this thread to say that Toby's out with a family emergency. I know many of you put a lot of time and care into the Forum, and I'm sorry the rest of moderation team doesn't have capacity to engage in this comment section in his stead.
8
Vasco Grilo🔸
Thanks for the clarifications, Toby. They made sense to me.
1
fergusq
I feel that there could be greater transparency in how these moderation decisions are done. In my opinion, although Yarrow has been quite confrontational at times, they haven't really done anything that would necessitate such a punishment. While the forum is allowed to moderate its users in any way it likes, one shouldn't be rate-limited just because they have disagreements, especially if their opinions go against the mainstream opinion on the forum. This puts the neutrality of the forum moderation at question.
Personally, I have pretty bad experiences writing on this forum. I get highly-upvoted comments from people who seemingly haven't even read what I wrote and instead throw in bad-faith arguments, which I find borderline insulting and certainly unwelcoming. See, for example, this comment that was given as a reply to my text, where I pointed out that many sources use narrative arguments (stories, parables, etc.) instead of technical evidence. The commenter accused me of writing a narrative argument... which is simply not true, as my text was not written in a form of a story. Either they didn't read at all what I wrote or intentionally misunderstood it. And that is the most upvoted comment!
In general, the quality of discussion on this forum is very low. I feel it is pretty harsh to say this, but it is my experience and it strongly discourages me from participating at all. In this atmosphere, rate-limiting Yarrow appears not as an impartial decision, since many other, much more problematic comments appear in the forum all the time.
7
mjkerrison🔸️
1. I think it reflects poorly on you that a "highly-upvoted" comment was a stronger update towards 'the Forum at large is wrong' more than 'you were wrong' or even 'you should change how you communicate'.
2. On the object level: I read your post, and like ~half of people who reacted to it, disagreed with it; specifically I think it has narrative-like qualities, and regardless of how much we quibble over the precise point at which text can be reasonably called "narrative", I thought it was non-evidence-based argumentation, which you were arguing against.
3. I think in general it is bad forum etiquette to "snitch tag" like this. I think you're misrepresenting my comment and raising it in an irrelevant context.
1
fergusq
In the context of my post, the term "narrative argument" was really clearly used to refer to fictional stories embedded in a text as part of its argument, such as AI2027's short story or the parables in Y&S's book. My text simply does not have that.
By completely misunderstanding even basic concepts such as "narrative argument", your comment was just a snarky jab and had little value. It didn't comment the actual content of my text, instead, and largely unjustified by any real analysis, it tried to "get back at me" by blaming me for things I blame others for, like non-evidence based argumentation and narrative arguments.
Since I find it difficult to believe that you really don't understand what a narrative argument is, I reasonably count a high probability for you simply acting in bad faith.
I don't know if "snitching" you is allowed or not, and I apologize for the moderators if it's not, but I think using examples of bad behavior is relevant in discussions of what is allowed on the forum.
3
Yarrow Bouchard 🔸
I empathize with your experience on the EA Forum. The top comment on this post is a clear example where someone just fundamentally misunderstood the point you were trying to make, and responded in an unhelpful and kinda snarky way. Then when you clarified your point to them, they just repeated their original point again. Really frustrating. Sometimes it seems like people don’t have the patience to deeply engage, yet they do have the patience to comment (often rudely). Which is not a good headspace for discussion.
It could potentially be nice to have alternatives to the EA Forum where stronger discussion norms around civility and generosity are upheld. I don’t know if there is a critical mass of people to support that, though. I can potentially send a group chat invite to anyone who wants to message me privately here or on Substack, or email me.
The biggest thing for the mods to understand is when you let people act abusively, it pisses people off so much, and it’s so hurtful and feels so yucky, people don’t want to engage anymore. There are people on the EA Forum who effectively have a heckler’s veto because they just make the experience of using the forum too unpleasant to tolerate for anyone they disagree with. And the end result is you get an insular culture, where even pointing out objective, uncontroversial flaws in research or analysis gets strongly discouraged.
I had a horrible time pointing out errors in a survey. I got a brusque, dismissive response. And a lot of downvotes. The errors were later corrected, but I didn’t get any apology or even acknowledgement. What a thankless job!
You have to be able to tolerate the discomfort of disagreement and intellectual criticism, to have the patience to deeply engage, to have curiosity about other people’s perspectives and humility about your own, in order to think well and do good analysis. Using personal insults and hostility to shut down disagreement and criticism is not intellectually healthy.
-1
Yarrow Bouchard 🔸
Hi Toby. Me and Toby had a back in forth about this back via private messages in December and January, and this public exchange of comments now is pretty much recreating the conversation we had privately. I didn't know then and still don't know now to what extent I just don't understand Toby's perspective but could understand it better with more investment, versus to what extent Toby and I just have an editorial, intellectual, or moral disagreement.
For instance, I noticed that some people were using the "Made me laugh" reaction not for its intended purpose but to express ridicule and mockery on comments they disagreed with. I reported this, and I think I may have even separately messaged the mods about this, and I also sent screenshots to Toby when we were talking privately about civility. I never got to the bottom of whether Toby (or the other mods) thinks this is within the bounds of acceptable civility on the EA Forum, or what.
To me, I feel like openly laughing at someone, expressing ridicule or mockery, is, like, paradigmatically uncivil. Right? Or maybe not everyone sees it that way. I don't know if Toby or the other mods agree with me on that or not. I never heard anything back about that other than Toby just saying he disagreed that some of the examples of behaviours I thought broke civility norms actually broke civility norms. Does this include laugh reacting in order to mock people? I really don't know. Toby, any thoughts on that? Maybe would be good to clarify to people if it's open season on doing this or if they might get punished for it. I feel like it would be a good thing to discourage...
If you find yourself in a position in the future where you want to encourage someone to engage differently on the EA Forum, it would be helpful to be more specific in what you're asking them to do. For instance, phrases like "unproductive disagreements", "overly literal interpretation", "excess defensiveness" are very general. What do you specifically mean? The
But, I dunno, sneering and snarking just seems like par for the course for the EA Forum — including using laugh emojis to mock people you disagree with.
Again, this is another issue with the current lack of moderation transparency on the Forum. Misuse of the laugh react is seen as breaking forum norms, and (AFAIK) in almost every case has led to at least a moderator message.
Civility enforcement is a major part of what we do as moderators. It's inevitable that some borderline cases will remain on the site such that someone motivated to find them - as you were - will be able to.
Bit of a side-point but the amount of civility enforcement we do has been one of the reasons I've been reluctant to publicise mod actions. I appreciate being able to send someone a more casual message for butting up against guidelines rather than having to call them out publicly (which is a far more embarassing result of what can be a fairly minor infraction).
If you find yourself in a position in the future where you want to encourage someone to engage differently on the EA Forum, it would be helpful to be more specific in what you're asking them to do.
It's hard to respond to snark like this w... (read more)
One section from that post raises the concept of 'Asymmetric effort ratios'. This is definitely part of our moderation decision. At one point, if I remember correctly, you wrote almost a fifth of the words on the Forum in a week. You are very productive of long comments, which are often packed with difficult to dispel misunderstandings. This is part of why a rate-limit was the solution we arrived at. In small doses, you can be a valuable contributor, but without limit, it becomes unfairly taxing on your interlocutors.
I feel like the issue addressed in the Said Saga is somewhat the opposite of what it seems like Yarrow does. A paradigmatic example of the bad bahavior attributed to Said was posting extremely short comments, such as just saying "examples?". Its obviously extremely easy to type a one word comment, and the possibility of adding additional examples of something a poster describes isn't exactly a deep insight. The asymetry is that it is very easy for Said to post a comment like that but would take a lot of effort for the original poster to respond providing examples, giving rise to the complaint that Said wasn't willing to put in equal effort.
Thank you so much for this. I really appreciate it. I’m grateful for how kind and thoughtful this comment is.
On the moderator time thing, I just want to say that Toby only ever sent me 3 messages regarding moderation on the EA Forum. There was an initial message (452 words) informing me of a temporary rate limit/soft ban. This was my first communication from any of the moderators. After I replied, there was a shorter follow-up message (200 words) where Toby mentioned he didn’t have time to provide examples of problematic comments. Then there was a third and final message (319 words) informing me the temporary rate limit/soft ban had been made permanent. That was the full extent of the communication.
That third and final comment was the first and only time Toby explained this reason for the ban, which he’s quoted in this thread a few times:
However there is a clear pattern in your comments — you seem to have particularly unproductive disagreements with other users, generally due to an overly literal interpretation on your part, or excess defensiveness. This is no great sin of yours, but it isn’t great for the quality of Forum discussion, especially when you are naturally so prolific
Yeah, I'm trying to maintain openness to different possibilities on the time issue to an extent since I don't really know what happened. If I had to venture a guess (which could obviously be wrong), I'd say something like this:
Other forum users who got frustrated with your posts/comments reach out to the mod team privately, mod team has extended discussions amongst themselves, decides on the soft ban, and then reaches out to you to tell you. If this is what happened, I can imagine that it did take a reasonable amount of time and also its understandable that the mod team would want to incorporate feedback from users, but I would say this is a mistake on the part of the mod team if that's what happened. If you're talking here about the value of being able to reach out privately, wouldn't this be the time to do that, before going to the soft ban? If you're not making the decision lightly and discussing a lot and writing google docs, couldn't you copy some examples of problematic comments or posts from these documents fairly easily?
I think a prcess where there is a lot of back channelling has a siginificant risk of filter bubble/echo chamber issues as you mention, similar to what Habry... (read more)
Yeah, totally, I wondered the same thing. Am I seeing the tip of the iceberg and the other 90% of the time spent is all behind the scenes? Could totally be possible, but is that really the best way to do things?
I really want to see stricter moderation of online spaces to maintain respect, kindness, and psychological safety, as much as possible. I think this would mean mods often intervening in cases they don’t normally now (not just on the EA Forum, elsewhere too). I want mods to be less hands-off in that way. I think the end result is online spaces would bring more enjoyment, less suffering, and would facilitate more open-minded, curious, creative, stimulating discussions, rather than the amount of arguing to win we see.
But mods making editorial decisions about content from the perspective of what they think is incorrect vs. correct or unreasonable vs. reasonable is dicey territory. Obviously, you just have to ban people posting ChatGPT rants about homeopathy or whatever. But if you’re getting into really subtle and contested ideas about which there is widespread disagreement, then do the users of a site like the EA Forum actually want moderators to make the call about what’s correct and acceptable vs. incorrect and ban-worthy?
It’s particularly relevant to the EA Forum for the reasons you said. Some popular ideas on the EA Forum, most of all that there will soon be an existentially dangerous AGI, are things that, I don’t know, something like 95% or 99% or 99.9% of people disagree with. If you want to convince the skeptics, who are the overwhelming majority, then how is this approach going to work? You say skeptics’ ideas are stupid and then ban them. Okay… is that… scalable?
8
TFD
Here are some random thoughts on the topic.
Moderators have a hard job, I think it can't be entirely on moderation to drive the culture of a website. A lot of the work has to be on the users. Starting with moderation issues though, I have a couple ideas:
Clarity: I think moderation benefits from simplicity and clarity. It isn't a good sign when you are taking mod action against someone but can't really explain why because its too difficult or would take too long. I feel like that indicates that the underlying rules/principles aren't really clear or simple enough. It is hard for people to adapt to comply with complicated and unclear rules and the road to motivated reasoning is also paved in vague principles that are easily applied differently to different stituations.
Proportionality: This one goes in both directions. I think sometimes it would be better for mods to step in early but with a lighter touch, something like "this seems to be getting a bit heated/unproductive, friendly reminder to everyone to keep it civil". An ounce of prevention and all that.
For what users can do:
Stick to the topic/don't go meta: Stay grounded in the discussion, try not to import assumptions based on previous arguments with people vaguely on the same "side" as who you are talking to, focus on their arguments. Try to make the discussion more specific rather than more general. Try not to take your argument in a meta direction, don't talk about what arguments are good in the abstract or focus too much on claiming that the other person's arguments are an example of a general phenonemon, try to respond to their claims specifically.
Try to stay calm: It is common that people feel a bit uncomfortable when faced with strong disagreements, including that the person they are talking to is being unfair or mean in some way. The problem is that if both people go along with this feeling it often leads to a bad place. If one person temporarily lapses into a less friendly tonw but the other per
2
Yarrow Bouchard 🔸
Love this! Thank you! I love the term “epistolary debates”. I have floated using this format with a few people, like David Mathers (who I find fun to talk to, respectful, curious, and full of intellectually stimulating ideas). The EA Forum has the Dialogues feature which is purpose-built for this, but you don’t need anything fancy. Substack or any blog or website will allow you to just copy and paste a bunch of emails or Slack/Discord messages or whatever.
This format seems to have fallen out of favour, but there was a time on the Internet back in the 2000s when I enjoyed reading email debates like the Edge.org debate on the Anthropic Principle between physicists Leonard Susskind and Lee Smolin. Sam Harris and Andrew Sullivan also did an email debate on religion. To me, this format is 100x more interesting than live, verbal debates. I don’t really care if someone is good at thinking on the spot and coming up with a clever rejoinder right away. I would much rather they take a day or two to think about it and then respond. Let’s bring back this forgotten format!
I totally agree users also have a responsibility for the culture they create, and not everything can be pinned on the moderators. I think it’s pretty rare, but I’ve experienced at least once an online community where the moderators were trying their best and doing a good job, but the users’ behaviour became so acrimonious it overwhelmed the moderators’ capacity to moderate the community.
Why I think moderators play such a key role is that: on a site like the EA Forum, if there were zero moderation, literally just one person could ruin the whole site for everyone by posting spam, malware links, porn gifs, etc. across the site. The site would become unusable and the users would have to leave. More subtly, 1% or 5% or 10% of users can ruin a site for everyone else if they habitually insult other people’s intelligence, or shame or mock or belittle them, or relentlessly make passive-aggressive insults. I think
7
TFD
Thanks for your kind words.
On the bad faith/"truth seeking" point, I've also noted some issues in the way "truth seeking" is used in a previous post, and thinking about this case gave me an idea. It seems like there is a general phenomenon in EA/rationalist discourse where intent gets obscured or ignored somehow. Perhaps not surprising for intellectual communities that are very into consequentialism?
I think the effect of using the "truth seeking" terminology is to confuse multiple possibilities around intent:
Lying: intentional
Insufficient rigor/evidence/etc.: can be an unintentional mistake
Callousness about the truth: I think people often feel like even if someone isn't lying, they can demonstrate a disregard for truth that feels like its intentionally misleading
Being "insufficiently truth seeking" could refer to any of these, and thus using the phrase fails my principle of clarity. It also becomes strongly subject to motivate reasoning or motte/bailey dynamics because the meaning can shift among these different meanings.
I feel like something similar has happened here with the whole "hard to dispell misunderstandings" thing. Tskeen laterally, it doesn't make a whole lot of sense to soft ban someone for this. Is the implicict message that only easy to dispell misunderstandings are allowed? Surely not. But it makes more sense if you imagine its implicitly standing in for a spectrum of actions based on intention:
Bad faith: intentionally obscuring your real views, often with the goal of making them harder to respond to.
Genuine mistake: unintentional, even of hard to dispell
Game playing: being coy or cagey about what your views really are or in some way deliberately making your position confusing or hard to respond to.
The last one I think is kind of what Said was being accused of in the post referenced above? But IMO its extremely clear that you haven't been doing anything intentionally misleading. I think this is an instance of concerns about "epis
2
Yarrow Bouchard 🔸
Oh, you wrote that post on “truthseeking” too!! I forgot that!! Another helpful post.
“Truthseeking” as a term drives me bananas because it’s so vague and ambiguous — I looked really hard and couldn’t find any real attempts to define it clearly, if even to define it at all — and pretty much the only way I see the term get used is when someone wants to slam someone else they disagree with. And it’s definitely never clear to me that the person who’s accused of “not truthseeking” is doing anything wrong, or making bad points, or that their views are wrong. It just seems like an argument got heated.
If people say “not truthseeking” and they just mean “bad evidence/bad arguments/ill-informed points”, they should just say that. Ditto for “bad faith” if that’s what it’s supposed to mean.
Thank you for seeing me clearly. It’s a huge relief. I found it super confusing and hurtful when Toby said I’m not engaging in good faith, that I was snarky when I was really being heartfelt and sincere, and that I was “motivated” to find examples of uncivil behaviour on the EA Forum (?). These all feel like such foreign understandings of my intent. And the throughline between them feels like Toby is telling himself a story about me where I’m out to cause trouble or something. I don’t know. I really can’t understand what’s happening here.
I do feel like I’m transparent. I don’t know why someone would think I’m sneaking around.
Do you think this is a LessWrong subculture thing? I notice in the LessWrong-o-sphere, there’s all this emphasis on secrets, game theory, strategizing, signalling, counter-signalling, yada yada. Does that make people feel especially suspicious of each other? Or of “outsiders”?
This line from a post by a pseudonymous person involved in the LessWrong community always sticks out in my mind:
Somehow this rings true to me, although I don’t know if I can put my finger on why. Maybe it’s because there is such a lack of psychological safety in the LessWrong community,
For what it's worth, whilst I find arguing with Yarrow quite stressful, and I definitely don't always agree with her, and sometimes think she's a bit reluctant to concede, I don't think a ban was a good idea. Her comments are usually substantive, people who don't want to engage with them can just ignore, critical perspectives are always valuable etc. In some cases, she's actually spotted pretty important stuff that people had missed, i.e. that Waymo's still have humans in the loop, or a problem in a Forecasting Research Institute study that I cited during one our arguments.
Thank you, David, that’s very generous of you. I just want to say that you were, to me, definitely the best commenter I talked to on the EA Forum. Although it can also kinda be fun to spar a bit over intellectual topics, thinking back now, I worry whether I was too harsh with you at times. Maybe sparring can sometimes get too heated, and if I ever fail to treat someone with kindness and empathy and respect, that’s wrong and my fault.
I feel a bit guilty and regretful to hear you say you’ve found arguing with me stressful. I definitely don’t want to make someone feel stressed, and now I’m wondering what I could do differently to help people feel less stressed in the future. Now that I think about it, I also feel stressed during a lot of intellectual arguments, and I wonder why. It seems like it shouldn’t be be like that, and like something is going wrong.
What I’ve appreciated most about talking to you is your level of sincere curiosity. The topics we’ve discussed seem like live issues to you, where you’re genuinely trying to make up your mind and get the best information and argumentation, and not just argue to win. Conversely, you’ve often made me pause, get curious, and give things... (read more)
I can see both general guidance and specific examples about how to improve comments being useful. I believe they are often more effective together.
I feel like it should often be possible to give guidance to someone in public without embarassing them. One could explicitly say the infraction is minor, and use a casual tone. Public moderation may help reinforce good discussion norms in addition to increasing transparency.
I do not think the number of comments people can write should be greatly limited due to them having written long comments (supposedly) packed with difficult to dispel misunderstandings (I am not saying this applies to Yarrow or not; I do not know which comments you have in mind). Readers can lightly or strongly downvote comments they do not find useful, disagree with them, and point out clear mistake. What is a difficult to dispel misunderstanding is quite subjective. So effective moderation here seems difficult to me.
When writing a lot is the problem (I personally do not think this can by itself be a problem), it would be better for the rate limit to be defined in terms of words per week instead of number of comments per week? Alternatively, there could be a limit for the number of words, and a less strict limit for the number of comments.
I can see both general guidance and specific examples about how to improve comments being useful. I believe they are often more effective together.
I do agree Vasco, but I don't endorse me spending much more time on this right now.
I feel like it should often be possible to give guidance to someone in public without embarassing them. One could explicitly say the infraction is minor, and use a casual tone.
Maybe? I personally would be much more discouraged from posting by a public comment than a private message. However it'd be nice if we could have a live log somewhere where forum users could see that we dm'd an anonymous user.
When writing a lot is the problem (I personally do not think this can by itself be a problem)
Writing a lot in itself isn't the problem. It's that the writing contains difficult-to-dispel misunderstandings, and therefore creates an asymmetric effort ratio.
This is the first time I'm hearing about your or the other mods' reasoning about why I was soft banned/rate limited. I was never told this before. (Edit: Sorry, this was confusing. I meant all this reasoning that Toby is presenting now is new to me, and stuff I hadn't heard before. Toby did give me a short, very general explanation that I found confusing, which he's quoted both above and below.)
How does one determine whether comments are "packed with difficult to dispel misunderstandings"? Does this mean you read comments and decide whether they're correct or incorrect, or whether you agree or disagree?
Can you think of a good example of a misunderstanding on my part that's representative of the overall problem, as you see it?
8
Toby Tremlett🔹
It's not the first time in the sense that it is part of what I meant by this paragraph. I'm sorry, but I really cannot invest further in this right now.
I can say that many of your negative karma comments are negative karma for ~ this reason. I have no doubt that you will disagree at length with any particular example.
2
Yarrow Bouchard 🔸
Sorry, I should clarify. Yes, you definitely sent me that short explanation before. But all the reasoning you're presenting now is brand new to me.
I can believe that you intended "particularly unproductive disagreements with other users, generally due to an overly literal interpretation on your part, or excess defensiveness" to include "packed with difficult to dispel misunderstandings", but of course there was no way for me to know that.
I'm not sure how this doesn't just come down to moderating based on the content of someone's views as at least one major factor under consideration — whether you decide someone's understanding of some topic is correct or incorrect. It's just that I thought you were disagreeing with me above when I said it seemed like an editorial decision to me. But this really makes it sound like, at least in significant part, an editorial decision.
Just to be clear, I'm not saying editorial decisions are bad. If I moderated a forum like this, I would also make editorial decisions.
2
Vasco Grilo🔸
Yes, I can see many people preferring a private message. One could ask people whether they would be fine with the private message being made public.
I also think public messages are more valuable when the moderation is more contentious, and this makes them less embarrassing because it will be less clear that people did something wrong.
2
Yarrow Bouchard 🔸
I don't know if it really matters if the message from a mod is public or private as long as comments that break the rules get removed.
If someone writes a comment that says "you're stupider than a potted plant", that can just turn into [comment removed by moderator] or whatever. Or if someone writes an otherwise fine comment and ends it with "you're stupider than a potted plant", a moderator could edit out just the last part that breaks the rules.
What is problematic is when the comments are never removed or edited, reports to the mods get no response, and there is no observable moderator action. None of the uncivil comments I've ever reported have gotten removed, and only one (among many, many, many) got a public comment from a moderator. I just checked and some mean laugh reacts (intended for mockery) are still up. Those were never removed.
There's so many different options. Instead of taking the comment down, mods could leave a brief public comment saying "Please stay civil" or whatever and then engage more deeply with the person in a private message.
I haven't noticed the incivility problem improving at any point that I've been active on the forum, and I've noticed some repeat offenders. I feel like whatever is being done isn't working. And maybe part of the reason is that people understand what's okay from what they see on the forum, and a lot of what's on the forum is uncivil.
I also get the impression that the mods just have a much more lax view than I do about what counts as incivility. There are cases where mods seemingly just disagreed there was any reason for them to take action. I also think Toby, a moderator, using phrases like "packed with hard to dispel misunderstandings", etc., is sort of a sign of how normalized harsh language is on the EA Forum. I think this sort of language is just... I don't know, it's so unpleasant that I just don't want to be around it.
I don't want to say definitively that I'll never participate on the EA Forum at all ev
1
Yarrow Bouchard 🔸
I'm not sure what this is supposed to mean. What do you mean I was motivated to find examples of incivility? I would have preferred not to see them. I made posts or comments, and people wrote mean replies, or did the mean laugh react thing. I didn't intentionally look for them.
To my recollection, I never saw a single comment or mean laugh react get removed in the entire time I've been active on the forum.
I find this a bit hurtful and confusing. I was not trying to be snarky at all, I'm trying to give earnest feedback. I am now rethinking what you said about how I'm repeatedly snarky and it's an overall pattern. Because I'm trying to be earnest and diplomatic and polite, and you're saying it's snark. Huh? Is this how you've been reading everything I've been saying all along? That I say something genuine and you think it's mean-spirited or sarcastic?
I mean, that is buck wild. I was not aware of that. I can see how that could be a problem, but why not say that from the beginning? Why am I only just hearing about this now? I feel like this is something I would have been responsive to if you had brought it up to me whenever this was happening.
5
Toby Tremlett🔹
Apologies, I didn't want to confuse or hurt you. I found the comment somewhat patronising, but that may well not have been your intention.
I used to feel so strongly about effective altruism. But my heart isn't in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.
But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk h... (read more)
I'd distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they've always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil's website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn't much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
Yes, this seems similar to how I feel: I think the major donor(s) have re-prioritized, but am not so sure how many people have switched from other causes to AI. I think EA is more left to the grassroots now, and the forum has probably increased in importance. As long as the major donors don't make the forum all about AI - then we have to create a new forum! But as donors change towards AI, the forum will inevitable see more AI content. Maybe some functions to "balance" the forum posts so one gets representative content across all cause areas? Much like they made it possible to separate out community posts?
2
Jeroen Willems🔸
Thanks for sharing this, while I personally believe the shift in focus on AI is justified (I also believe working on animal welfare is more impactful than global poverty), I can definitely sympathize with many of the other concerns you shared and agree with many of them (especially LessWrong lingo taking over, the underreaction to sexism/racism, and the Nonlinear controversy not being taken seriously enough). While I would completely understand in your situation if you don't want to interact with the community anymore, I just want to share that I believe your voice is really important and I hope you continue to engage with EA! I wouldn't want the movement to discourage anyone who shares its principles (like "let's use our time and resources to help others the most"), but disagrees with how it's being put into practice, from actively participating.
I don't think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
Good point, I guess my lasting impression wasn't entirely fair to how things played out. In any case, the most important part of my message is that I hope he doesn't feels discouraged from actively participating in EA.
2
Benevolent_Rain
On cause prioritization, is there a more recent breakdown of how more and less engaged EAs prioritize? Like an update of this? I looked for this from the 2024 survey but could not find it easily: https://forum.effectivealtruism.org/posts/sK5TDD8sCBsga5XYg/ea-survey-cause-prioritization
I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.
Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.
In January 2020, some stores sold out of face masks in several different cities in North America. (One example of many.) The oldest post on LessWrong tagged with "covid-19" is from well after this started happening. (I also searched the forum for posts containing "covid" or "coronavirus" and sorted by oldest. I couldn’t find an older post that was relevant.) The LessWrong post is written by a self-described "prepper" who strikes a cautious tone and, oddly, advises buying vitamins to boost the immune system. (This seems dubious, possibly pseudoscientific.) To me, that first post strikes a similarly ambivalent, cautious tone as many... (read more)
YARROW: Boy, one would have to be a complete moron to think that COVID-19 would not be a big deal as late as Feb 28 2020, i.e. something that would imminently upend life-as-usual. At this point had China locked down long ago, and even Italy had started locking down. Cases in the USA were going up and up, especially when you correct for the (tiny) amount of testing they were doing. The prepper community had certainly noticed, and was out in force buying out masks and such. Many public health authorities were also sounding alarms. What kind of complete moron would not see what’s happening here? Why is lesswrong patting themselves on the back for noticing something so glaringly obvious?
MY REPLY: Yes!! Yes, this is true!! Yes, you would have to be a complete moron to not make this inference!! …But man, by that definition, there sure were an awful lot of complete morons around, i.e. most everyone. LessWrong deserves credit for rising WAY above the incredibly dismal standards set by the public-at-large in the English-speaking world, even if they didn’t particularly surpass the higher standards of many virologists, preppers, etc.
The version of the claim I have heard is not that LW was early to suggest that there might be a pandemic but rather that they were unusually willing to do something about it because they take small-probability high-impact events seriously. Eg. I suspect that you would say that Wei Dai was "late" because their comment came after the nyt article etc, but nonetheless they made 700% betting that covid would be a big deal.
I think it can be hard to remember just how much controversy there was at the time. E.g. you say of March 13, "By now, everyone knows it's a crisis" but sadly "everyone" did not include the California department of public health, who didn't issue stay at home orders for another week.
[I have a distinct memory of this because I told my girlfriend I couldn't see her anymore since she worked at the department of public health (!!) and was still getting a ton of exposure since the California public health department didn't think covid was that big of a deal.]
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
I like this comment. This topic is always at risk to devolving into a generalized debate between rationalists and their opponents, creating a lot of heat but not light. So it's helpful to keep a fairly tight focus on potentially action-relevant questions (of which the comment identifies one).
0
Yarrow Bouchard 🔸
I've been around EA pretty deeply since 2015, and to some degree since around 2009. My impression is that overall it's what you guessed it might be: "selective memory, or post-hoc narrative construction." Particularly around AI, but also in general with such claims.
(There's a good reason to make specific, dated predictions publicly, in advance, ideally with some clear resolution criteria.)
8
niplav
Thank you, this is very good. Strong upvoted.
I don't exactly trust you to do this in an unbiased way, but this comment seems the state-of-the-art and I love retrospectives on COVID-19. Plausibly I should look into the extent that your story checks out, plus how EA itself, the relevant parts of twitter or prediction platforms like Metaculus compared at the time (which I felt was definitely ahead).
9
Linch
See eg traviswfisher's prediction on Jan 24:
https://x.com/metaculus/status/1248966351508692992
Or this post on this very forum from Jan 26:
https://forum.effectivealtruism.org/posts/g2F5BBfhTNESR5PJJ/concerning-the-recent-2019-novel-coronavirus-outbreak
I wrote this comment on Jan 27, indicating that it's not just a few people worried at the time. I think most "normal" people weren't tracking covid in January.
I think the thing to realize/people easily forget is that everything was really confusing and there was just a ton of contentious debate during the early months. So while there was apparently a fairly alarmed NYT report in early Feb, there were also many other reports in February that were less alarmed, many bad forecasts, etc.
6
Yarrow Bouchard 🔸
It would be easy to find a few examples like this from any large sample of people. As I mentioned in the quick take, in late January, people were clearing out stores of surgical masks in cities like New York.
5
Linch
Why does this not apply to your original point citing a single NYT article?
0
Yarrow Bouchard 🔸
It might, but I cited a number of data points to try to give an overall picture. What's your specific objection/argument?
8
Linch
My overall objection/argument is that you appear to selectively portray data points that show one side, and selectively dismiss data points that show the opposite view. This makes your bottom-line conclusion pretty suspicious.
I also think the rationalist community overreached and their epistemics and speed in early COVID were worse compared to, say, internet people, government officials, and perhaps even the general public in Taiwan. But I don't think the case for them being slower than Western officials or the general public in either the US or Europe is credible, and your evidence here does not update me much.
4
Yarrow Bouchard 🔸
Let's look at the data a bit more thoroughly.
It's clear that in late January 2020, many people in North America were at least moderately concerned about covid-19.
I already gave the example of some stores in a few cities selling out of face masks. That's anecdotal, but a sign of enough fear among enough people to be noteworthy.
What about the U.S. government's reaction? The CDC issued a warning about travelling to China on January 28 and on January 31, the U.S. federal government declared a public health emergency, implemented a mandatory 14-day quarantine for travelers returning to China, and implemented other travel restrictions. Both the CDC warning and the travel restrictions were covered in the press, so many people knew about it, but even before that happened, a lot of people said they were worried.
Here's a Morning Consult poll from January 24-26, 2020:
An Ipsos poll of Canadians from January 27-28 found similar results:
Were significantly more than 37% of LessWrong users very concerned about covid-19 around this time? Did significantly more than 16% think covid-19 posed a threat to themselves and their family?
It's hard to make direct, apples-to-apples comparisons between the general public and the LessWrong community. We don't have polls of the LessWrong community to compare to. But those examples you gave from January 24-January 27, 2020 don't seem different from what we'd expect if the LessWrong community was at about the same level of concern at about the same time as the general public. Even if the examples you gave represented the worries of ~15-40% of the LessWrong community, that wouldn't be evidence that LessWrong users were doing better than average.
I'm not claiming that the LessWrong community was clearly significantly behind. If it was behind at all, it was only by a few days or maybe a week tops (not much in the grand scheme of things), and the evidence isn't clear or rigorous enough to definitively draw a conclusion like that. My cla
6
Linch
Thanks, I find the polls to be much stronger evidence than the other things you've said.
2
Yarrow Bouchard 🔸
I recommend looking at the Morning Consult PDF and checking the different variations of the question to get a fuller picture. People also gave surprisingly high answers for other viruses like Ebola and Zika, but not nearly as high as for covid.
2
Yarrow Bouchard 🔸
If you want a source who is biased in the opposite direction and who generally agrees with my conclusion, take a look here and here. I like this bon mot:
This is their conclusion from the second link:
3
parconley
This is a cool write-up! I'm curious how much/if you Zvi's COVID round-ups you take into account? I wasn't around LessWrong during COVID, but, if I understand correctly, those played a large role in the information flow during that time.
2
Yarrow Bouchard 🔸
I haven't looked into it, but any and all new information that can give a fuller picture is welcome.
3
parconley
Yeah! This is the series that I am referring to: https://www.lesswrong.com/s/rencyawwfr4rfwt5C.
As I understand it, Zvi was quite ahead of the curve with COVID and moved out of New York before others. I could be wrong, though.
2
Yarrow Bouchard 🔸
The first post listed there is from March 2, 2020, so that's relatively late in the timeline we're considering, no? That's 3 days later than the February 28 post I discussed above as the first/best candidate for a truly urgent early warning about covid-19 on LessWrong. (2020 was a leap year, so there was a February 29.)
That first post from March 2 also seems fairly simple and not particularly different from the February 28 post (which it cites).
2
Yarrow Bouchard 🔸
Following up a bit on this, @parconley. The second post in Zvi's covid-19 series is from 6pm Eastern on March 13, 2020. Let's remember where this is in the timeline. From my quick take above:
Zvi's post from March 13, 2020 at 6pm is about all the school closures that happened that day. (The U.S. state of emergency was declared that morning.) It doesn't make any specific claims or predictions about the spread of the novel coronavirus, or anything else that could be assessed in terms of its prescience. It mostly focuses on the topic of the social functions that schools play (particularly in the United States and in the state of New York specifically) other than teaching children, such as providing free meals and supervision.
This is too late into the timeline to count as calling the pandemic early, and the post doesn't make any predictions anyway.
The third post from Zvi is on March 17, 2020 and it's mostly a personal blog. There are a few relevant bits. For one, Zvi admits he was surprised at how bad the pandemic was at that point:
He argues New York City is not locking down soon enough and San Francisco is not locking down completely enough. About San Francisco, one thing he says is:
I don't know how sound this was given what experts knew at the time. It might have been the right call. I don't know. I will just say that, in retrospect, it seems like going outside was one of the things we originally thought wasn't fine that we later thought was actually fine after all.
The next post after that isn't until April 1, 2020. It's about the viral load of covid-19 infections and the question of how much viral load matters. By this point, we're getting into questions about the unfolding of the ongoing pandemic, rather than questions about predicting the pandemic in advance. You could potentially go and assess that prediction track record separately, but that's beyond the scope of my quick take, which was to assess whether LessWrong called covid early.
Overall, Zvi's
2
Yarrow Bouchard 🔸
I spun this quick take out as a full post here. When I submitted the full post, there was no/almost no engagement on this quick take. In the future, I'll try to make sure to publish things only as a quick take or only as a full post, but not both. This was a fluke under unusual circumstances.
Feel free to continue commenting here, cross-post comments from here onto the full post, make new comments on the post, or do whatever you want. Thanks to everyone who engaged and left interesting comments.
Rate limiting on the EA Forum is too strict. Given that people karma downvote because of disagreement, rather than because of quality or civility — or they judge quality and/or civility largely on the basis of what they agree or disagree with — there is a huge disincentive against expressing unpopular or controversial opinions (relative to the views of active EA Forum users, not necessarily relative to the general public or relevant expert communities) on certain topics.
This is a message I saw recently:
You aren't just rate limited for 24 hours once you fall below the recent karma threshold (which can be triggered by one comment that is unpopular with a handful of people), you're rate limited for as many days as it takes you to gain 25 net karma on new comments — which might take a while, since you can only leave one comment per day, and, also, people might keep downvoting your unpopular comment. (Unless you delete it — which I think I've seen happen, but I won't do, myself, because I'd rather be rate limited than self-censor.)
The rate limiting system is a brilliant idea for new users or users who have less than 50 total karma — the ones who have little plant icons next to their nam... (read more)
I think this highlights why some necessary design features of the karma system don't translate well to a system that imposes soft suspensions on users. (To be clear, I find a one-comment-per-day limit based on the past 20 comments/posts to cross the line into soft suspension territory; I do not suggest that rate limits are inherently soft suspensions.)
I wrote a few days ago about why karma votes need to be anonymous and shouldn't (at least generally) require the voter to explain their reasoning; the votes suggested general agreement on those points. But a soft suspension of an established user is a different animal, and requires greater safeguards to protect both the user and the openness of the Forum to alternative views.
I should emphasize that I don't know who cast the downvotes that led to Yarrow's soft suspension (which were on this post about MIRI), or why they cast their votes. I also don't follow MIRI's work carefully enough to have a clear opinion on the merits of any individual vote through the lights of the ordinary purposes of karma. So I do not intend to imply dodgy conduct by anyone. But: "Justice must not only be done, but must also be seen to be done." People who are... (read more)
Assorted thoughts
* Rate limits should not apply to comments on your own quick takes
* Rate limits could maybe not count negative karma below -10 or so, it seems much better to rate limit someone only when they have multiple downvoted comments
* 2.4:1 is not a very high karma:submission ratio. I have 10:1 even if you exclude the april fool's day posts, though that could be because I have more popular opinions, which means that I could double my comment rate and get -1 karma on the extras and still be at 3.5
* if I were Yarrow I would contextualize more or use more friendly phrasing or something, and also not be bothered too much by single downvotes
* From scanning the linked comments I think that downvoters often think the comment in question has bad reasoning and detracts from effective discussion, not just that they disagree
* Deliberately not opining on the echo chamber question
2
Yarrow Bouchard 🔸
Can you explain what you mean by "contextualizing more"? (What a curiously recursive question...)
You definitely have more popular opinions (among the EA Forum audience), and also you seem to court controversy less, i.e. a lot of your posts are about topics that aren't controversial on the EA Forum. For example, if you were to make a pseudonymous account and write posts/comments arguing that near-term AGI is highly unlikely, I think you would definitely get a much lower karma to submission ratio, even if you put just as much effort and care into them as the posts/comments you've written on the forum so far. Do you think it wouldn't turn out that way?
I've been downvoted on things that are clearly correct, e.g. the standard definitions of terms in machine learning (which anyone can Google); a methodological error that the Forecasting Research Institute later acknowledged was correct and revised their research to reflect. In other cases, the claims are controversial, but they are also claims where prominent AI experts like Andrej Karpathy, Yann LeCun, or Ilya Sutskever have said exactly the same thing as I said — and, indeed, in some cases I'm literally citing them — and it would be wild to think these sort of claims are below the quality threshold for the EA Forum. I think that should make you question whether downvotes are a reliable guide to the quality of contributions.
One-off instances of one person downvoting don't bother me that much — that literally doesn't matter, as long as it really is one-off — what bothers me is the pattern. It isn't just with my posts/comments, either, it's across the board on the forum. I see it all the time with other contributors as well. I feel uneasy dragging those people into this discussion without their permission — it's easier to talk about myself — but this is an overall pattern.
Whether reasoning is good or bad is always bound to be controversial when debating about topics that are controversial, about which there is a lo
2
Thomas Kwa🔹
I mean it in this sense; making people think you're not part of the outgroup and don't have objectionable beliefs related to the ones you actually hold, in whatever way is sensible and honest.
Maybe LW is better at using disagreement button as I find it's pretty common for unpopular opinions to get lots of upvotes and disagree votes. One could use the API to see if the correlations are different there.
2
Yarrow Bouchard 🔸
Huh? Why would it matter whether or not I'm part of "the outgroup"...? What does that mean?
6
Thomas Kwa🔹
I think this is a significant reason why people downvote some, but not all, things they disagree with. Especially a member of the outgroup who makes arguments EAs have refuted before and need to reexplain, not saying it's actually you
2
Yarrow Bouchard 🔸
What is "the outgroup"?
6
Thomas Kwa🔹
Claude thinks possible outgroups include the following, which is similar to what I had in mind
2
Yarrow Bouchard 🔸
a) I’m not sure all of those count as someone who would necessarily be an outsider to EA (e.g. Will MacAskill only assigns a 50% probability to consequentialism being correct, and he and others in EA have long emphasized pluralism about normative ethical theories; there’s been an EA system change group on Facebook since 2015 and discourse around systemic change has been happening in EA since before then)
b) Even if you do consider people in all those categories to be outsiders to EA or part of "the out-group", us/them or in-group/out-group thinking seems like a bad idea, possibly leading to insularity, incuriosity, and overconfidence in wrong views
c) It’s especially a bad idea to not only think in in-group/out-group terms and seek to shut down perspectives of "the out-group" but also to cast suspicion on the in-group/out-group status of anyone in an EA context who you happen to disagree with about something, even something minor — that seems like a morally, subculturally, and epistemically bankrupt approach
You're shooting the messenger. I'm not advocating for downvoting posts that smell of "the outgroup", just saying that this happens in most communities that are centered around an ideological or even methodological framework. It's a way you can be downvoted while still being correct, especially from the LEAST thoughtful 25% of EA forum voters
Please read the quote from Claude more carefully. MacAskill is not an "anti-utilitarian" who thinks consequentialism is "fundamentally misguided", he's the moral uncertainty guy. The moral parliament usually recommends actions similar to consequentialism with side constraints in practice.
I probably won't engage more with this conversation.
I don't know what he meant, but my guess FWIW is this 2014 essay.
0
Yarrow Bouchard 🔸
I understand the general concept of ingroup/outgroup, but what specifically does that mean in this context?
2
Mo Putera
I don't know, sorry. I admittedly tend to steer clear of community debates as they make me sad, probably shouldn't have commented in the first place...
5
NickLaing
I've really appreciated comments and reflections from @Yarrow Bouchard 🔸 and I think in his case at least this does feel a bit unfair. Its good to encourage new people on the forum, unless they are posting particularly egrarious thing which I don't think he has been.
If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI?
I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.
Scaling running out of steam actually means three things:
2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.
3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.
So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due t... (read more)
The economist Tyler Cowen linked to my post on self-driving cars, so it ended up getting a lot more readers than I ever expected. I hope that more people now realize, at the very least, self-driving cars are not an uncontroversial, uncomplicated AI success story. In discussions around AGI, people often say things along the lines of: ‘deep learning solved self-driving cars, so surely it will be able to solve many other problems'. In fact, the lesson to draw is the opposite: self-driving is too hard a problem for the current cutting edge in deep learning (and deep reinforcement learning), and this should make us think twice before cavalierly proclaiming that deep learning will soon be able to master even more complex, more difficult tasks than driving.
Since my days of reading William Easterly's Aid Watch blog back in the late 2000s and early 2010s, I've always thought it was a matter of both justice and efficacy to have people from globally poor countries in leadership positions at organizations working on global poverty. All else being equal, a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from Canada with an equal level of education, an equal ability to network with the right international organizations, etc.
In practice, this is probably hard to do, since it requires crossing language barriers, cultural barriers, geographical distance, and international borders. But I think it's worth it.
So much of what effective altruism does, including around global poverty, including around the most evidence-based and quantitative work on global poverty, relies on people's intuitions, and people's intuitions formed from living in wealthy, Western countries with no connection to or experience of a globally poor country are going to be less accurate than people who have lived in poor countries and know a lot about them.
Simply put, first-hand experience of poor countries is a form of expertise and organizations run by people with that expertise are probably going to be a lot more competent at helping globally poor people than ones that aren't.
I agree with most of you say here, indeed all things being equal a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from anywhere else. The problem is your caveats - things are almost never equal...
1) Education systems just aren't nearly as good in lower income countries. This means that that education is sadly barely ever equal. Even between low income countries - a Kenyan once joked with me that "a Ugandan degree holder is like a Kenyan high school leaver". If you look at the top echelon of NGO/Charity leaders from low-income who's charities have grown and scaled big, most have been at least partially educated in richer countries
2) Ability to network is sadly usually so so much higher if you're from a higher income country. Social capital is real and insanely important. If you look at the very biggest NGOs, most of them are founded not just by Westerners, but by IVY LEAGUE OR OXBRIDGE EDUCATED WESTERNERS. Paul Farmer (Partners in Health) from Harvard, Raj Panjabi (LastMile Health) from Harvard. Paul Niehaus (GiveDirectly) from Harvard. Rob Mathers (AMF) Harvard AND Cambridge. With those connections you ca... (read more)
I just want to point out that I have a degree in philosophy and have never heard the word "epistemics" used in the context of academic philosophy. The word used has always been either epistemology or epistemic as adjective in front of a noun (never on its own, always used as an adjective, not a noun, and certainly never pluralized).
From what I can tell, "epistemics" seems to be weird EA Forum/LessWrong jargon. Not sure how or why this came about, since this is not obscure philosophy knowledge, nor is it hard to look up.
I agree this is just a unique rationalist use. Same with 'agentic' though that has possibly crossed over into the more mainstream, at least in tech-y discourse.
However I think this is often fine, especially because 'epistemics' sounds better than 'epistemic practices' and means something distinct from 'epistemology' (the study of knowledge).
Always good to be aware you are using jargon though!
There’s no accounting for taste, but 'epistemics' sounds worse to my ear than 'epistemic practices' because the clunky jargoniness of 'epistemics' is just so evident. It’s as if people said 'democratics' instead of 'democracy', or 'biologics' instead of 'biology'.
I also don’t know for sure what 'epistemics' means. I’m just inferring that from its use and assuming it means 'epistemic practices', or something close to that.
'Epistemology' is unfortunately a bit ambiguous and primarily connotes the subfield of philosophy rather than anything you do in practice, but I think it would also be an acceptable and standard use to talk about 'epistemology' as what one does in practice, e.g., 'scientific epistemology' or 'EA epistemology'. It’s a bit similar to 'ethics' in this regard, which is both an abstract field of study and something one does in practice, although the default interpretation of 'epistemology' is the field, not the practice, and for 'ethics' it’s the reverse.
It’s neither here nor there, but I think talking about personal 'agency' (terminology that goes back decades, long predating the rationalist community) is far more elegant than talking about a person being 'agentic'. (For AI agents, it doesn’t matter.)
I find "epistemics" neat because it is shorter than "applied epistemology" and reminds me of "athletics" and the resulting (implied) focus on being more focused on practice. I don't think anyone ever explained what "epistemics" refers to, and I thought it was pretty self-explanatory from the similarity to "athletics".
I also disagree about the general notion that jargon specific to a community is necessarily bad, especially if that jargon has fewer syllables. Most subcultures, engineering disciplines, sciences invent words or abbreviations for more efficient communication, and while some of that may be due to trying to gatekeep, it's so universal that I'd be surprised if it doesn't carry value. There can be better and worse coinages of new terms, and three/four/five-letter abbreviations such as "TAI" or "PASTA" or "FLOP" or "ASARA" are worse than words like "epistemics" or "agentic".
I guess ethics makes the distinction between normative ethics and applied ethics. My understanding is that epistemology is not about practical techniques, and that one can make a distinction here (just like the distinction between "methodology" and "methods").
I tried to figure out if there's a pair of su... (read more)
Applied ethics is still ethical theory, it’s just that applied ethics is about specific ethical topics, e.g. vegetarianism, whereas normative ethics is about systems of ethics, e.g. utilitarianism. If you wanted to distinguish theory from practice and be absolutely clear, you’d have to say something like ethical practices.
I prefer to say epistemic practices rather than epistemics (which I dislike) or epistemology (which I like, but is more ambiguous).
I don’t think the analogy between epistemics and athletics is obvious, and I would be surprised if even 1% of the people who have ever used the term epistemics have made that connection before.
I am very wary of terms that are never defined or explained. It is easy for people to assume they know what they mean, that there’s a shared meaning everyone agrees on. I really don’t know what epistemics means and I’m only assuming it means epistemic practices.
I fear that there’s a realistic chance if I started to ask different people to define epistemics, we would quickly uncover that different people have different and incompatible definitions. For example, some people might think of it as epistemic practices and some people might think of it as epistemological theory.
I am more anti-jargon and anti-acronyms than a lot of people. Really common acronyms, like AI or LGBT, or acronyms where the acronym is far better known than the spelled-out version, like NASA or DVD, are, of course, absolutely fine. PASTA and ASARA are egregious.
I’m such an anti-acronym fanatic I even spell out artificial general intelligence (AGI) and large language model (LLM) whenever I use them for the first time in a post.
My biggest problem with jargon is that nobody knows what it means. The in-group who is supposed to know what it means also doesn’t know what it means. They think they do, but they’re just fooling themselves. Ask them probing questions, and they’ll start to disagree and fight about the definition. This isn’t always true,
People in effective altruism or adjacent to it should make some public predictions or forecasts about whether AI is in a bubble.
Since the timeline of any bubble is extremely hard to predict and isn’t the core issue, the time horizon for the bubble prediction could be quite long, say, 5 years. The point would not be to worry about the exact timeline but to get at the question of whether there is a bubble that will pop (say, before January 1, 2031).
For those who know more about forecasting than me, and especially for those who can think of good w... (read more)
My leading view is that there will be some sort of bubble pop, but with people still using genAI tools to some degree afterwards (like how people kept using the internet after the dot com burst).
Still major uncertainty on my part because I don't know much about financial markets, and am still highly uncertain about the level where AI progress fully stalls.
2
Yarrow Bouchard 🔸
I just realized the way this poll is set up is really confusing. You're currently at "50% 100% probability", which when you look at it on the number line looks like 75%. Not the best tool to use for such a poll, I guess!
2
Yarrow Bouchard 🔸
Oh, sure. People will keep using LLMs.
I don’t know exactly how you’d operationalize an AI bubble. If OpenAI were a public company, you could say its stock price goes down a certain amount. But private companies can control their own valuation (or the public perception of it) to a certain extent, e.g. by not raising more money so their last known valuation is still from their most recent funding round.
Many public companies like Microsoft, Google, and Nvidia are involved in the AI investment boom, so their stocks can be taken into consideration. You can also look at the level of investment and data centre construction.
I don’t think it would be that hard to come up with reasonable resolution criteria, it’s just that this is of course always a nitpicky thing with forecasting and I haven’t spent any time on it yet.
4
Benjamin M.
I'm not exactly sure about the operationalization of this question, but it seems like there's a bubble among small AI startups at the very least. The big players might be unaffected however? My evidence for this is some mix of not seeing a revenue pathway for a lot of these companies that wouldn't require a major pivot, few barriers to entry for larger players if their product becomes successful, and having met a few people who work in AI startups who claim to be optimistic about earnings and stuff but can't really back that up.
2
Yarrow Bouchard 🔸
I don't know much about small AI startups. The bigger AI companies have a problem because their valuations have increased so much and the level of investment they're making (e.g. into building datacentres) is reaching levels that feel unsustainable.
It's to the point where the AI investment, driven primarily by the large AI companies, has significant macroeconomic effects on the United States economy. The popping of an AI bubble could be followed by a U.S. recession.
However, it's a bit complicated, in that case, as to whether to say the popping of the bubble would have "caused" the recession, since there are a lot of factors, such as tariffs. Macroeconomics and financial markets are complicated and I know very little. I'm not nearly an expert.
I don't think small AI startups creating successful products and then large AI companies copying them and outcompeting them would count as a bubble. That sounds like the total of amount of revenue in the industry would be about the same as if the startups succeeded, it just would flow to the bigger companies instead.
The bubble question is about the industry as a whole.
1
Benjamin M.
I do think there's also a significant chance of a larger bubble, to be fair, affecting the big AI companies. But my instinct is that a sudden fall in investment into small startups and many of them going bankrupt would get called a bubble in the media, and that that investment wouldn't necessarily just go into the big companies.
2
niplav
I put 30% on this possiblility, maybe 35%. I don't have much more to say than "time horizons!", "look how useful they're becoming in my dayjob & personal life!", "look at the qualitative improvement over the last six years", "we only need to automate machine learning research, which isn't the hardest thing to automate".
Worlds in which we get a bubble pop are worlds in which we don't get a software intelligence explosion, and in which either useful products come too late for the investment to sustain itself or there's not really much many useful products after what we already have. (This is tied in with "are we getting TAI through the things LLMs make us/are able to do, without fundamental insights".
5
David Mathers🔸
I haven't done the sums myself, but do we know for sure that they can't make money without being all that useful, so long as a lot of people interact with them everyday?
Is Facebook "useful"? Not THAT much. Do people pay for it? No, it's free. Instagram is even less useful than Facebook which at least used to actually be good for organizing parties and pub nights. Does META make money? Yes. Does equally useless TikTok make money? I presume so, yes. I think tech companies are pretty expert in monetizing things that have no user fee, and aren't that helpful at work. There's already a massive user base for Chat-GPT etc. Maybe they can monetize it even without it being THAT useful. Or maybe the sums just don't work out for that, I'm not sure. But clearly the market thinks they will make money in expectation. That's a boring reason for rejecting "it's a bubble" claims and bubbles do happen, but beating the market in pricing shares genuinely is quite difficult I suspect.
Of course, there could also be a bubble even if SOME AI companies make a lot of money. That's what happened with the Dot.com bubble.
4
Yarrow Bouchard 🔸
This is an important point to consider. OpenAI is indeed exploring how to put ads on ChatGPT.
My main source of skepticism about this is that the marginal revenue from an online ad is extremely low, but that’s fine because the marginal cost of serving a webpage or loading a photo in an app or whatever is also extremely low. I don’t have a good sense of the actual numbers here, but since a GPT-5 query is considerably more expensive than serving a webpage, this could be a problem. (Also, that’s just the marginal cost. OpenAI, like other companies, also has to amortize all its fixed costs over all its sales, whether they’re ad sales or sales directly to consumers.)
It’s been rumoured/reported (not sure which) that OpenAI is planning to get ChatGPT to sell things to you directly. So, if you ask, "Hey, ChatGPT, what is the healthiest type of soda?", it will respond, "Why, a nice refreshing Coca‑Cola® Zero Sugar of course!" This seems horrible. That would probably drive some people off the platform, but, who knows, it might be a net financial gain.
There are other "useless" ways companies like OpenAI could try to drive usage and try to monetize either via ads or paid subscriptions. Maybe if OpenAI leaned heavily into the whole AI "boyfriends/girlfriends" thing that would somehow pay off — I’m skeptical, but we’ve got to consider all the possibilities here.
3
Yarrow Bouchard 🔸
What do you make of the fact that METR's time horizon graph and METR's study on AI coding assistants point in opposite directions? The graph says: exponential progress! Superhuman coders! AGI soon! Singularity! The study says: overhyped product category, useless tool, tricks people into thinking it helps them when it actually hurts them.
Pretty interesting, no?
3
niplav
Yep, I wouldn't have predicted that. I guess the standard retort is: Worst case! Existing large codebase! Experienced developers!
I know that there's software tools I use >once a week that wouldn't have existed without AI models. They're not very complicated, but they'd've been annoying to code up myself, and I wouldn't have done it. I wonder if there's a slowdown in less harsh scenarios, but it's probably not worth the value of information of running such a study.
I dunno. I've done a bunch of calibration practice[1], this feels like a 30%, I'm calling 30%. My probability went up recently, mostly because some subjectively judged capabilities that I was expecting didn't start showing up.
----------------------------------------
1. My metaculus calibration around 30% isn't great, I'm overconfident there, I'm trying to keep that in mind. My fatebook is slightly overconfident in that range, and who can tell with Manifold. ↩︎
2
Yarrow Bouchard 🔸
There’s a longer discussion of that oft-discussed METR time horizons graph that warrants a post of its own.
My problem with how people interpret the graph is that people slip quickly and wordlessly from step to step in a logical chain of inferences that I don’t think can be justified. The chain of inferences is something like:
AI model performance on a set of very limited benchmark tasks → AI model performance on software engineering in general → AI model performance on everything humans do
I don’t think these inferences are justifiable.
2
Yarrow Bouchard 🔸
I haven’t thought about my exact probability too hard yet, but for now I’ll just say 90% because that feels about right.
There are two philosophies on what the key to life is.
The first philosophy is that the key to life is separate yourself from the wretched masses of humanity by finding a special group of people that is above it all and becoming part of that group.
The second philosophy is that the key to life is to see the universal in your individual experience. And this means you are always stretching yourself to include more people, find connection with more people, show compassion and empathy to more people. But this is constantly uncomfortable because, again and again,... (read more)
I’m seeking second opinions on whether my contention in Edit #4 at the bottom of this post is correct or incorrect. See the edit at the bottom of the post for full details.
Brief info:
My contention is about the Forecasting Research Institute’s recent LEAP survey.
One of the headline results from the survey is about the probabilities the respondents assign to each of three scenarios.
However, the question uses an indirect framing — an intersubjective resolution or metaprediction framing.
Self-driving cars are not close to getting solved. Don’t take my word for it. Listen to Andrej Karpathy, the lead AI researcher responsible for the development of Tesla’s Full Self-Driving software from 2017 to 2022. (Karpathy also did two stints as a researcher at OpenAI, taught a deep learning course at Stanford, and coined the term "vibe coding".)
From Karpathy’s October 17, 2025 interview with Dwarkesh Patel:
Dwarkesh Patel01:42:55
You’ve talked about how you were at Tesla leading self-driving from 2017 to 2022. And you firsthand saw this progress from c
I’m taking a long-term, indefinite hiatus from the EA Forum.
I’ve written enough in posts, quick takes, and comments over the last two months to explain the deep frustrations I have with the effective altruist movement/community as it exists today. (For one, I think the AGI discourse is completely broken and far off-base. For another, I think people fail to be kind to others in ordinary, important ways.)
But the strongest reason for me to step away is that participating in the EA Forum is just too unpleasant. I’ve had fun writing stuff on the EA Forum. I tha... (read more)
What AI model does SummaryBot use? And does whoever runs SummaryBot use any special tricks on top of that model? It could just be bias, but SummaryBot seems better at summarizing stuff then GPT-5 Thinking, o3, or Gemini 2.5 Pro, so I'm wondering if it's a different model or maybe just good prompting or something else.
@Toby Tremlett🔹, are you SummaryBot's keeper? Or did you just manage its evil twin?
Hey! @Dane Valerie runs SummaryBot, maybe she'd like to comment.
2
Yarrow Bouchard 🔸
Thanks, Toby!
3
Dane Valerie
It used to run on Claude, but I’ve since moved it to a ChatGPT project using GPT-5. I update the system instructions quarterly based on feedback, which probably explains the difference you’re seeing. You can read more in this doc on posting SummaryBot comments.
2
Yarrow Bouchard 🔸
Thank you very much for the info! It's probably down to your prompting, then. Squeezing things into 6 bullet points might be just a helpful format for ChatGPT or for summaries (even human-written ones) in general. Maybe I will try that myself when I want to ask ChatGPT to summarize something.
I also think there's an element of "magic"/illusion to it, though, since I just noticed a couple mistakes SummaryBot made and now its powers seem less mysterious.
Here is the situation we're in with regard to near-term prospects for artificial general intelligence (AGI). This is why I'm extremely skeptical of predictions that we'll see AGI within 5 years.
-Current large language models (LLMs) have extremely limited capabilities. For example, they can't score above 5% on the ARC-AGI-2 benchmark, they can't automate any significant amount of human labour,[1] and they can only augment human productivity in minor ways in limited contexts.[2] They make ridiculous mistakes all the time, like saying somethin... (read more)
Have Will MacAskill, Nick Beckstead, or Holden Karnofsky responded to the reporting by Time that they were warned about Sam Bankman-Fried's behaviour years before the FTX collapse?
I typically don’t agree with much that Dwarkesh Patel, a popular podcaster, says about AI,[1] but his recent Substack post makes several incisive points, such as:
Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.
Yes, exactly. The idea of a non-AGI AI researcher inventing AGI is a skyhook. It’s pulling yourself up by your bootstraps, a b... (read more)
Slight update to the odds I’ve been giving to the creation of artificial general intelligence (AGI) before the end of 2032. I’ve been anchoring the numerical odds of this to the odds of a third-party candidate like Jill Stein or Gary Johnson winning a U.S. presidential election. That’s something I think is significantly more probable than AGI by the end of 2032. Previously, I’d been using 0.1% or 1 in 1,000 as the odds for this, but I was aware that these odds were probably rounded.
I took a bit of time to refine this. I found that in 2016, FiveThirtyEight ... (read more)
I don't think this sort of anchoring is a useful thing to do. There is no logical reason for third party presidency success and AGI success to be linked mathematically. It seems like the third party thing is based on much greater empirical grounding.
You linked them because your vague impression of the likelihood of one was roughly equal to the vague impression of the likliehood of the other: If your vague impression of the third party thing changes, it shouldn't change your opinion of the other thing. You think that AGI is 5 times less likely than you previously thought because you got more precise odds about one guy winning the presidency ten years ago?
My (perhaps controversial) view is that forecasting AGI is in the realm of speculation where quantification like this is more likely to obscure understanding than to help it.
2
Yarrow Bouchard 🔸
I don’t think AGI is five times less likely than I did a week ago, I realized the number I had been translating my qualitative, subjective intuition into was five times too high. I also didn’t change my qualitative, subjective intuition of the probability of a third-party candidate winning a U.S. presidential election. What changed was just the numerical estimate of that probability — from an arbitrarily rounded 0.1% figure to a still quasi-arbitrary but at least somewhat more rigorously derived 0.02%. The two outcomes remain logically disconnected.
I agree that forecasting AGI is an area where any sense of precision is an illusion. The level of irreducible uncertainty is incredibly high. As far as I’m aware, the research literature on forecasting long-term or major developments in technology has found that nobody (not forecasters and not experts in a field) can do it with any accuracy. With something as fundamentally novel as AGI, there is an interesting argument that it’s impossible, in principle, to predict, since the requisite knowledge to predict AGI includes the requisite knowledge to build it, which we don’t have — or at least I don't think we do.
The purpose of putting a number on it is to communicate a subjective and qualitative sense of probability in terms that are clear, that other people can understand. Otherwise, its hard to put things in perspective. You can use terms like extremely unlikely, but what does that mean? Is something that has a 5% chance of happening extremely unlikely? So, rolling a natural 20 is extremely unlikely? (There are guides to determining the meaning of such terms, but they rely on assigning numbers to the terms, so we’re back to square one.)
Something that works just as well is comparing the probability of one outcome to the probability of another outcome. So, just saying that the probability of near-term AGI is less than the probability of Jill Stein winning the next presidential election does the trick. I don’t know why I
6
MichaelDickens
I don't think this should be downvoted. It's a perfectly fine example of reasoning transparency. I happen to disagree, but the disagree-vote button is there for a reason.
1
Yarrow Bouchard 🔸
Thank you. Karma downvotes have ceased to mean anything to me.
People downvote for no discernible reason, at least not reasons that are obvious to me, nor that they explain. I'm left to surmise what the reasons might be, including (in some cases) possibly disagreement, pique, or spite.
Neutrally informative things get downvoted, factual/straightforward logical corrections get downvoted, respectful expressions of mainstream expert opinion get downvoted — everything, anything. The content is irrelevant and the tone/delivery is irrelevant. So, I've stopped interpreting downvotes as information.
4
MichaelDickens
What do you mean by this? What is it that you're 95% confident about?
2
Yarrow Bouchard 🔸
Maybe this is a misapplication of the concept of confidence intervals — math is not my strong suit, nor is forecasting, so let me know — but what I had in mind is that I'm forecasting a 0.00% to 0.02% probability range for AGI by the end of 2034, and that if I were to make 100 predictions of a similar kind, more than 95 of them would have the "correct" probability range (whatever that ends up meaning).
But now that I'm thinking about it more and doing a cursory search, I think with a range of probabilities for a given date (e.g. 0.00% to 0.02% by end of 2034) as opposed to a range of years (e.g. 5 to 20 years) or another definite quantity, the probability itself is supposed to represent all the uncertainty and the confidence interval is redundant.
As you can tell, I'm not a forecaster.
6
MichaelDickens
I kinda get what you're saying but I think this is double-counting in a weird way. A 0.01% probability means that if you make 10,000 predictions of that kind, then about one of them should come true. So your 95% confidence interval sounds like something like "20 times, I make 10,000 predictions that each have a probability between 0.00% and 0.02%; and 19 out of 20 times, about one out of the 10,000 predictions comes true."
You could reduce this to a single point probability. The math is a bit complicated but I think you'd end up with a point probability on the order of 0.001% (~10x lower than the original probability). But if I understand correctly, you aren't actually claiming to have a 0.001% credence.
I think there are other meaningful statements you could make. You could say something like, "I'm 95% confident that if I spend 10x longer studying this question, then I would end up with a probability between 0.00% and 0.02%."
2
Yarrow Bouchard 🔸
Yeah, I’m saying the probability is significantly less than 0.02% without saying exactly how much less — that’s much harder to pin down, and there are diminishing returns to exactitude here — so that means it’s a range from 0.00% to <0.02%. Or just <0.02%.
The simplest solution, and the correct/generally recommended solution, seems to be to simply express the probability, unqualified.
Yann LeCun (a Turing Award-winning pioneer of deep learning) leaving Meta AI — and probably, I would surmise, being nudged out by Mark Zuckerberg (or another senior Meta executive) — is a microcosm for everything wrong with AI research today.
LeCun is the rare researcher working on fundamental new ideas to push AI forward on a paradigm level. Zuckerberg et al. seem to be abandoning that kind of work to focus on a mad dash to AGI via LLMs, on the view that enough scaling and enough incremental engineering and R&D will push current LLMs all the way ... (read more)
LeCun is also probably one of the top people to have worsened the AI safety outlook this decade, and from that perspective perhaps his departure is a good thing for the survival of the world, and thus also Meta’s shareholders?
-4
Yarrow Bouchard 🔸
I couldn't disagree more strongly. LeCun makes strong points about AGI, AGI alignment, LLMs, and so on. He's most likely right. I think the probability of AGI by the end of 2032 is significantly less than 1 in 1,000 and the probability of LLMs scaling to AGI is even less than that. There's more explanation in a few of my posts. In order of importance: 1, 2, 3, 4, and 5.
The core ideas that Eliezer Yudkowsky, Nick Bostrom, and others came up with about AGI alignment/control/friendliness/safety were developed long before the deep learning revolution kicked off in 2012. Some of Yudkowsky's and Bostrom's key early writings about these topics are from as far back as the early 2000s. To quote Clara Collier writing in Asterisk:
So, regardless of the timeline of AGI, that's dubious.
LessWrong's intellectual approach has produced about half a dozen cults, but despite many years of effort, millions of dollars in funding, and the hard work of many people across various projects, and despite many advantages, such as connections that can open doors, it has produced nothing of objective, uncontroversial, externally confirmable intellectual, economic, scientific, technical, or social value. The perceived value of anything it has produced is solely dependent on whether you agree or disagree with its worldview — I disagree. LessWrong claims to have innovated a superior form of human thought, and yet has nothing to show for it. The only explanation that makes any sense is that they're wrong, and are just fooling themselves. Otherwise, to quote Eliezer Yudkowsky, they'd be "smiling from on top of a giant heap of utility."
Yudkowsky's and LessWrong's views on AGI are correctly seen by many experts, such as LeCun, as unserious and not credible, and, in turn, the typical LessWrong response to LeCun is unacceptably intellectually bad and doesn't understand his views on a basic level, let alone respond to them convincingly.
Why would any rational person take that seriously?
Just calling yourself rational doesn't make you more rational. In fact, hyping yourself up about how you and your in-group are more rational than other people is a recipe for being overconfidently wrong.
Getting ideas right takes humility and curiosity about what other people think. Some people pay lip service to the idea of being open to changing their mind, but then, in practice, it feels like they would rather die than admit they were wrong.
This is tied to the idea of humiliation. If disagreement is a humiliation contest, changing one's mind can fe... (read more)
Two pieces of evidence commonly cited for near-term AGI are AI 2027 and the METR time horizons graph. AI 2027 is open to multiple independent criticisms, one of which is its use of the METR time horizons graph to forecast near-term AGI or AI capabilities more generally. Using the METR graph to forecast near-term AGI or AI capabilities more generally is not supported by the data and methodology used to make the graph.
Two strong criticisms that apply specifically to the AI 2027 forecast are:
Reason: Turned into a full post: https://forum.effectivealtruism.org/posts/GgesGHQmnb6G63peB/microsoft-s-ceo-satya-nadella-says-he-doesn-t-believe-in-agi
Here are my rules of thumb for improving communication on the EA Forum and in similar spaces online:
- Say what you mean, as plainly as possible.
- Try to use words and expressions that a general audience would understand.
- Be more casual and less formal if you think that means more people are more likely to understand what you're trying to say.
- To illustrate abstract concepts, give examples.
- Where possible, try to let go of minor details that aren't important to the main point someone is trying to make. Everyone slightly misspeaks (or mis... writes?) all the time. Attempts to correct minor details often turn into time-consuming debates that ultimately have little importance. If you really want to correct a minor detail, do so politely, and acknowledge that you're engaging in nitpicking.
- When you don't understand what someone is trying to say, just say that. (And be polite.)
- Don't engage in passive-aggressiveness or code insults in jargon or formal language. If someone's behaviour is annoying you, tell them it's annoying you. (If you don't want to do that, then you probably shouldn't try to communicate the same idea in a coded or passive-aggressive way, either.)
- If you're using an uncommon word
... (read more)My three most recent posts on Substack are relevant to effective altruism:
I can’t discuss them on the EA Forum, but I’m happy to do so on Substack.
Hey Vasco:
It's likely we should invest in public mod comments again, as this case underlines, it's pretty unfair to Yarrow to have to explain it to their interlocutors.
As you might guess I strongly disagree with pretty much every aspect of Yarrow's framing of the rate-limit we applied to their account. I'll talk to other mods about whether we want to reinstate the public comments, and if so I'll share more detail on the Yarrow decision in said public comment.
A quick version is that we made this decision because of the way that Yarrow disagrees with people, not the fact that they do. Primarily because of repeatedly sneer-y and snarky comments[1], but also because of generally unproductive and attritional disagreements, for example, see this description I sent to Yarrow at the time of the rate-limit:
"There is a clear pattern in your comments — you seem to have particularly unproductive disagreements with other users, generally due to an overly literal interpretation on your part, or excess defensiveness. This is no great sin of yours, but it isn’t great for the quality of Forum discussion, especially when you are naturally so prolific."
This wasn't a decision taken lightly... (read more)
Again, this is another issue with the current lack of moderation transparency on the Forum. Misuse of the laugh react is seen as breaking forum norms, and (AFAIK) in almost every case has led to at least a moderator message.
Civility enforcement is a major part of what we do as moderators. It's inevitable that some borderline cases will remain on the site such that someone motivated to find them - as you were - will be able to.
Bit of a side-point but the amount of civility enforcement we do has been one of the reasons I've been reluctant to publicise mod actions. I appreciate being able to send someone a more casual message for butting up against guidelines rather than having to call them out publicly (which is a far more embarassing result of what can be a fairly minor infraction).
It's hard to respond to snark like this w... (read more)
I feel like the issue addressed in the Said Saga is somewhat the opposite of what it seems like Yarrow does. A paradigmatic example of the bad bahavior attributed to Said was posting extremely short comments, such as just saying "examples?". Its obviously extremely easy to type a one word comment, and the possibility of adding additional examples of something a poster describes isn't exactly a deep insight. The asymetry is that it is very easy for Said to post a comment like that but would take a lot of effort for the original poster to respond providing examples, giving rise to the complaint that Said wasn't willing to put in equal effort.
This doesn't seem to be tr... (read more)
Thank you so much for this. I really appreciate it. I’m grateful for how kind and thoughtful this comment is.
On the moderator time thing, I just want to say that Toby only ever sent me 3 messages regarding moderation on the EA Forum. There was an initial message (452 words) informing me of a temporary rate limit/soft ban. This was my first communication from any of the moderators. After I replied, there was a shorter follow-up message (200 words) where Toby mentioned he didn’t have time to provide examples of problematic comments. Then there was a third and final message (319 words) informing me the temporary rate limit/soft ban had been made permanent. That was the full extent of the communication.
That third and final comment was the first and only time Toby explained this reason for the ban, which he’s quoted in this thread a few times:
... (read more)Yeah, I'm trying to maintain openness to different possibilities on the time issue to an extent since I don't really know what happened. If I had to venture a guess (which could obviously be wrong), I'd say something like this:
Other forum users who got frustrated with your posts/comments reach out to the mod team privately, mod team has extended discussions amongst themselves, decides on the soft ban, and then reaches out to you to tell you. If this is what happened, I can imagine that it did take a reasonable amount of time and also its understandable that the mod team would want to incorporate feedback from users, but I would say this is a mistake on the part of the mod team if that's what happened. If you're talking here about the value of being able to reach out privately, wouldn't this be the time to do that, before going to the soft ban? If you're not making the decision lightly and discussing a lot and writing google docs, couldn't you copy some examples of problematic comments or posts from these documents fairly easily?
I think a prcess where there is a lot of back channelling has a siginificant risk of filter bubble/echo chamber issues as you mention, similar to what Habry... (read more)
For what it's worth, whilst I find arguing with Yarrow quite stressful, and I definitely don't always agree with her, and sometimes think she's a bit reluctant to concede, I don't think a ban was a good idea. Her comments are usually substantive, people who don't want to engage with them can just ignore, critical perspectives are always valuable etc. In some cases, she's actually spotted pretty important stuff that people had missed, i.e. that Waymo's still have humans in the loop, or a problem in a Forecasting Research Institute study that I cited during one our arguments.
Thank you, David, that’s very generous of you. I just want to say that you were, to me, definitely the best commenter I talked to on the EA Forum. Although it can also kinda be fun to spar a bit over intellectual topics, thinking back now, I worry whether I was too harsh with you at times. Maybe sparring can sometimes get too heated, and if I ever fail to treat someone with kindness and empathy and respect, that’s wrong and my fault.
I feel a bit guilty and regretful to hear you say you’ve found arguing with me stressful. I definitely don’t want to make someone feel stressed, and now I’m wondering what I could do differently to help people feel less stressed in the future. Now that I think about it, I also feel stressed during a lot of intellectual arguments, and I wonder why. It seems like it shouldn’t be be like that, and like something is going wrong.
What I’ve appreciated most about talking to you is your level of sincere curiosity. The topics we’ve discussed seem like live issues to you, where you’re genuinely trying to make up your mind and get the best information and argumentation, and not just argue to win. Conversely, you’ve often made me pause, get curious, and give things... (read more)
I do agree Vasco, but I don't endorse me spending much more time on this right now.
Maybe? I personally would be much more discouraged from posting by a public comment than a private message. However it'd be nice if we could have a live log somewhere where forum users could see that we dm'd an anonymous user.
Writing a lot in itself isn't the problem. It's that the writing contains difficult-to-dispel misunderstandings, and therefore creates an asymmetric effort ratio.
I used to feel so strongly about effective altruism. But my heart isn't in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.
But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk h... (read more)
I'd distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they've always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil's website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn't much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
My memory is a large number of people to the NL controversy seriously, and the original threads on it were long and full of hostile comments to NL, and only after someone posted a long piece in defence of NL did some sympathy shift back to them. But even then there are like 90-something to 30-something agree votes and 200 karma on Yarrow's comment saying NL still seem bad: https://forum.effectivealtruism.org/posts/H4DYehKLxZ5NpQdBC/nonlinear-s-evidence-debunking-false-and-misleading-claims?commentId=7YxPKCW3nCwWn2swb
I don't think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.
Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.
In January 2020, some stores sold out of face masks in several different cities in North America. (One example of many.) The oldest post on LessWrong tagged with "covid-19" is from well after this started happening. (I also searched the forum for posts containing "covid" or "coronavirus" and sorted by oldest. I couldn’t find an older post that was relevant.) The LessWrong post is written by a self-described "prepper" who strikes a cautious tone and, oddly, advises buying vitamins to boost the immune system. (This seems dubious, possibly pseudoscientific.) To me, that first post strikes a similarly ambivalent, cautious tone as many... (read more)
My gloss on this situation is:
YARROW: Boy, one would have to be a complete moron to think that COVID-19 would not be a big deal as late as Feb 28 2020, i.e. something that would imminently upend life-as-usual. At this point had China locked down long ago, and even Italy had started locking down. Cases in the USA were going up and up, especially when you correct for the (tiny) amount of testing they were doing. The prepper community had certainly noticed, and was out in force buying out masks and such. Many public health authorities were also sounding alarms. What kind of complete moron would not see what’s happening here? Why is lesswrong patting themselves on the back for noticing something so glaringly obvious?
MY REPLY: Yes!! Yes, this is true!! Yes, you would have to be a complete moron to not make this inference!! …But man, by that definition, there sure were an awful lot of complete morons around, i.e. most everyone. LessWrong deserves credit for rising WAY above the incredibly dismal standards set by the public-at-large in the English-speaking world, even if they didn’t particularly surpass the higher standards of many virologists, preppers, etc.
My personal experience: As som... (read more)
Thanks for collecting this timeline!
The version of the claim I have heard is not that LW was early to suggest that there might be a pandemic but rather that they were unusually willing to do something about it because they take small-probability high-impact events seriously. Eg. I suspect that you would say that Wei Dai was "late" because their comment came after the nyt article etc, but nonetheless they made 700% betting that covid would be a big deal.
I think it can be hard to remember just how much controversy there was at the time. E.g. you say of March 13, "By now, everyone knows it's a crisis" but sadly "everyone" did not include the California department of public health, who didn't issue stay at home orders for another week.
[I have a distinct memory of this because I told my girlfriend I couldn't see her anymore since she worked at the department of public health (!!) and was still getting a ton of exposure since the California public health department didn't think covid was that big of a deal.]
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidenti... (read more)
The Ezra Klein Show (one of my favourite podcasts) just released an episode with GiveWell CEO Elie Hassenfeld!
Rate limiting on the EA Forum is too strict. Given that people karma downvote because of disagreement, rather than because of quality or civility — or they judge quality and/or civility largely on the basis of what they agree or disagree with — there is a huge disincentive against expressing unpopular or controversial opinions (relative to the views of active EA Forum users, not necessarily relative to the general public or relevant expert communities) on certain topics.
This is a message I saw recently:
You aren't just rate limited for 24 hours once you fall below the recent karma threshold (which can be triggered by one comment that is unpopular with a handful of people), you're rate limited for as many days as it takes you to gain 25 net karma on new comments — which might take a while, since you can only leave one comment per day, and, also, people might keep downvoting your unpopular comment. (Unless you delete it — which I think I've seen happen, but I won't do, myself, because I'd rather be rate limited than self-censor.)
The rate limiting system is a brilliant idea for new users or users who have less than 50 total karma — the ones who have little plant icons next to their nam... (read more)
I think this highlights why some necessary design features of the karma system don't translate well to a system that imposes soft suspensions on users. (To be clear, I find a one-comment-per-day limit based on the past 20 comments/posts to cross the line into soft suspension territory; I do not suggest that rate limits are inherently soft suspensions.)
I wrote a few days ago about why karma votes need to be anonymous and shouldn't (at least generally) require the voter to explain their reasoning; the votes suggested general agreement on those points. But a soft suspension of an established user is a different animal, and requires greater safeguards to protect both the user and the openness of the Forum to alternative views.
I should emphasize that I don't know who cast the downvotes that led to Yarrow's soft suspension (which were on this post about MIRI), or why they cast their votes. I also don't follow MIRI's work carefully enough to have a clear opinion on the merits of any individual vote through the lights of the ordinary purposes of karma. So I do not intend to imply dodgy conduct by anyone. But: "Justice must not only be done, but must also be seen to be done." People who are... (read more)
I probably won't engage more with this conversation.
The NPR podcast Planet Money just released an episode on GiveWell.
If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI?
I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.
Scaling running out of steam actually means three things:
1) Each new 10x increase in compute is less practically or qualitatively valuable than previous 10x increases in compute.
2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.
3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.
So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due t... (read more)
The economist Tyler Cowen linked to my post on self-driving cars, so it ended up getting a lot more readers than I ever expected. I hope that more people now realize, at the very least, self-driving cars are not an uncontroversial, uncomplicated AI success story. In discussions around AGI, people often say things along the lines of: ‘deep learning solved self-driving cars, so surely it will be able to solve many other problems'. In fact, the lesson to draw is the opposite: self-driving is too hard a problem for the current cutting edge in deep learning (and deep reinforcement learning), and this should make us think twice before cavalierly proclaiming that deep learning will soon be able to master even more complex, more difficult tasks than driving.
Since my days of reading William Easterly's Aid Watch blog back in the late 2000s and early 2010s, I've always thought it was a matter of both justice and efficacy to have people from globally poor countries in leadership positions at organizations working on global poverty. All else being equal, a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from Canada with an equal level of education, an equal ability to network with the right international organizations, etc.
In practice, this is probably hard to do, since it requires crossing language barriers, cultural barriers, geographical distance, and international borders. But I think it's worth it.
So much of what effective altruism does, including around global poverty, including around the most evidence-based and quantitative work on global poverty, relies on people's intuitions, and people's intuitions formed from living in wealthy, Western countries with no connection to or experience of a globally poor country are going to be less accurate than people who have lived in poor countries and know a lot about them.
Simply put, first-hand experience of poor countries is a form of expertise and organizations run by people with that expertise are probably going to be a lot more competent at helping globally poor people than ones that aren't.
I agree with most of you say here, indeed all things being equal a person from Kenya is going to be far more effective at doing anti-poverty work in Kenya than someone from anywhere else. The problem is your caveats - things are almost never equal...
1) Education systems just aren't nearly as good in lower income countries. This means that that education is sadly barely ever equal. Even between low income countries - a Kenyan once joked with me that "a Ugandan degree holder is like a Kenyan high school leaver". If you look at the top echelon of NGO/Charity leaders from low-income who's charities have grown and scaled big, most have been at least partially educated in richer countries
2) Ability to network is sadly usually so so much higher if you're from a higher income country. Social capital is real and insanely important. If you look at the very biggest NGOs, most of them are founded not just by Westerners, but by IVY LEAGUE OR OXBRIDGE EDUCATED WESTERNERS. Paul Farmer (Partners in Health) from Harvard, Raj Panjabi (LastMile Health) from Harvard. Paul Niehaus (GiveDirectly) from Harvard. Rob Mathers (AMF) Harvard AND Cambridge. With those connections you ca... (read more)
A number of podcasts are doing a fundraiser for GiveDirectly: https://www.givedirectly.org/happinesslab2025/
Podcast about the fundraiser: https://pca.st/bbz3num9
I just want to point out that I have a degree in philosophy and have never heard the word "epistemics" used in the context of academic philosophy. The word used has always been either epistemology or epistemic as adjective in front of a noun (never on its own, always used as an adjective, not a noun, and certainly never pluralized).
From what I can tell, "epistemics" seems to be weird EA Forum/LessWrong jargon. Not sure how or why this came about, since this is not obscure philosophy knowledge, nor is it hard to look up.
If you Google "epistemics" phil... (read more)
I agree this is just a unique rationalist use. Same with 'agentic' though that has possibly crossed over into the more mainstream, at least in tech-y discourse.
However I think this is often fine, especially because 'epistemics' sounds better than 'epistemic practices' and means something distinct from 'epistemology' (the study of knowledge).
Always good to be aware you are using jargon though!
I find "epistemics" neat because it is shorter than "applied epistemology" and reminds me of "athletics" and the resulting (implied) focus on being more focused on practice. I don't think anyone ever explained what "epistemics" refers to, and I thought it was pretty self-explanatory from the similarity to "athletics".
I also disagree about the general notion that jargon specific to a community is necessarily bad, especially if that jargon has fewer syllables. Most subcultures, engineering disciplines, sciences invent words or abbreviations for more efficient communication, and while some of that may be due to trying to gatekeep, it's so universal that I'd be surprised if it doesn't carry value. There can be better and worse coinages of new terms, and three/four/five-letter abbreviations such as "TAI" or "PASTA" or "FLOP" or "ASARA" are worse than words like "epistemics" or "agentic".
I guess ethics makes the distinction between normative ethics and applied ethics. My understanding is that epistemology is not about practical techniques, and that one can make a distinction here (just like the distinction between "methodology" and "methods").
I tried to figure out if there's a pair of su... (read more)
People in effective altruism or adjacent to it should make some public predictions or forecasts about whether AI is in a bubble.
Since the timeline of any bubble is extremely hard to predict and isn’t the core issue, the time horizon for the bubble prediction could be quite long, say, 5 years. The point would not be to worry about the exact timeline but to get at the question of whether there is a bubble that will pop (say, before January 1, 2031).
For those who know more about forecasting than me, and especially for those who can think of good w... (read more)
There are two philosophies on what the key to life is.
The first philosophy is that the key to life is separate yourself from the wretched masses of humanity by finding a special group of people that is above it all and becoming part of that group.
The second philosophy is that the key to life is to see the universal in your individual experience. And this means you are always stretching yourself to include more people, find connection with more people, show compassion and empathy to more people. But this is constantly uncomfortable because, again and again,... (read more)
Your help requested:
I’m seeking second opinions on whether my contention in Edit #4 at the bottom of this post is correct or incorrect. See the edit at the bottom of the post for full details.
Brief info:
-
-
-
-
... (read more)My contention is about the Forecasting Research Institute’s recent LEAP survey.
One of the headline results from the survey is about the probabilities the respondents assign to each of three scenarios.
However, the question uses an indirect framing — an intersubjective resolution or metaprediction framing.
The specific phrasing of the question is q
Self-driving cars are not close to getting solved. Don’t take my word for it. Listen to Andrej Karpathy, the lead AI researcher responsible for the development of Tesla’s Full Self-Driving software from 2017 to 2022. (Karpathy also did two stints as a researcher at OpenAI, taught a deep learning course at Stanford, and coined the term "vibe coding".)
From Karpathy’s October 17, 2025 interview with Dwarkesh Patel:
... (read more)I’m taking a long-term, indefinite hiatus from the EA Forum.
I’ve written enough in posts, quick takes, and comments over the last two months to explain the deep frustrations I have with the effective altruist movement/community as it exists today. (For one, I think the AGI discourse is completely broken and far off-base. For another, I think people fail to be kind to others in ordinary, important ways.)
But the strongest reason for me to step away is that participating in the EA Forum is just too unpleasant. I’ve had fun writing stuff on the EA Forum. I tha... (read more)
What AI model does SummaryBot use? And does whoever runs SummaryBot use any special tricks on top of that model? It could just be bias, but SummaryBot seems better at summarizing stuff then GPT-5 Thinking, o3, or Gemini 2.5 Pro, so I'm wondering if it's a different model or maybe just good prompting or something else.
@Toby Tremlett🔹, are you SummaryBot's keeper? Or did you just manage its evil twin?
Here is the situation we're in with regard to near-term prospects for artificial general intelligence (AGI). This is why I'm extremely skeptical of predictions that we'll see AGI within 5 years.
-Current large language models (LLMs) have extremely limited capabilities. For example, they can't score above 5% on the ARC-AGI-2 benchmark, they can't automate any significant amount of human labour,[1] and they can only augment human productivity in minor ways in limited contexts.[2] They make ridiculous mistakes all the time, like saying somethin... (read more)
Have Will MacAskill, Nick Beckstead, or Holden Karnofsky responded to the reporting by Time that they were warned about Sam Bankman-Fried's behaviour years before the FTX collapse?
Will responded here.
I typically don’t agree with much that Dwarkesh Patel, a popular podcaster, says about AI,[1] but his recent Substack post makes several incisive points, such as:
Yes, exactly. The idea of a non-AGI AI researcher inventing AGI is a skyhook. It’s pulling yourself up by your bootstraps, a b... (read more)
Slight update to the odds I’ve been giving to the creation of artificial general intelligence (AGI) before the end of 2032. I’ve been anchoring the numerical odds of this to the odds of a third-party candidate like Jill Stein or Gary Johnson winning a U.S. presidential election. That’s something I think is significantly more probable than AGI by the end of 2032. Previously, I’d been using 0.1% or 1 in 1,000 as the odds for this, but I was aware that these odds were probably rounded.
I took a bit of time to refine this. I found that in 2016, FiveThirtyEight ... (read more)
Yann LeCun (a Turing Award-winning pioneer of deep learning) leaving Meta AI — and probably, I would surmise, being nudged out by Mark Zuckerberg (or another senior Meta executive) — is a microcosm for everything wrong with AI research today.
LeCun is the rare researcher working on fundamental new ideas to push AI forward on a paradigm level. Zuckerberg et al. seem to be abandoning that kind of work to focus on a mad dash to AGI via LLMs, on the view that enough scaling and enough incremental engineering and R&D will push current LLMs all the way ... (read more)
Just calling yourself rational doesn't make you more rational. In fact, hyping yourself up about how you and your in-group are more rational than other people is a recipe for being overconfidently wrong.
Getting ideas right takes humility and curiosity about what other people think. Some people pay lip service to the idea of being open to changing their mind, but then, in practice, it feels like they would rather die than admit they were wrong.
This is tied to the idea of humiliation. If disagreement is a humiliation contest, changing one's mind can fe... (read more)
[Adapted from this comment.]
Two pieces of evidence commonly cited for near-term AGI are AI 2027 and the METR time horizons graph. AI 2027 is open to multiple independent criticisms, one of which is its use of the METR time horizons graph to forecast near-term AGI or AI capabilities more generally. Using the METR graph to forecast near-term AGI or AI capabilities more generally is not supported by the data and methodology used to make the graph.
Two strong criticisms that apply specifically to the AI 2027 forecast are:
- It depends crucially on the subject
... (read more)