Lots of people in the Bay seem to be thinking about/preparing for/making funding decisions based on the idea that lots of philanthropy will be given to AIS/EA cause areas very soon (i.e. end of year-ish). I would love for someone to write the comprehensive steel man case against this, as I think it’s probably underrated (some reasons to think they won’t give the money/it won’t be as much as some assume. Happy to comment/ speak to whoever is interested in doing this.
This is a great idea, although this would be a big call for many of us to do who might hope that our org could get some of that funding. Even someone like myself who fires shots pretty freely might think twice. We wouldn't want to rule our organisation out by writing too strong a Steelman against this. Any steelman would involve arguing that previous promises from founders and employees don't carry much weight, which could be seen as a personal slight against someone who could conceivably give you money. Sometimes I wonder if I've already said too much!
@Marcus Abramovitch 🔸has written a bit on this he might have a comment here?
I have so much intended writing on my plate and, unfortunately, little time to do it.
In a nutshell, I think some employees are dedicated EAs who will give a lot of money to causes I deeply care about. Some of them have a binding agreement to donate a portion of their stock (and a match).
I don't trust the founders. There is no legal mechanism I know of that binds them. The base rate of people effectively giving away large sums of wealth, even when they said they would, is shockingly low. They have said nothing EA-related in years apart from distancing themselves.
That said, I agree that Anthropic has a much more EA culture, for whatever that is worth. A lot of their employees are genuine EAs.
Being in San Francisco already warps your view of the world. Being inside Anthropic's doors must do more. I expect them to probably be above-average, typical billionaires, but to have their views on the world be changed by their experiences of the last few years and years to come.
@NickLaing I don't need their money, I'll say what I want. I think too highly of some of their employees to believe that my saying something bad about them that I thought was true would cause them to not listen to things I say or donate money differently than they thought optimal. If I'm wrong, I will happily eat my words.
Wasn't Dario Amodei one of the earliest signers of the GWWC pledge? Long before Anthropic existed? That makes him quite atypical amongst very rich people right? Though for what it's worth I would probably bet against him giving most of his wealth away, even if he has pledge to do that.
As a first pass, I asked GPT Pro to consider and model this, and Codex to host it, with interactive BOTEC tools etc.
https://uj-ai-wealth-philanthropy-steelman.netlify.app/
I'm just looking through it now (I'll respond/adapt to hypothes.is comments). Let me know if this sort of thing is useful or annoying.
Some updates, mostly automated, based on my inspection and your comments https://uj-ai-wealth-philanthropy-steelman.netlify.app/
An important (and to me fairly open) related question is to what extent this ends up being action-guiding?
Eg if I lower my probability estimate of this materialising by 10 percentage points, how much will it affect resources I spend on helping to prepare for the possible outcome? Perhaps people here have thoughts on that - my impression is that working on improving the future opportunity set for such donations is relatively robustly good right now?
The EA Forum seems super dead, which is pretty bad. I think part of the reason is that much of the Forum gets cross-posted to LessWrong, so you get EA Forum content there plus everything else (especially AI stuff) — which means there's little reason to actually come here.
One take: don't cross-post to LessWrong. There are real benefits to having two intellectual communities that take the same kinds of ideas seriously but develop distinct cultures, emphases, and standards around them.
Other ways to help:
I keep seeing people use the word "dead" in this way and I don't get it. Today so far I see 8 posts and 2 shortforms. Dead would be 0 posts and 0 shortforms. Do you have some standard for how many posts the EA Forum should have per day?
In the online gaming community, people often say a game is "dead" to mean either "Has reduced activity from a previous peak" or, if sufficiently principled about it, "Has had declining activity for a significant period of time, with no expected boost in the near future".
So, if you mentally replace "dead" with "more inactive than it used to be" I think this confusion will resolve. The standard I expect Noah is using internally is "At least as much activity as I perceived the EA Forum to have in its most active period since I joined it". (But I am not Noah, and am guessing entirely from previous usage I've seen, not from knowledge of them specifically)
I also don't like this word being used in this way. "Dead" implies it's no longer worth engaging with, and for online games there is a very real risk of hyperstition - people don't want to invest in a dead game. "Dying" would be a better term, but even then that exaggerates it - it is possible to preserve a healthy game or a forum with a lower, but stable, level of activity for a long time. "Declining" would be the best term imo, to honestly describe what is happening, if that is indeed what's happening.
I think the other thing is that we've tucked away the "community section". People used to come for the drama and stay for well, partly drama and partly other stuff they saw. I think its fine if they come for the drama and read other stuff.
I think Substack isn't great for comments and the forum is much better for comments.
The drama was bad for my mood & alignment w/ EA. I am glad it is comparatively hidden.
I agree that the Forum is (fairly) dead; I don't read it much anymore, and feel somewhat sad about this. I also think a lot of energy has moved to Substack, and would guess that this is a stronger effect than LessWrong cross-posting.
"Super dead" is a bit of an exaggeration, but there is less activity than there should be. The main issue is pretty apparent to me though: there isn't sufficient cross-posting to the forum. There's tons of good stuff on substack, but also on various org websites and blogs.
I don't think other causes are nearly as important as this.
Very keen to hear suggestions of blogs to crosspost - I agree that getting more substack content on here seems important (we've done this a bit, but in a fairly ad hoc way)
From what I understand, Forum usage declined for a couple of years and has now stabilised.
My guess is that, since the Forum is mostly used by fairly engaged EAs, the post-FTX slowdown in top-of-funnel growth is showing up here on a lag. People don't usually arrive on the Forum cold; my impression is there's typically a runway of a year or two between first contact with EA and posting or commenting in earnest. So a recruitment hole in 2023–24 produces a Forum hole now, even if intake has since recovered.
If that's roughly right, two things should follow: the share of activity from accounts under ~12 months old will have dropped more than overall activity has, and Forum recovery will lag whatever recovery CEA is seeing at the top of the funnel by another year or two.
And then it's probably compounded by dispersion — as you point out, a lot of the conversation that would once have happened here has migrated to LW, X, or Substack.
Thanks Noah! I cosign your suggestions, but probably not your 'super dead' diagnosis.
I'd add:
Also FWIW - I don't see the LessWrong thing as the reason for the Forum seeming more dead, that isn't a new development.
PS- The Forum team will be on a retreat this week, and we are having a team day on Friday, so all comments on the state of the Forum/ suggestions and requests for what you want to see are massively appreciated and timely.
It is incorrect to characterize the EA forum as "intellectual".
The term "pseudo-intellectual" is more appropriate.
Dwarkesh (of the famed podcast) recently posted a call for new guest scouts. Given how influential his podcast is likely to be in shaping discourse around transformative AI (among other important things), this seems worth flagging and applying for (at least, for students or early career researchers in bio, AI, history, econ, math, physics, AI that have a few extra hours a week).
The role is remote, pays ~$100/hour, and expects ~5–10 hours/week. He’s looking for people who are deeply plugged into a field (e.g. grad students, postdocs, or practitioners) with high taste. Beyond scouting guests, the role also involves helping assemble curricula so he can rapidly get up to speed before interviews.
More details are in the blog post; link to apply (due Jan 23 at 11:59pm PST).
+1 I would love an EA to be working on this.
Why don’t EA chapters exist at very prestigious high schools (e.g., Stuyvesant, Exeter, etc.)?
It seems like a relatively low-cost intervention (especially compared to something like Atlas), and these schools produce unusually strong outcomes. There’s also probably less competition than at universities for building genuinely high-quality intellectual clubs (this could totally be wrong).
Without a strongly supportive faculty member, I feel like you would struggle to make a group last longer than 2 years, since the succession and turnover dynamics of uni groups would be amplified.
Could still be worthwhile even with the lack of sustainability.
Students for High Impact Charity was a project to support groups at high schools, but it had difficulty getting traction. It's hard to start a group from afar if you're not a student or faculty there.
Seems right, though plausibly there are some EA/EA-adjacent students at some of these schools.
I think it sounds like an exciting idea. In my role funding EA CB work over the years I've seen a few of these clubs, so there's not literally nothing, but it's true that it's much less common than at universities, and I'm not aware of EA groups at these specific high schools.
The answer to many questions of the form "why isn't there an EA group for XYZ" tends to be "no organizer / no one else working to make it happen" and I'm guessing that's the main answer here too.
FWIW I went to the best (or second best lol) high school in Chicago, Northside, and tbh the kids at these top city highschools are of comparable talent to the kids at northwestern, with a higher tail as well. More over everyone has way more time and can actually chew on the ideas of EA. There was a jewish org that sent an adult once a week with food and I pretty much went to all of them even tho i would barely even self identify as jewish because of the free food and somewhere to sit and chat about random stuff while I waited for basketball practice.
So yes I think it would be highly successful. But I think you would need adult actual staff to come at least every other week (as brian mentioned) and as far as I can tell EA is currently struggling pretty hard w/organizing capacity and it seems to be getting worse (in part because as I have said many times, we don't celebrate organizers enough and we focus the movement too much on intellectualism rather than coordination and organizing). So I kind of doubt there is a ton of capacity for this. But if there is it's a good idea. I'm happy to help you understand how you could implement this at CPS selective enrollment schools if you want to help do it yourself.
Tho lmk if you disagree.
I think the "most topics" thing is ambiguous. There are some topics on which mainstream experts tend to be correct and some on which they're wrong, and although expertise is valuable on topics experts think about, they might be wrong on most topics central to EA. [1] Do we really wish we deferred to the CEO of PETA on what animal welfare interventions are best? EAs built that field in the last 15 years far beyond what "experts" knew before.
In the real world, assuming we have more than five minutes to think about a question, we shouldn't "defer" to experts or immediately "embrace contrarian views", rather use their expertise and reject it when appropriate. Since this wasn't an option in the poll, my guess is many respondents just wrote how much they like being contrarian, and EAs have to often be contrarian on topics they think about so it came out in favor of contrarianism.
[1] Experts can be wrong because they don't think in probabilities, they have a lack of imagination, there are obvious political incentives to say one thing over another, and probably other reasons, and lots of the central EA questions don't have actual well-developed scientific fields around them, so many of the "experts" aren't people who have thought about similar questions in a truth-seeking way for many years
I agree with Yarrow's anti-'truth-seeking' sentiment here. That phrase seems to primarily serve as an epistemic deflection device indicating 'someone whose views I don't want to take seriously and don't want to justify not taking seriously'.
I agree we shouldn't defer to the CEO of PETA, but CEOs aren't - often by their own admission - subject matter experts so much as people who can move stuff forwards. In my book the set of actual experts is certainly murky, but includes academics, researchers, sometimes forecasters, sometimes technical workers - sometimes CEOs but only in particular cases - anyone who's spent several years researching the subject in question.
Sometimes, as you say, they don't exist, but in such cases we don't need to worry about deferring to them. When they do, it seems foolish to not to upweight their views relative to our own unless we've done the same, or unless we have very concrete reasons to think they're inept or systemically biased (and perhaps even then).
Yeah, while I think truth-seeking is a real thing I agree it's often hard to judge in practice and vulnerable to being a weasel word.
Basically I have two concerns with deferring to experts. First is that when the world lacks people with true subject matter expertise, whoever has the most prestige--maybe not CEOs but certainly mainstream researchers on slightly related questions-- will be seen as experts and we will need to worry about deferring to them.
Second, because EA topics are selected for being too weird/unpopular to attract mainstream attention/funding, I think a common pattern is that of the best interventions, some are already funded, some are recommended by mainstream experts and remain underfunded, and some are too weird for the mainstream. It's not really possible to find the "too weird" kind without forming an inside view. We can start out deferring to experts, but by the time we've spent enough resources investigating the question that you're at all confident in what to do, the deferral to experts is partially replaced with understanding the research yourself as well as the load-bearing assumptions and biases of the experts. The mainstream experts will always get some weight, but it diminishes as your views start to incorporate their models rather than their views (example that comes to mind is economists on whether AGI will create explosive growth, and how recently good economic models have been developed by EA sources, now including some economists that vary assumptions and justify differences from the mainstream economists' assumptions).
Wish I could give more concrete examples but I'm a bit swamped at work right now.
What’s the definition of "truth-seeking"? Not your personal definition, but the pre-existing, canonical definition that’s been written down somewhere and that everyone agrees on.
Not "everyone agrees" what "utilitarianism" means either and it remains a useful word. In context you can tell I mean someone whose attitude, methods and incentives allow them to avoid the biases I listed and others.
If I want to know what “utilitarianism” means, including any disagreements among scholars about the meaning of the term (I have a philosophy degree, I have studied ethics, and I don’t have the impression there are meaningful disagreements among philosophers on the definition of “utilitarianism”), I can find this information in many places, such as:
So, it’s easy for me to find out what “utilitarianism” means. There is no shortage of information about that.
Where do I go to find out what “truth-seeking” means? Even if some people disagree on the definition, can I go somewhere and read about, say, the top 3 most popular definitions of the term and why people prefer one definition over the other?
It seems like an important word. I notice people keep using it. So, what does it mean? Where has it been defined? Is there a source you can cite that attempts to define it?
I have tried to find a definition for “truth-seeking” before, more than once. I’ve asked what the definition is before, more than once. I don’t know if there is a definition. I don’t know if the term means anything definite and specific. I imagine it probably doesn’t have a clear definition or meaning, and that different people who say “truth-seeking” mean different things when they say it — and so people are largely talking past each other when they use this term.
Incidentally, I think what I just said about “truth-seeking” probably also largely applies to “epistemics”. I suspect “epistemics” probably either means epistemic practices or epistemology, but it’s not clear, and there is evidently some confusion on its intended meaning. Looking at the actual use of “epistemics”, I’m not sure different people mean the same thing by it.
Probably(?) big news on PEPFAR (title: White House agrees to exempt PEPFAR from cuts): https://thehill.com/homenews/senate/5402273-white-house-accepts-pepfar-exemption/. (Credit to Marginal Revolution for bringing this to my attention)
MrBeast just released a video about “saving 1,000 animals”—a well-intentioned but inefficient intervention (e.g. shooting vaccines at giraffes from a helicopter, relocating wild rhinos before they fight each other to the death, covering bills for people to adopt rescue dogs from shelters, transporting lions via plane, and more). It’s great to see a creator of his scale engaging with animal welfare, but there’s a massive opportunity here to spotlight interventions that are orders of magnitude more impactful.
Given that he’s been in touch with people from GiveDirectly for past videos, does anyone know if there’s a line of contact to him or his team? A single video/mention highlighting effective animal charities—like those recommended by Animal Charity Evaluators (e.g. The Humane League, Faunalytics, Good Food Institute)—could reach tens of millions and (potentially) meaningfully shift public perception toward impact-focused giving for animals.
If anyone’s connected or has thoughts on how to coordinate outreach, this seems like a high-leverage opportunity I really have no idea how this sorta stuff works, but it seemed worth a quick take — feel free to lmk if I’m totally off base here).
Manifesting
Yooo - nice! Seems good and would cost under ~100k.
Agreed, Noah. For 15 k shrimps helped per $, it would cost 9.60 k$ (= 144*10^6/(15*10^3)).
Yep—Beast Philanthropy actually did an AMA here in the past! My takeaway was that the video comes first, so that your chances of a partnership would greatly increase if you can make it entertaining. This is somewhat in contrast with a lot of EA charities, which are quite boring, but I suspect on the margins you could find something good.
What IMHO worked for GiveDirectly in that video, and for Shrimp Welfare in their public outreach, has been the counterintuitiveness of some of these interventions. Wild animals, cultured meat, shrimp, are more likely to fit in this bucket than corporate campaigns for chickens I reckon.
As Huw says, the video comes first. I think this puts almost anything you'd be excited about off the table. Factory farming is a really aversive topic for people, and people are quite opposed to large scale WAS interventions. The intervention in the video he did make wasn't chosen at random. People like charismatic megafauna.
While I don't have the bandwidth for this atm, someone should make a public (or private for, say, policy/reputation reasons) list of people working in (one or multiple of) the very neglected cause areas — e.g., digital minds (this is a good start), insect welfare, space governance, AI-enabled coups, and even AI safety (more for the second reason than others). Optional but nice-to-have(s): notes on what they’re working on, time contributed, background, sub-area, and the rough rate of growth in the field (you probably don’t want to decide career moves purely on current headcounts). And remember: perfection is gonna be the enemy of the good here.
Coordination.
It’s surprisingly hard to know who’s in these niches (independent researchers, part-timers, new entrants, maybe donors). A simple list would make it easier to find collaborators, talk to the right people, and avoid duplicated work.
Neglectedness clarity.
A major reason to work on ultra-neglected causes is… neglectedness. But we often have no real headcount, and that may push people into (or out of) fields they wouldn’t otherwise choose. Even technical AI safety numbers are outdated — the last widely cited 80k estimate (2022) was ~200 people, which is clearly very false now. (To their credit, they emphasized the difficulty and tried to update.)
Even rough FTE (full time equivalent) estimates + who’s active in each area would be a huge service for some fields.
Started something sorta similar about a month ago: https://saul-munn.notion.site/A-collection-of-content-resources-on-digital-minds-AI-welfare-29f667c7aef380949e4efec04b3637e9?pvs=74
One reason a comprehensive version of this would be difficult for insect welfare is that a couple of projects are 'undercover'. Rethink Priorities have guidance on donating to insects, shrimp and wild animals that might be relevant.
Separately, I understand @JordanStone has a pretty comprehensive sense of who's who in space governance, and would be a good person to contact if you're thinking about getting into this field.
Yeah, lists exist for all the people working on space governance from a longtermist perspective, and they tend to list about 10-15 people. I'm like 90% sure I know of everyone working on longtermist space governance, and I'd estimate that there are the equivalent of ~3 people working full time on this. There's not as much undercover work required for space governance, but I don't like to share lists of names publicly without permission.
At the moment, the main hub for space governance is Forethought and most people contact Fin Moorhouse to learn more about space governance as he's the author of the 80K problem profile on space governance and has been publishing work with Forethought on or related to space governance. From there, people tend to get a lay of the land, introductions are made, and newcomers will get a good idea of what people are working on and where they might be able to contribute.
What a wonderful idea! Mayank referred me over to this post, and I think EA at UIUC might have to hop on this project. I'll see about starting something in the next month or so and sharing a link to where I'm compiling things in case anyone else is interested in collaborating on this. Or, it's possible an initiative like it already exists that I'll stumble upon while investigating (though such a thing may well be outdated).
Here’s a random org/project idea: hire full-time, thoughtful EA/AIS red teamers whose job is to seriously critique parts of the ecosystem — whether that’s the importance of certain interventions, movement culture, or philosophical assumptions. Think engaging with critics or adjacent thinkers (e.g., David Thorstad, Titotal, Tyler Cowen) and translating strong outside critiques into actionable internal feedback.
The key design feature would be incentives: instead of paying for generic criticism, red teamers receive rolling “finder’s fees” for critiques that are judged to be high-quality, good-faith, and decision-relevant (e.g., identifying strategic blind spots, diagnosing vibe shifts that can be corrected, or clarifying philosophical cruxes that affect priorities).
Part of why I think this is important is because I generally think have the intuition that the marginal thoughtful contrarian is often more valuable than the marginal agreer, yet most movement funding and prestige flows toward builders rather than structured internal critics. If that’s true, a standing red-team org — or at least a permanent prize mechanism — could be unusually cost-effective.
There have been episodic versions of this (e.g., red-teaming contests, some longtermist critiquing stuff), but I’m not sure why this should come in waves rather than exist as ongoing infrastructure (org or just some prize pool that's always open for sufficiently good criticisms).
While I like the potential incentive alignment, I suspect finder’s fees are unworkable. It’s much easier to promise impartiality and fairness in a single game as opposed to an iterated one, and I suspect participants relying on the fees for income would become very sensitive to the nuances of previous decisions rather than the ultimate value of their critiques.
Ultimately, I don’t think there are many shortcuts in changing the philosophy of a movement. If something is worth challenging, than people strongly believe it and there will have to be a process of contested diffusion from the outside in. You can encourage this in individual cases, but systemizing it seems difficult.
Very random but:
If anyone is looking for a name for a nuclear risk reduction/ x-risk prevention org, consider (The) Petrov Institute. It's catchy, symbolic, and sounds like it has prestige.
Unfortunately it also sounds Russian, which has some serious downsides at the moment....
Perhaps this downside could be partly mitigated by expanding the name to make it sound more global or include something Western, for example: Petrov Center for Global Security or Petrov–Perry Institute (in reference to William J. Perry). (Not saying these are the best names.)
For me at least, that implies an institute founded or affiliated with somebody named Petrov, not just inspired by somebody, and it would seem slightly sketchy for it not to be.
The Fallacy of Similar Magnitudes
A name I came up with for a pattern that comes up often enough to deserve one. Here’s a particular instance:
Suppose you're a longtermist who thinks the vast majority of the (net positive) moral weight is in the far future, that AI existential risk is above 1%, and that it's reasonably tractable to reduce. Some people who hold all three premises still say things like: "AI safety has maybe two orders of magnitude more resources than animal welfare, so the marginal dollar is better spent on animal welfare."
This is the fallacy of similar magnitudes: in this case, the application is that neglectedness comparisons only matter when the causes are within a couple orders of magnitude of each other in stakes. Under the stated premises, they aren’t: the a priori difference in expected value is way larger. Two orders of magnitude in resources doesn't move the needle even a bit.
Note: this AW response could be ok if you have certain views about risk aversion or worldview diversification, but that’s besides the point.
The Forum should normalize public red-teaming for people considering new jobs, roles, or project ideas.
If someone is seriously thinking about a position, they should feel comfortable posting the key info — org, scope, uncertainties, concerns, arguments for — and explicitly inviting others to stress-test the decision. Some of the best red-teaming I’ve gotten hasn’t come from my closest collaborators (whose takes I can often predict), but from semi-random thoughtful EAs who notice failure modes I wouldn’t have caught alone (or people think pretty differently so can instantly spot things that would have taken me longer to figure out).
Right now, a lot of this only happens at EAGs or in private docs, which feels like an information bottleneck. If many thoughtful EAs are already reading the Forum, why not use it as a default venue for structured red-teaming?
Public red-teaming could:
Obviously there are tradeoffs — confidentiality, social risk, signaling concerns — but I’d be excited to see norms shift toward “post early, get red-teamed, iterate publicly,” rather than waiting for a handful of coffee chats.
I would love to see that normalised, I'm personally in the ideation stage for a new org, and have been considering asking the forum to help red team, but it's not something I've seen done all that much, so I've somewhat held off.
RoastMyPost by the Quantified Uncertainty Research Institute might be useful for your idea too if you have a gdoc proposal.
More EA in da news: https://x.com/DavidSacks/status/2034047505336295904
And the spicy CAIS take: https://x.com/cais/status/2034389842076025164?s=46
From the CAIS tweet:
We believe the effective altruism movement is, unfortunately, controlled opposition. The less influence it has on AI safety, the better.
I really don't like it when people paint the whole movement with one brush, but they're not wrong that there's a subset of AI safety/EA that behaves like controlled opposition. Obvious example is Anthropic/Dario (who was one of the first GWWC signatories); Good Ventures basically doesn't fund anything that might be bad for AI company stock prices; I can think of some other possible examples but I don't want to denigrate anyone in case I'm wrong.
It's definitely not true that the whole movement is controlled opposition. Here I am, an EA, loudly advocating for a pause on AI development. There are plenty of us low-level EAs who oppose AI companies, but we don't control funding.
This might feel obvious, but I think it's under-appreciated how much disagreement on AI progress just comes down to priors (in a pretty specific way) rather than object-level reasoning.
I was recently arguing the case for shorter timelines to a friend who leans longer. We kept disagreeing on a surprising number of object-level claims, which was weird because we usually agree more on the kinda stuff we were arguing about.
Then I basically realized what I think was going on: she had a pretty strong prior against what I was saying, and that prior is abstract enough that there's no clear mechanism by which I can push against it. So whenever I made a good object-level case, she'd just take the other side — not necessarily because her reasons were better all else equal, but because the prior was doing the work underneath without either of us really knowing it.
There's something clearly rational here that's kinda unintuitive to get a grip on. If you have a strong prior, and someone makes a persuasive argument against it, but you can't identify the specific mechanism by which their argument defeats it, you should probably update that the arguments against their case are better than they appear, even if you can't articulate them yet. From the outside, this totally just looks like motivated reasoning (and often is), but I think it can be pretty importantly different.
The reason this is so hard to disentangle is that (unless your belief web is extremely clear to you, which seems practically impossible) it's just enormously complicated. Your prior on timelines isn't an isolate thing — it's load-bearing for a bunch of downstream beliefs all at once. So the resistance isn't obviously irrational, it's more like... the system protecting its own coherence.
I think this means that people should try their best to disentangle whether some object level argument they’re having comes from real object level beliefs or pretty abstract priors (in which case, it seems less worthwhile to press on them).
My go-to diagram for illustrating your point, from (who else?) Scott Alexander's varieties of argumentative experience:
[Graham’s hierarchy of disagreements] is useful for its intended purpose, but it isn’t really a hierarchy of disagreements. It’s a hierarchy of types of response, within a disagreement. Sometimes things are refutations of other people’s points, but the points should never have been made at all, and refuting them doesn’t help. Sometimes it’s unclear how the argument even connects to the sorts of things that in principle could be proven or refuted.
If we were to classify disagreements themselves – talk about what people are doing when they’re even having an argument – I think it would look something like this:
Most people are either meta-debating – debating whether some parties in the debate are violating norms – or they’re just shaming, trying to push one side of the debate outside the bounds of respectability.
If you can get past that level, you end up discussing facts (blue column on the left) and/or philosophizing about how the argument has to fit together before one side is “right” or “wrong” (red column on the right). Either of these can be anywhere from throwing out a one-line claim and adding “Checkmate, atheists” at the end of it, to cooperating with the other person to try to figure out exactly what considerations are relevant and which sources best resolve them.
If you can get past that level, you run into really high-level disagreements about overall moral systems, or which goods are more valuable than others, or what “freedom” means, or stuff like that. These are basically unresolvable with anything less than a lifetime of philosophical work, but they usually allow mutual understanding and respect.
More on the high-level generators of disagreement (emphasis mine, other than 1st sentence):
High-level generators of disagreement are what remains when everyone understands exactly what’s being argued, and agrees on what all the evidence says, but have vague and hard-to-define reasons for disagreeing anyway. In retrospect, these are probably why the disagreement arose in the first place, with a lot of the more specific points being downstream of them and kind of made-up justifications. These are almost impossible to resolve even in principle. ...
Some of these involve what social signal an action might send; for example, even a just war might have the subtle effect of legitimizing war in people’s minds. Others involve cases where we expect our information to be biased or our analysis to be inaccurate; for example, if past regulations that seemed good have gone wrong, we might expect the next one to go wrong even if we can’t think of arguments against it. Others involve differences in very vague and long-term predictions, like whether it’s reasonable to worry about the government descending into tyranny or anarchy. Others involve fundamentally different moral systems, like if it’s okay to kill someone for a greater good. And the most frustrating involve chaotic and uncomputable situations that have to be solved by metis or phronesis or similar-sounding Greek words, where different people’s Greek words give them different opinions.
You can always try debating these points further. But these sorts of high-level generators are usually formed from hundreds of different cases and can’t easily be simplified or disproven. Maybe the best you can do is share the situations that led to you having the generators you do. Sometimes good art can help.
The high-level generators of disagreement can sound a lot like really bad and stupid arguments from previous levels. “We just have fundamentally different values” can sound a lot like “You’re just an evil person”. “I’ve got a heuristic here based on a lot of other cases I’ve seen” can sound a lot like “I prefer anecdotal evidence to facts”. And “I don’t think we can trust explicit reasoning in an area as fraught as this” can sound a lot like “I hate logic and am going to do whatever my biases say”. If there’s a difference, I think it comes from having gone through all the previous steps – having confirmed that the other person knows as much as you might be intellectual equals who are both equally concerned about doing the moral thing – and realizing that both of you alike are controlled by high-level generators. High-level generators aren’t biases in the sense of mistakes. They’re the strategies everyone uses to guide themselves in uncertain situations.
(also related: Value Differences As Differently Crystallized Metaphysical Heuristics and the previous essays in that series)
Regarding your "something clearly rational here that's kinda unintuitive to get a grip on", I think of it as epistemic learned helplessness as a "social safety valve" to the downside risk of believing persuasive arguments that can (potentially catastrophically) harm the believer, cf. Reason as memetic immune disorder.
There's a lot more to the study of disagreement if you're keen, shame it's mostly just one person working on it and they're busy writing a book nowadays.
I think there are ways to get the info without threatening the coherence of your system. For example, you can try to understand, and then absorb into intuition, alternative root/basic intuitions. Cf. https://en.wikipedia.org/wiki/World_Hypotheses by Pepper, and Lakoff's ideas on metaphors. As a concrete example in the case of timelines, I would offer https://www.lesswrong.com/posts/wgqcExv9AgN5MuJuY/bioanchors-2-electric-bacilli as an intuition for longer / less confident timelines that you could try to understand intuitively (without necessarily believing). Having multiple of these hypotheses is one good way to make yourself more fluent in viewing multiple alternative states of the world as being intuitively plausible / worth investigating / asking for falsification.
Experts currently treat being persuaded as reasonably good evidence that something is true — their judgment is calibrated enough that when they find an argument convincing, that's correlated with the argument actually being correct. This allows them to update readily in light of new evidence, and is a big part of how intellectual progress happens: lots of innovation and advances in basically every subject come down to experts taking sometimes weird new ideas seriously.
One worry I have about superpersuasive AI is that it could erode this. If a superpersuasive AI can convince experts of things regardless of whether those things are true, experts may cease to see themselves being persuaded as good evidence that something is true — and start treating it the way laypeople do. Laypeople are typically hesitant to take on new, truth-tracking beliefs in light of new information, and (to some degree) rationally so: the fact that someone was able to convince a layperson of something is just not very strong evidence that it is in fact true. Experts might end up in the same position — only updating rarely, and in ways that are often unrelated to the truth.
This would be quite bad. If experts lose their capacity to reliably update on genuine evidence, we could significantly slow the rate of intellectual progress (which could be very important for making AI go well!). This is, I think, an underappreciated argument for caring about AI for epistemics — curious what others think.
I think this is technically true but irrelevant: if we have superpersuasive AI, then there won't be human experts anymore, because the AI will have more expertise than any human. Unless somehow the AI is superpersuasive while still having sub-human performance in most ways, which seems unlikely to me.
Looking back on old 80k podcasts, and this is what I see (lol):
Here is a claim I made to my friend yesterday that I think is true and believe people continually underrate: given reasonable assumptions about the influx of incoming capital, getting into (ie working in/thinking about) AI/ AI Safety early has probably been better for GHD or AW than lots (maybe all, tho I haven’t crunched the numbers) of direct work in GHD and AW.
I think this could even be true for EA CB (with much less confidence). AIS groups grow a lot faster than EA groups historically do and while the groups are often not that EA aligned, the leaders (who are, in my view, often very talented) are to various degrees. This is especially true for the ones who end up working full time in AIS: they often have proto-EA dispositions and these become intensified when they work around a bunch of EAs full time (though I can see two ways in which this is not as good as normal EAs: (1) being EA for social reasons and (2) being EA in stated but not revealed preference). TBC, I think having an EA group at a school make it much more likely that the leaders of the AIS group are aligned.
I can also see worlds (though def not my mainline at the moment) where the best thing one could have done for quality adjusted EA CB is actually just supporting AIS groups.
I don’t know how we should update/ by how much, but I think a reasonable way is the following: being ahead on some big emerging issue might just be more important than what seems like the obviously important thing that nobody does for bad reasons (ie GHD or AW).