Shower thought: A lot of the talking past each other that happens between vegan and non-vegan^[1] EAs^[2] might come from selection effects plus typical mind fallacy.^[3]

Let’s say there are two types of people: type-A, for whom a vegan diet imposes little or no costs, and type-B, for whom a vegan diet imposes substantial costs (in things like health, productivity,^[4] social life). My hunch is that most long-time vegans are type-A, while most type-B people who try going vegan bounce.

Now, to a type-A vegan who doesn’t realize type-B is a thing, those who claim that a vegan diet has costs are obviously lying. And to a type-B person who’s tried and failed to go vegan, vegans are obviously lying about how easy it is. (To put things crudely.)

I think an awareness from both sides of the vegan/non-vegan divide that the other type—B/A—exists, and that the other side is mostly made up of the other type, could go quite a long way toward circumventing frustrating or otherwise unproductive debates.

(I doubt this take is original to me, but I didn’t find it on this forum already with a quick search, so I figured I’d share.)

1. ^{^}

    I expect the same argument to apply to vegetarians versus non-vegetarians, but for brevity I’ll just write “vegan” throughout. (I’m avoiding the veg*n shorthand just because I don’t find *’s aesthetically appealing in prose.)

2. ^{^}

    I’ve specified “EAs” rather than just people in general because my argument works better if most non-vegans have actually tried to go vegan. My sense is that this is true for the EA population, but (very) not true for the general population.

3. ^{^}

    My preferred definition of typical mind fallacy: The mistake of jumping to conclusions—often subconsciously—about other people's experiences based on your own experiences. In other words, the mistake of assuming that other people are more like you than they really are. (Here’s the LessWrong Wiki’s definition.)

^{^}

 I’ve previously written about my experience here.

Show all footnotes

The Forum moderation team (which includes myself) is revisiting thinking about this forum’s norms. One thing we’ve noticed is that we’re unsure to what extent users are actually aware of the norms. (It’s all well and good writing up some great norms, but if users don’t follow them, then we have failed at our job.)

Our voting guidelines are of particular concern,^[1] hence this poll. We’d really appreciate you all taking part, especially if you don’t usually take part in polls but do take part in voting. (We worry that the ‘silent majority’ of our users—i.e., those who vote, and thus shape this forum’s incentive landscape, but don’t generally engage beyond voting—may be less in tune with our norms than our most visibly engaged users. Therefore, we would love to see this demographic represented in the poll above.)

Depending on the poll’s results, we may take action up to and including building new features into the forum’s UI, to help remind users of the guidelines.^[2]

For reference, the tl;dr version of our voting guidelines is pasted below. You can find the full version here.^[3]

1. ^{^}

   Firstly, these guidelines are kind of buried deep within our canonical ‘Guide to the norms’ post. Secondly, one doesn’t receive feedback in response to an ‘incorrect’ vote (i.e., a vote that’s not in line with our voting guidelines) in the same way one receives feedback to an incorrect post or comment (via downvotes and replies). And so, it’s possible to continue voting in the same incorrect way, oblivious to the fact that one is voting incorrectly.

2. ^{^}

   H/t @Ebenezer Dukakis for nudging us down this path of thinking.

3. ^{^}

   What I’ve been calling ‘guidelines’ in this quick take are technically ‘suggestions’ in our published voting norms as of right now. But this is something we are revisiting; we think ‘guidelines’ is more accurate. (We are similarly revisiting ‘rules’ versus ‘norms’—h/t @leillustrations and @richard_ngo for calling us out, here, and sorry it’s taken us so long to address the concern.)

‘Five Years After AGI’ Focus Week happening over at Metaculus.

Inspired in part by the EA Forum’s recent debate week, Metaculus is running a “focus week” this week, aimed at trying to make intellectual progress on the issue of “What will the world look like five years after AGI (assuming that humans are not extinct)^[1]?”

Leaders of AGI companies, while vocal about some things they anticipate in a post-AGI world (for example, bullishness in AGI making scientific advances), seem deliberately vague about other aspects. For example, power (will AGI companies have a lot of it? all of it?), whether some of the scientific advances might backfire (e.g., a vulnerable world scenario or a race-to-the-bottom digital minds takeoff), and how exactly AGI will be used for “the benefit of all.”

Forecasting questions for the week range from “Percentage living in poverty?” to “Nuclear deterrence undermined?” to “‘Long reflection’ underway?”

Those interested: head over here. You can participate by:

• Forecasting
• Commenting
  • Comments are especially valuable on long-term questions, because the forecasting community has less of a track record at these time scales.^[2][3]
• Writing questions
  • There may well be some gaps in the admin-created question set.^[4] We welcome question contributions from users.

The focus week will likely be followed by an essay contest, since a large part of the value in this initiative, we believe, lies in generating concrete stories for how the future might play out (and for what the inflection points might be). More details to come.^[5]

1. ^{^}

   This is not to say that we firmly believe extinction won’t happen. I personally put p(doom) at around 60%. At the same time, however, as I have previously written, I believe that more important trajectory changes lie ahead if humanity does manage to avoid extinction, and that it is worth planning for these things now.

2. ^{^}

   Moreover, I personally take Nuño Sempere’s “Hurdles of using forecasting as a tool for making sense of AI progress” piece seriously, especially the “Excellent forecasters and Superforecasters™ have an imperfect fit for long-term questions” part.

   With short-term questions on things like geopolitics, I think one should just basically defer to the Community Prediction. Conversely, with certain long-term questions I believe it’s important to interrogate how forecasters are reasoning about the issue at hand before assigning their predictions too much weight. Forecasters can help themselves by writing comments that explain their reasoning.

3. ^{^}

   In addition, stakeholders we work with, who look at our questions with a view to informing their grantmaking, policymaking, etc., frequently say that they would find more comments valuable in helping bring context to the Community Prediction.

^{^}

All blame on me, if so.

^{^}

Update: I ended up leaving Metaculus fairly soon after writing this post. I think that means the essay contest is less likely to happen, but I guess stay tuned in case it does.

Show all footnotes

Meta: I’m seeing lots of blank comments in response to the DIY polls. Perhaps people are thinking that they need to click ‘Comment’ in order for their vote to count? If so, PSA: your vote counted as soon as you dropped your slider. You can simply close the pop-up box that follows if you don’t also mean to leave a comment.

Happy voting!

Looking back on leaving academia for EA-aligned research, here are two things I'm grateful for:

1. Being allowed to say why I believe something.
2. Being allowed to hold contradictory beliefs (i.e., think probabilistically).

In EA research, I can write: 'Mortimer Snodgrass, last author on the Godchilla paper (Gopher et al., 2021), told me "[x, y, z]".'

In academia, I had to find a previous paper to cite for any claim I made in my paper, even if I believed the claim because I heard it elsewhere. (Or, rather, I did the aforementioned for my supervisor's papers - I used to be the research assistant who cherry picked citations off Google Scholar.)

In EA research, I can write, 'I estimate that the model was produced in May 2021 (90% confidence interval: March–July 2021)', or, 'I'm about 70% confident in this claim', and even, 'This paper is more likely than not to contain an important error.'

In academia, I had to argue for a position, without conceding any ground. I had to be all-in on whatever I was claiming; I couldn't give evidence and considerations for and against. (If I did raise a counterargument, it would be as setup for a counter-counterargument.)

That's it. No further point to be made. I'm just grateful for my epistemic freedom nowadays.

TAI makes everything else less important.

One of my CERI fellows asked me to elaborate on a claim I made that was along the lines of,* "If AI timelines are shorter, then this makes (direct) nuclear risk work less important because the time during which nuclear weapons can wipe us out is shorter."

There's a general point here, I think, which isn't limited to nuclear risk. Namely, AI timelines being shorter not only makes AI risk more important, but makes everything else less important. Because the time during which the other thing (whether that be an asteroid, engineered pandemic, nuclear war, nanotech-caused grey goo scenario, etc.) matters as a human-triggered x-risk is shortened.

To give the nuclear risk example:^

If TAI is 50 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^50) = 22%

If TAI is 15 years away, and per-year risk of nuclear conflict is 0.5%, then risk of nuclear conflict before TAI is 1-(0.995^15) = 7%

This does rely on the assumption that we'll be playing a different ball game after TAI/AGI/HLMI arrives (if not, then there's no particular reason to view TAI or similar as a cut-off point), but to me this different ball game assumption seems fair (see, e.g., Muehlhauser, 2019).

*My background thinking behind my claim here has been inspired by conversations with Michael Aird, though I'm not certain he'd agree with everything I've written in this shortform.

^A couple of not-that-important caveats:

• "Before TAI" refers to the default arrival time of TAI if nuclear conflict does not happen.
• The simple calculations I've performed assume mutual independence between nuclear-risk-in-given-year-x and nuclear-risk-in-given-year-y.

One thing the AI Pause Debate Week has made salient to me: there appears to be a mismatch between the kind of slowing that on-the-ground AI policy folks talk about, versus the type that AI policy researchers and technical alignment people talk about.

My impression from talking to policy folks who are in or close to government—admittedly a sample of only five or so—is that the main^[1] coordination problem for reducing AI x-risk is about ensuring the so-called alignment tax gets paid (i.e., ensuring that all the big labs put some time/money/effort into safety, and that none “defect” by skimping on safety to jump ahead on capabilities). This seems to rest on the assumption that the alignment tax is a coherent notion and that technical alignment people are somewhat on track to pay this tax.

On the other hand, my impression is that technical alignment people, and AI policy researchers at EA-oriented orgs,^[2] are not at all confident in there being a viable level of time/money/effort that will produce safe AGI on the default trajectory. The type of policy action that’s needed, so they seem to say, is much more drastic. For example, something in the vein of global coordination to slow, limit, or outright stop development and deployment of AI capabilities (see, e.g., Larsen’s,^[3] Bensinger’s, and Stein-Perlman’s debate week posts), whilst alignment researchers scramble to figure out how on earth to align frontier systems.

I’m concerned by this mismatch. It would appear that the game plans of two adjacent clusters of people working to reduce AI x-risk are at odds. (Clearly, this is an oversimplification and there are a range of takes from within both clusters, but my current epistemic status is that this oversimplification gestures at a true and important pattern.)

Am I simply mistaken about there being a mismatch here? If not, is anyone working to remedy the situation? Or does anyone have thoughts on how this arose, how it could be rectified, or how to prevent similar mismatches from arising in the future?

1. ^{^}

   In the USA, this main is served with a hearty side order of “Let’s make sure China in particular never races ahead on capabilities.”

2. ^{^}

   e.g., Rethink Priorities, AI Impacts

3. ^{^}

   I’m aware that Larsen recently crossed over into writing policy bills, but I’m counting them as a technical person on account of their technical background and their time spent in the Berkeley sphere of technical alignment people. Nonetheless, perhaps crossovers like this are a good omen for policy and technical people getting onto the same page.

I'd heart react if this forum introduced reactions.^[1]

1. ^{^}

   There have been times in the past (e.g., here) when I've wished there were a reaction feature, and I agree with the LessWrong post's thesis that a reaction feature would positively shape forum culture.

I just came across this old comment by Wei Dai which has aged well, for unfortunate reasons.

I think a healthy dose of moral uncertainty (and normative uncertainty in general) is really important to have, because it seems pretty easy for any ethical/social movement to become fanatical or to incur a radical element, and end up doing damage to itself, its members, or society at large. (“The road to hell is paved with good intentions” and all that.)