JWS's Quick takes

Many people find the Forum anxiety inducing because of the high amount of criticism. So, in the spirit of Giving Season, I'm going to give some positive feedback and shout-outs for the Forum in 2023 (from my PoV). So, without further ado, I present the 🎄✨🏆 totally-not-serious-worth-no-internet-points-JWS-2023-Forum-Awards: 🏆✨🎄^[1]
 

Best Forum Post I read this year:

10 years of Earning to Give by @AGB: A clear, grounded, and moving look at what it actually means to 'Earn to Give'. In particular, the 'Why engage?' section really resonated with me.

Honourable Mentions:

• Why I'm still going out to bat for the EA movement post FTX by @Gemma Paterson: This post was a reflection on EAs tough year which was similar to my own journey, but was also able to make a case for why EA is worth fighting for.
• Third Wave Effective Altruism by @Ben_West: A short post, but one that came at exactly the right time. I think this is a really important topic for EA to be thinking about right now and look forward to more posts inspired by it next year.
• The Seeker’s Game – Vignettes from the Bay by @Yulia: I really enjoyed both the human perspective of this post, and the style with which it was written.

Best Sequence I read this year:

• The CURVE Sequence: Causes and Uncertainty: Rethinking Value in Expectation from Rethink Priorities: This sequence was utterly brilliant. All its posts are well-written and led to good discussion from the community. I particularly want to shout out Arvo's piece that stress-tested what EV from x-risk looks like under specific assumptions.

EA Forum Posters of the Year:

• @titotal  - I really value titotal's contributions to the Forum. Even though I sometimes disagree I always learn a lot by thinking about where and why I do so, and I hope they stick around so I can continue to learn from them.
• @Holly_Elmore  - Similarly to Titotal, while I don't always agree with Holly she just makes really solid points that always positively add to the discussion. Often she's able to clearly find the crux of a debate and ask the key questions.
• @Ivy Mazzola  - Whenever I've seen Ivy post on the Forum, even on heated topics, she always approaches them with integrity and grace that I really wanted to give recognition to.
• @David Mathers  - When David discusses ideas in the Forum he consistently does so in a good faith and truth-seeking way that really shows engagement on the Forum at its best.

Non-Forum EA Poasters of the Year:

• David Thorstad (on his blog) - I've really appreciated David's blog this year. I don't always agree, and you can find me in the comments pushing back sometimes, I've found his content high quality overall, especially on Existential Risk and the Time of Perils.
• Agustín (on Twitter) - Agus has been really fighting the good fighting in the Twitter trenches recently, on all sorts of topics (though sadly on the terrible state of AI discourse). Somehow he manages to translate complicated EA ideas clearly, convincingly, and remain calm in the face of less-than-charitable interlocutors.
• Nathan Young (on Twitter) - Nathan's commitment to honest exploration of his beliefs, dialogue with those who disagree, and finding mechanisms of belief resolution has been constantly inspiring.

Congrats to the winners! Feel free to give them more plaudits in the comments or just shout out your own Forum favourites from this year! I wish everyone who’s reading this every success in the year to come and I look forward to reading more good content on the Forum in 2024!

1. ^{^}

   Note that all of the award categories and choices are arbitrary, this is an intensely vibes-based post

On Stepping away from the Forum and "EA"

I'm going to stop posting on the Forum for the foreseeable future^[1]. I've learned a lot from reading the Forum as well as participating in it. I hope that other users have learned something from my contributions, even if it's just a sharper understanding of where they're right and I'm wrong! I'm particularly proud of What's in a GWWC Pin? and 5 Historical Case Studies for an EA in Decline.

I'm not deleting the account so if you want to get in touch the best way is probably DM here with an alternative way to stay in contact. I'm happy to discuss my reasons for leaving in more detail,^[2] or find ways of collaborating on future projects.^[3]

I don't have the data to hand but I'd guess I'm probably one of the higher percentile Forum users in recent years, so why such a seemingly sudden change? The reason is that it's less a relation to the Forum and more due to my decision to orient away from "EA" in my life - and being consistent there means stepping away from the Forum. I could write a whole post on the reasons I have for not feeling 'EA', but some paragraph summaries are:

• I don't really know what "EA" means anymore: There's always been a tension between the 'philosophy or movement' framings of EA, so in practice it's been used as a fuzzy label for sets of ideas or people. In practice, "EA" seems to have been defined by its enemies for the past ~2.5 years, though I know CEA seems to change this. But I think this lack of clarity is actually a sign that EA doesn't really have a coherent identity at the moment. It doesn't seem right to strongly associate with something I can't clearly define.
• I have increasing differences with 'philosophical' EA (as I understand it): This difference has been growing recently, and it's a fairly long list of which I'll only include a few things. I think viewing morality as about 'the best' instead of 'doing right' is a mistake, especially if the former leads to viewing morality as about global/perspectiveless maximisation.^[4] I'm a virtue ethicist and not a consequentialist/utilitarian. I think my special relationships to others in my life create important and partial moral obligations/duties. I don't think Expected Value is the only or best way to make decisions for individuals or institutions. I think cluelessness/Knightian uncertainty arguments defeat most of the cases for longtermism in practice. The various differences I have seem significant enough that I can't really claim to be EA philosophically unless 'EA' is drawn arbitrarily and trivially wide.
• I also don't feel connected to the 'movement' side of EA either: I'm not personally or socially connected to much of EA. My friends and close personal relationships are not related to EA. While I'm more professionally involved with AI Safety than before, I also make clear that my positions are fairly unorthodox in this space too. While I've done my own bit of defending EA on and off the Forum, I no longer feel the identification or need to. So I'm starting to leave the various online EA spaces I am a part of one-by-one.^[5] Given my limited connection to EA personally or philosophically, it seems odd to be part of the movement in ways that imply I'm giving it more support than I actually do.
• I think there are better ways for me to spend my time than engaging with EA: In the last ~6 months EA engagement hasn't made me happy. This has often been from seeing EA criticisms left unresponded to, and I don't want to be associated with something society views negatively if I don't support that thing! I also think that there are more interesting and fulfilling pathways for my life to pursue which are either orthogonal to EA, or outside the 'orthodoxy' of cause areas/interventions. Finally, I just think I spend too much time reading the Forum or EA Twitter, and going cold turkey would be a good way to reallocate my attention better. I don't think that the 'right' thing for me to do in terms of doing the right thing, or for personal flourishing, is to engage with EA.
• Overall Takeaway: I never really claimed the label 'EA' for myself - it was never the basis of my identity and I've never really had the 'taking ideas seriously' genie take over my life. But I think, given my differences, I want to put clearer distance between me and EA into the future.

Anyway, that turned out to be a fair bit longer than I intended! If you made to the end, then I wish you all the best in your future endeavours 👋^[6]

1. ^{^}

   Definitely 'for now', but possibly for longer. I haven't quite decided yet.

2. ^{^}

   Though see below first

3. ^{^}

   See the end of this comment for some ideas

^{^}

Indeed, from a certain point of view EA could be seen as a philosophical version of Bostrom's paperclip monster. If there is a set definition of the good, and you want to maximise, then the right thing to do is paperclip the universe with your definition. My core commitment to pluralism views this as wrong, and makes me deeply suspicious of any philosophy which allows this, or directionally points towards it

^{^}

I do, however, intend to continue with the GWWC Pledge for the foreseeable future

^{^}

I also wish you the best if you didn't

Show all footnotes

Ho-ho-ho, Merry-EV-mas everyone. It is once more the season of festive cheer and especially effective charitable donations, which also means that it's time for the long-awaited-by-nobody-return of the 🎄✨🏆 totally-not-serious-worth-no-internet-points-JWS-Forum-Awards 🏆✨🎄, updated for the 2024! Spreading Forum cheer and good vibes instead of nitpicky criticism!!
 

Best Forum Post I read this year:

Explaining the discrepancies in cost effectiveness ratings: A replication and breakdown of RP's animal welfare cost effectiveness calculations by @titotal 

It was a tough choice this year, but I think this deep, deep dive into the different cost effectiveness calculations that were being used to anchor discussion in the GH v AW Debate Week was thorough, well-presented, and timely. Anyone could have done this instead of just taking the Saulius/Rethink estimates at face value, but titotal actually put in the effort. It was the culmination of a lot of work across multiple threads and comments, especially this one, and the full google doc they worked through is here. 

This was, I think, an excellent example of good epistemic practices on the EA Forum. It was a replication which involved people on the original post, drilling down into models to find the differences, and also surfacing where the disagreements are based on moral beliefs rather than empirical data. Really fantastic work. 👏

Honourable Mentions:

• Towards more cooperative AI safety strategies by @richard_ngo: This was a post that I read at exactly the right time for me, as it was a point that I was also highly concerned that the AI Safety field was having a "legitimacy problem".^[1] As such, I think Richard's call to action to focus on legitimacy and competence is well made, and I would urge those working explicitly in the field to read it (as well as the comments and discussion on the LessWrong version), and perhaps consider my quick take on the 'vibe shift' in Silicon Valley as a chaser.
• On Owning Our EA Affiliation by @Alix Pham: One of the most wholesome EA posts this year on the Forum? The post is a bit bittersweet to me now, as I was moved by it at the time but now I affiliate and identify less with EA now that than I have for a long time. The vibes around EA have not been great this year, and many people are explicitly or implicitly abandoning the movement, alix actually took the radical idea of "do the opposite". She's careful to try to draw a distinction between affiliation and identity, and really engages in the comments leading to very good discussion.
• Policy advocacy for eradicating screwworm looks remarkably cost-effective by @MathiasKB🔸: EA Megaprojects are BACK baby! More seriously, this post people had the most 'blow my mind' effect on me this year. Who knew that the US Gov already engages in a campaign of strategic sterile-fly bombing, dropping millions of them on Central America every week? I feel like Mathias did great work finding a signal here, and I'm sure other organisations (maybe an AIM-incubated kind of one) are well placed to pick up the baton.

Forum Posters of the Year:

• @Vasco Grilo🔸 - I presume that the Forum has a bat-signal of sorts, where long discussions are made without anyone trying to do an EV calculation. And in such dire times, vasco appears, and always with amazing sincerity and thoroughness. Probably the Forum's current postchild of 'calculate all the things' EA. I think this year he's been an awesome presence on the Forum, and long may it continue.
• @Matthew_Barnett - Matthew is somewhat of an engima to me ideologically, there have been many cases where I've read a position of his and gone "no that can't be right". Nevertheless, I think the consistently high-quality nature of his contributions on the Forum, often presenting an unorthodox view compared to the rest of EA, is worth celebrating regardless of whether I personally agree. Furthermore, one of my major updates this year has been towards viewing the Alignment Problem as one of political participation and incentives, and this can probably traced back significantly to his posts this year.

Non-Forum Poasters of the Year:

• Matt Reardon (mjreard on X) - Currently, X is not a nice place to be an Effective Altruist at the moment. It seems to be attacked from all directions, and it means it's not fun at all to push back on people and defend the EA point-of-view. Yet Matt has just consistently pushed back on some of the most egregious cases of this,^[2] and also has had good discussion in EA Twitter too.
• Jacques Thibodeau (JacquesThibs on X) - I think Jacques is great. He does interesting cool work on Alignment and you should consider working with him if you're also in that space. I think one of the most positivie things that Jacques does on X to build bridges across the wider 'AGI Twitter', including many who are sceptical or even hostile to AI Safety work like teortaxesTex or jd_pressman? I think this to his great credit, and I've never (or rarely) seen him get that angry on the platform, which might even deserve another award!

Congratulations to all of the winners! I also know that there were many people who made excellent posts and contributions that I couldn't shout out, but I want to know that I appreciate all of you for sharing things on the Forum or elsewhere.

My final ask is, once again, for you all to share your appreciation for others on the Forum this year and tell me what your best posts/comments/contributors were this year!

1. ^{^}

   I think that the fractured and mixed response to the latest Apollo reports (both for OpenAI and Anthropic) is partially downstream of this loss of trust and legitimacy

2. ^{^}

   e.g. here and here

Reflections 🤔 on EA & EAG following EAG London (2024):

• I really liked the location this year. The venue itself was easy to get to on public transport, seemed sleek and clean, and having lots of natural light on the various floors made for a nice environment. We even got some decent sun (for London) on Saturday and Sunday. Thanks to all the organisers and volunteers involved, I know it’s lot of work setting up an event like this us and making it run smoothly.
• It was good to meet people in person who I previous had only met or recognised from online interaction. I won’t single out individual 1-on-1s I had, but it was great to be able to put faces to names, and hearing peoples stories and visions in person was hugely inspiring. I talked to people involved in all sorts of cause areas and projects, and that combination of diversity, compassion, and moral seriousness is one of the best things about EA.
• Listening to the two speakers from the Hibakusha Project at the closing talk was very moving, and clear case of how knowing something intellectually is not the same thing as hearing personal (and in-person) testimony. I think it would’ve been one of my conference highlights in the feedback form if we hadn’t already been asked to fill it out a few minutes beforehand!
• I was going to make a point about a ‘lack of EA leadership’ turning up apart from Zach Robinson, but when I double-checked the event attendee list I think I was just wrong on this. Sure, a couple of big names didn’t turn up, and it may depend on what list of ‘EA leaders’ you’re using as a reference, but I want to admit I was directionally wrong here.
• I thought Zach gave a good opening speech, but many people noted on the apparent dissonance between saying that CEA wanted to focus on ‘principles-first’ approach to EA, but that they also expected AI to be their area of most focus/highest priority and that they don't expect that to change in the near future.
• Finally, while I’m sure the people I spoke to (and those who wanted to speak to me) is strongly affected by selection-effects, and my own opinions on this are fairly strong, it did feel that there was consensus on there being a lack of trust/deference/shared beliefs from ‘Bay-Area EA’:^[1]
  • Many people think that working on AI Safety and Governance is important and valuable, but not 'overwhelmingly important' or 'the most important thing human has done/will ever do'. This included some fairly well-known names from those who attended, and basically nobody there (as far as I could tell) I interacted with held extremely 'doomer' beliefs about AI.
  • There was a lot of uncomfortable feeling at the community-building funding being directed to ‘longtermism’ and AI Safety in particular. This is definitely a topic I'm want to investigate more post-EAG, as I'm not sure what the truth of the matter is, but I'd certainly find it problematic if some of the anecdotes I heard were a fair representation of reality.
  • In any case, I think it's clear that AI Safety is no longer 'neglected' within EA, and possibly outside of it.^[2] (Retracted this as, while it's not true, commenters have pointed out that it's not really the relevant metric to be tracking here) 
  • On a personal level, it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG. Feels like it could be a coincidence, but on the other hand this is not a coincidence because nothing is ever a coincidence. It feeds into my impression that the Bay is not very interested in what the rest of EA has to say.
  • One point which I didn't get any clear answers to was 'what are the feedback mechanisms in the community to push back on this', and do such feedback mechanisms even exist?
  • In summary: It feels like, from my perspective, that the Bay Area/Exclusively Longtermist/AI Safety Maximalist version of EA has 'lost of the mandate of heaven', but nonetheless at the moment controls a lot of the community's money and power. This, again, is a theme I want to explicitly explore in future posts.
• I am old (over 30yo) and can’t party like the young EAs anymore 😔

1. ^{^}

   I'm not sure I have a good name for this, or concrete dividing lines. But in discussions people seemed to understand what it was meant to capture.

2. ^{^}

   To me it does seem like the case for the overwhelming importance of AI has been under-argued for and under-scrutinised.

<edit: Ben deleted the tweets, so it doesn't feel right to keep them up after that. The rest of the text is unchanged for now, but I might edit this later. If you want to read a longer, thoughtful take from Ben about EA post-FTX, then you can find one here>

This makes me feel bad, and I'm going to try and articulate why. (This is mainly about my gut reaction to seeing/reading these tweets, but I'll ping @Benjamin_Todd because I think subtweeting/vagueposting is bad practice and I don't want to be hypocritical.) I look forward to Ben elucidating his thoughts if he does so and will reflect and respond in greater detail then.

• At a gut-level, this feels like an influential member of the EA community deciding to 'defect' and leave when the going gets tough. It's like deciding to 'walk away from Omelas' when you had a role in the leadership of the city and benefitted from that position. In contrast, I think the right call is to stay and fight for EA ideas in the 'Third Wave' of EA.
• Furthermore,if you do think that EA is about ideas, then I don't think dissassociating from the name of EA without changing your other actions is going to convince anyone about what you're doing by 'getting distance' from EA. Ben is a GWWC pledger, 80k founder, and is focusing his career on (existential?) threats from advanced AI. To do this and then deny being an EA feels disingenuous for ~most plausible definitions of EA to me.
  • Similar considerations to the above make me very pessimisitic about the 'just take the good parts and people from EA, rebrand the name, disavow the old name, continue operating as per usual' strategy to work at all
  • I also think that actions/statements like this make it more likely for the whole package of the EA ideas/community/brand/movement to slip into a negative spiral which ends up wasting its potential, and given my points above such a collapse would also seriously harm any attempt to get a 'totally not EA yeah we're definitely not those guys' movement off the ground.
• In general it's easy pattern for EA criticism to be something like "EA ideas good, EA community bad" but really that just feels like a deepity. For me a better criticism would be explicit about focusing on the funding patterns, or focusing on epistemic responses to criticism, because attacking the EA community at large to me means ~attacking every EA thing in total. 
  • If you think that all of EA is bad because certain actors have had overwhelmingly negative impact, you could just name and shame those actors and not implicitly attack GWWC meetups and the like. 
  • In the case of these tweets, I think a future post from Ben would ideally benefit from being clear about what 'the EA Community' actually means, and who it covers.

Quick^[1] thoughts on the Silicon Valley 'Vibe-Shift'

I wanted to get this idea out of my head and into a quick-take. I think there's something here, but a lot more to say, and I've really haven't done the in-depth research for it. There was a longer post idea I had for this, but honestly diving more than I have here into it is not a good use of my life I think.

The political outlook in Silicon Valley has changed.

Since the attempted assassination attempt on President Trump, the mood in Silicon Valley has changed. There have been open endorsements, e/acc has claimed political victory, and lots of people have noticed the 'vibe shift'.^[2] I think that, rather than this being a change in opinions, it's more an event allowing for the beginning of a preference cascade, but at least in Silicon Valley (if not yet reflected in national polling) it has happened. 

So it seems that a large section of Silicon Valley is now openly and confidently supporting Trump, and to a greater or lesser extent aligned with the a16z/e-acc worldview,^[3] we know it's already reached the ears of VP candidate JD Vance.

How did we get here

You could probably write a book on this, so this is a highly opinionated take. But I think this is somewhat, though not exclusively, an own goal of the AI Safety movement.

• As ChatGPT starts to bring AI, and AI Safety, into the mainstream discourse, the e/acc countermovement begins. It positions itself as opposite effective altruism, especially in the wake of SBF.
• Guillaume Verdon, under the alias "Beff Jezos", realises the memetic weakness of the AI Safety movement and launches a full memetic war against it. Regardless of his rightness or wrongness, you do to some extent got to hand it to him. He's like right-wing Émile Torres, ambitious and relentless and driven by ideological zeal against a hated foe.
• Memetic war is total war. This means nuance dies to get it to spread. I don't know if, for example, Marc Andreessen actually thinks antimalarial bednets are a 'triple threat' of badness, but it's a war and you don't take prisoners. Does Beff think that people running a uni-group session on Animal Welfare are 'basically terrorists', I don't know. But EA is the enemy, and the enemy must be defeated, and the war is total.
• The OpenAI board fiasco is, I think, a critical moment here. It doesn't matter what the reasoning we've come out with at the end of the day was, I think it was perceived as 'a doomer coup' and it did radicalize the valley. In his recent post Richard Ngo called on the AI Safety movement to show more legitimacy and competence. The board fiasco torpedoed my trust in the legitimacy and competence of many senior AI safety people, so god knows how strong the update was for Silicon Valley as a whole.
  • As some evidence this is known in EA circles, I think this is exactly what Dwarkesh is alluding to when asked 'what happened to the EA brand'. For many people in Silicon Valley, I think the answer is that it got thrown in the dustbin of history.
• This new movement became increasingly right-wing coded. Partly as a response to the culture wars in America and the increasing vitriol thrown by the left against 'tech bros', partly as a response to the California Ideology being threatened by any sense of AI oversight or regulation, and partly because EA is the enemy and EA was being increasingly seen by this group as left-wing, woke, or part of the Democratic Party due to the funding patterns of SBF and Moskovitz. I think this has led, fairly predictably, to the right-ward shift in SV and direct political affiliation with a (prospective) second Trump presidency
• Across all of this my impression is that, just like with Torres, there was little to no direct pushback. I can understand not wanting to be dragged into a memetic war, or to be involved in the darker parts of Twitter discourse. But the e-acc/technooptimist/RW-Silicon-Valley movement was being driven by something, and I don't think AI Safety ever really argued against it convincingly, and definitely not in a convincing enough way to 'win' the memetic war. Like, the a16z cluster literally lied to Congress and to Parliament, but nothing much come of that fact.
  • I think this is very much linked to playing a strong 'inside game' to access the halls of power and no 'outside game' to gain legitimacy for that use of power. It's also I think due to EA not wanting to use social media to make its case, whereas the e-acc cluster was born and lives on social media.

Where are we now?

I'm not a part of the Bay Area scene and culture,^[4] but it seems to me that the AI Safety movement has lost the 'mandate of heaven' to whatever extent it did have it. SB-1047 is a push to change policy that has resulted in backlash, and may result in further polarisation and counter-attempts to fight back in a zero-sum political game. I don't know if it's constitutional for a Trump/Vance administration to use the Supremacy Clause to void SB-1047 but I don't doubt that they might try. Biden's executive order seems certain for the chopping block. I expect a Trump administration to be a lot less sympathetic to the Bay Area/DC AI Safety movements, and the right-wing part of Silicon Valley will be at the very least energised to fight back harder.

One concerning thing for both Silicon Valley and the AI Safety movement is what happens as a result of the ideological consequences of SV accepting this trend. Already a strong fault-line is the extreme social conservatism and incipient nationalism brought about by this. In the recent a16z podcast, Ben Horowitz literally accuses the Biden administration of breaking the rule of law, and says nothing about Trump literally refusing to concede the 2020 election and declaring that there was electoral fraud. Mike Solana seems to think that all risks of democratic backsliding under a Trump administration were/are overblown (or at least that people in the Bay agreeing was preference falsification). On the Moments-of-Zen Podcast (which has also hosted Curtis Yarvin twice), Balaji Srinivasan accused the 'Blue Tribe' of ethnically cleansing him out of SF^[5] and called on the grey tribe to push all the blues out of SF. e-acc sympathetic people are noting that anti-trans ideas bubbling up in the new movement. You cannot seriously engage with ideas and shape them without those ideas changing you.^[6] This right-wing shift will have further consequences, especially under a second Trump presidency.

What next for the AI Safety field?

I think this is a bad sign for the field of AI Safety. Political polarisation has escaped AI for a while. Current polls may lean in support , but polls and political support are fickle, especially in the age of hyper-polarisation.^[7] I feel like my fears around the perception of Open Philanthropy are re-occuring here but for the AI Safety movement at large. 

I think the consistent defeats to the e-acc school and the fact that the tech sector as a whole seems very much unconvinced by the arguments for AI Safety should at some point lead to a reflection from the movement. Where you stand on this very much depends on your object-level beliefs. While this is a lot of e-acc discourse around transhumanism, replacing humanity, and the AI eschaton, I don't really buy it. I think that they don't think ASI is possible soon, and thus all arguments for AI Safety are bunk. Now, while the tech sector as a whole might not be as hostile, they don't seem at all convinced of the 'ASI-soon' idea.

A key point I want to emphasise is that one cannot expect to wield power successfully without also having legitimacy.^[8] And to the extent that the AI Safety movement's strategy is trying to thread this needle it will fail.

Anyway, long ramble over, and given this was basically a one-shot ramble it will have many inaccuracies and flaws. Nevertheless I hope that it can be directionally useful and lead to productive discussion.

1. ^{^}

   lol, lmao

2. ^{^}

   See here, here, and here. These examples are from Twitter because, for better or for worse, it seems much of SV/tech opinions are formed by Twitter discourse.

3. ^{^}

   Would be very interested to hear the thoughts of people in the Bay on this

^{^}

And if invited to be I would almost certainly decline,

^{^}

He literally used the phrase 'ethnically cleanse'. This is extraordinarily dangerous language in a political context.

^{^}

A good example in fiction is in Warhammer40K, where Horus originally accepts the power of Chaos to fight against Imperial Tyranny, but ends up turning into their slave.

^{^}

Due to polarisation, views can dramatically shift on even major topics such as the economy and national security (i know these are messy examples!). Current poll leads for AI regulation should not, in any way, be considered secure

^{^}

I guess you could also have overwhelming might and force, but even that requires legitimacy. Caesar needed to be seen as legitimate by Marc Anthony, Alexander didn't have the legitimacy to get his army to cross the Hyphasis etc.

Show all footnotes

I want to register that my perspective on medium-term^[1] AI existential risk (shortened to AIXR from now on) has changed quite a lot this year. Currently, I'd describe it as moving from 'Deep Uncertainty' to 'risk is low in absolute terms, but high enough to be concerned about'. I guess atm I'd think that my estimates are moving closer toward the Superforecasters in the recent XPT report (though I'd still say I'm still Deeply Uncertain on this issue, to the extent that I don't think the probability calculus is that meaningful to apply)

Some points around this change:

• I'm not sure it's meaningful to cleanly distinguish AIXR from other anthropogenic x-risks, especially since negative consequences of AI may plausibly increase other x-risks (e.g. from Nuclear War, biosecurity, Climate Change etc.)
• I think in practice, the most likely risks from AI would come from deployment of powerful systems that have catastrophic consequences that are then rolled back. I'm think of Bing 'Sydney' here as the canonical empirical case.^[2] I just don't believe we're going to get no warning shots.
• Similary, most negative projections of AI don't take into account negative social reaction and systematic human response to these events. You either assume we'll get no warning shots and then get exterminated ('sharp left turn') or think that humanity is doomed to not co-operate ('Moloch'). I think that the evidence suggests instead that societies and governments will react against increasing AI capability if they view it negatively, rather than simply stand by and watch it happen, which is what many AIXR seem to assume or at least imply to me.
• I think the AIXR community underestimated the importance of AI Governance and engaging with politics. Instead of politicians/the public 'not getting it', they infact seem to have 'got it'. The strategy of only letting people vetted to be smart enough think about AIXR seems to have been flawed, the same kind of thinking that led to thinking a 'pivotal act' strategy was viable.
• In fact, I've been pleasantly surprised by how welcoming both politicians and the public have been to the Overton Window being opened. Turns out the median voter in a liberal democracy doesn't like 'let large co-orporations create powerful models with little understanding of their consequences without significant oversight'. I think people's expectations of positive co-ordination on AI issues should have gone up this year, and conversely it should make your AIXR estimate go down unless you think relative alignment progress has declined even more.
• While there have been an awful lot of terrible arguments against AIXR raised this year, some have been new and valuable to me. Some examples:
  • As titotal has written, I expect 'early AGI' will be a 'buggy mess'. More specifically, I doubt it could one-shot solve 'take over the world' unless you assume god-like capability by assumption, instead of 'more capable than humans in some important ways, but not invincible'.
  • Progress via 'stack moar layers lol' will plausibly slow down, or at least run into some severe issues.^[3] The current hype-wave seems to be just following this as far as it will go with compute and data, rather than exploring alternative architectures that could be much more efficient.
  • I'm still unimpressed by how Frontier systems perform on tests such as The ARC Challenge which test hypothesis creativity and testing in a few-shot scenario (and on hidden data) without 'ground truth', as opposed to training on trillions-upon-trillions of training examples with masks and true labels.
  • Related to the above, I view creativity and explanation as critical to science and the progress of science, so I'm not sure what 'automation of science' would actually look like. It makes me very sceptical of claims like Davidson's with larger numbers like '1000x capability' (edit, it's actually 1000x in a year! Is that a one-time increase or perpetual explosive growth? It just seems way too strong a claim).^[4]
  • Progress in AI Safety has empirically been correlated, at least weakly, with increases in capability. I didn't agree with everything in Nora's recent post, but I think she's right about assessing a fully thoeretical approach to progress (such as MIRI's strategy, as far as I can tell).
• In summary, it doesn't seem like the field of AIXR comes out particularly strongly to me on the I-T-N Framework relative to other existential and catastrophic risks. It may be 'more important', but seems similarly intractable in the style that these global problems generally are, and definitely not as neglected any more (in terms of funding, where does it stand relative to nuclear or bio for example? would be interesting to know).
• However, I still think x-risk and catastrophic risk from AI this century is unjustifiably high. I just don't think it's plausible to have levels of pDoom ~>90% under current evidence unless you have private insight.
• I think the main existential/catastrophic issues around AI in the foreseeable future revolve around political institutions and great power conflict, rather than humans being wiped out agentically (either deceptively by a malicious power-seeker or unintentionally by an idiot-savant)

Anyway, these thoughts aren't fully worked out, I'm still exploring what I think on this issue, but just wanted to register where my current thoughts are at in case it helps others in the community.

1. ^{^}

   Say on a ~50 year time scale, or to the end of the century

2. ^{^}

   Clarification: I'm not saying 'Sydney' had catastrophic consequences, but that a future system could be released in a similar way due to internal corporate pressures, and that system could then act negatively in the real world in a way its developers did not expect.

3. ^{^}

   Btw, xuan's twitter is one of the best I know of to get insights into the state of AI. Sometimes I agree, sometimes I don't, but her takes are always legit

^{^}

What does it even mean to say 1,000x capability, especially in terms of science? Is there a number here that Davidson is tracking? When would he view it as falsified?

Show all footnotes

The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was  'getting Eulered' as Scott once described. 

I'm not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what it's worth,^[1] is:

• the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things aren't that effective
• it's worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
• it's very difficult to do good social science, and the RCT won't give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
• HLI has used language too loosely in the past and overclaimed/been overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the 'epistemic probation' comments)
• GiveWell's own 'best guess' analysis seems to be that Strongminds is 2.3x that of GiveDirectly.^[2] I'm generally a big fan of the GiveDirectly approach for reasons of autonomy - even if Strongminds got reduced in efficacy to around ~1x GD, it'd still be a good intervention? I'm much more concerned with what this number is than the tone of HLIs or Michael's claims tbh (though not at the expense of epistemic rigour).
• The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,^[3] but so much of the world is worse.

I'm also quite unsettled by a lot of what I call 'drive-by downvoting'. While writing a comment is a lot more effort than clicking to vote on a comment/post, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).

So I'm very unsure how to feel. It's an important issue, but I'm not sure the Forum has shown itself in a good light in this instance.

1. ^{^}

   And I stress this isn't much in this area, I generally defer to evaluators

2. ^{^}

   On the table at the top of the link, go to the column 'GiveWell best guess' and the row 'Cost-effectiveness, relative to cash'

3. ^{^}

   Again, I don't think I have the ability to adjudicate here, which is part of why I'm so confused.

This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong

Right now I am scared

Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board - including Toner and MacAulay - removed from the company

The mood in the Valley, and in general intellectual circles, seems to have snapped against EA^[1]

This could be as bad for EA's reputation as FTX

At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed

And this time it'd be second-impact syndrome

I am scared EA in its current form may not handle the backlash that may come

I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves

I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I've met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I've heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves is more full of kindness, happiness, and flourishing

I am scared because this could be lost, I don't know what to do to prevent it

I am scared

1. ^{^}

   Not just AI Safety, but EA as a whole. I don't think those upset and concerned with EA really care about making that distinction