That archetype has largely evaporated. These people are now 30-something

I agree that those specific people are now 30-something, but I still see a lot of this archetype on twitter, substack, etc. 

You can just do things

My experience is that people often don't post for the mundane reason that they didn't really think about it.^[1] Contests are a good way of counteracting this, but the monetary reward is usually not very motivating, it's just that it makes "posting about X" become a Thing with a Deadline etc.

An implication of this is that anyone (even you, dear reader) can solve this problem with a couple days of work:

1. Message a few people that authors would value feedback from, say "hey, if I got people to write about X, would you commit to replying?"
2. Message a few prospective authors, say "hey, I've gotten these people to commit to giving feedback, would you be willing to post something?"
3. Once you've got 5-10 people to agree, post that it's happening
4. Follow up a few times with reminders because people are disorganized

This is basically all I did for the AI Pause Debate, for example, and that seemed to go fairly well. 

1. ^{^}

   For example: I am commenting here because someone messaged me to suggest that I do so. Otherwise I likely wouldn't have seen this post.

Thanks for writing this, I thought it was interesting.

I do think that early work was making a claim more like "reward maximization will lead to IC" (which, as you link to, is provably true in certain setups), and it's worth re-evaluating those claims in light of the fact that a lot of modern ML doesn't look like "reward the model for maximizing a fixed objective."

One prediction which would come out of this is that we should see more instrumental convergence if/when RL becomes a larger part of the training picture.

Isn't the answer to "At what level of capability is this mythical hard left turn towards meglomania meant to suddenly show up simultaneously in all these AI's" simply "when megalomania is instrumentally useful?"

It seems to me that when I, e.g., ask an agent to add a button to my website, it would not be more likely to be successful if it first tried to overthrow the US government. So the fact that it doesn't try to overthrow the US government doesn't seem to give much evidence either way about IC.

It seems to me that negative PR of EA over time has correlated quite well with how influential we were at that time.^[1] "People hate philosophers" doesn't explain why the hate focuses on EA in particular:

I'm in favor of reiterating the points you make about our track record, but also we should be realistic that "I support [policy] but hate those extremists who advocate for [policy]" is what success looks like. E.g. MLK was extremely unpopular^[2] whilst civil rights legislation was being passed: 

1. ^{^}

   And also how generally newsworthy we were. One of the clearest learnings I have about EA comms is that a story will get >10x the traction if the journalist can somehow include "Elon Musk" in the title.

2. ^{^}

   As comparison: LBJ, a president so unpopular he didn't even attempt to run for re-election, bottomed out at a net approval 13 points higher than MLK

Thanks for starting this group!

Thanks Dan, I was about to tell you that I meant that the null hypothesis can't be rejected,^[1] but then I asked Opus to confirm. Opus says that, while all of the metrics I posted here do indeed show a nonsignificant difference, age-adjusted post-chatGPT YC companies are worth 1.5x more on a log scale, and that this is significant at p = 0.001 (which survives corrections for multiple comparisons).

I've changed the title to say that they aren't growing "much" faster, which hopefully conveys the accurate takeaway.

1. ^{^}

   I tried to say this informally in the article: "YC companies are arguably growing a bit faster than pre-2023 companies (including AfterQuery, YC’s fastest-ever unicorn), but the difference isn’t large relative to the underlying variance"

Thanks for writing this Geoffrey, it seems important.

it looks like most AI safety orgs don't have a mass communications function

I think this is true, with some exceptions (e.g. MIRI)

why, if AI truly makes most developers 10x more productive and transforms the unit economics of customers, are the valuations of a cohort of prestigious "AI-enabled" startups round about their non-AI enabled 2019 peers?

Is this true? In my previous post I noted that the YC data was out of step with other data sets like Stripe's, but I haven't found a great dataset to answer this conclusively.