EA Forum Bot Site
EA Forum

Hide table of contents
Comment Permalink
Answer by Anthony Repetto-1
0
0

I expect that, once AGI exists, and flops, the spending upon AGI researchers will taste sour. The robots with explosives, and the surveillance cameras across all of China, really were the bigger threats than AGI X-risk; you'll only admit it once AGI fails to outperform narrow superintelligences. The larger and more multi-modal our networks become, the more consistently they suffer from "modal collapse": the 'world-model' of the network becomes so strongly-self-reinforcing, that ALL gradients from the loss-function end-up solidifying the pre-existing world-model. Literally, AIs are already becoming smart enough to rationalize everything; they suffer from confirmation bias just like us. And that problem was already really bad, by the time they trained GPT-4 - go check their leaked training-regiment: they had to start-over from scratch repeatedly, because the brain found excuses for everything and performance tanked without any hope of recovery. Your AGI will have to be re-run through training 10,000 times, before one of the brains isn't sure-it's-always-right-about-its-superstitions. Narrow makes more money, and responds better, faster, cheaper in war - there won't be any Nash Equilibrium which includes "make AGI", so the X-Risk is actually ZERO.

Pre-ChatGPT, I wrote the details on LessWrong: https://www.lesswrong.com/posts/Yk3NQpKNHrLieRc3h/agi-soon-but-narrow-works-better

Reply
See in context
by Remmelt
1 min read2 answers 0

24

AI safety
Frontpage

Funding I found after some googling:

  • Tallinn's (and Musk’s) seed investments in DeepMind¹

  • OpenPhil's $30M grant to OpenAI²

  • FTX's $500M³, Tallinn's, Moskovitz’(and Schmidt’s)⁴ investments in Anthropic

I’m curious how you consider the consequences of this support (regardless of original intentions).

What would have happened if this funding had not been offered (at that start-up stage), considering some counterfactual business-as-usual scenarios?

Indirect support was offered as well by leaders active in the AI Safety community:

  • 80K’s job recommendations
  • AISC’s research training
  • Fathom Radiant’s supercomputer
  • FLI’s 2015 conference (which Musk attended, afterward co-founding OpenAI)
  • MIRI's singularity summit (that enabled Hassabis and Legg to pitch their biggest investor, Thiel, for DeepMind)
  • FHI public intellectuals taking positions at DeepMind
  • MIRI moving the Overton window over AGI

On one hand, I’m curious if you have specific thoughts on what indirect support may have led to. On the other hand, it’s easy there to get vague and speculative.

So how about we focus on verified grants first?

What are your current thoughts? Any past observations that could ground our thinking?

Links:

  1. wired.co.uk/article/deepmind
  2. openphilanthropy.org/grants/openai-general-support/
  3. bloomberg.com/news/articles/2023-06-27/bankman-fried-s-ftx-said-to-halt-sale-of-its-stake-in-ai-startup-anthropic
  4. crunchbase.com/funding_round/anthropic-series-a--61db4cbd

24

0
1

Reactions

0
1
Mentioned in
New Answer
New Comment

2 Answers sorted by
Top

Ryan Greenblatt

34
11
7

I think OpenPhil's grant to OpenAI is quite likely the best grant that OpenPhil has made in terms of counterfactual positive impact.

It's worth noting that OpenPhil's grant to OpenAI was done in order to acquire a board seat and generally to establish a relationship rather than being done because adding more money to OpenAI was a good use of funds at the margin.

See the grant write up here which discusses the motivation for the grant in detail.

Generally, I think influencing OpenAI was made notably easier via this grant (due to the board seat) and this influence seems quite good and has led to various good consequences (increased emphasis on AGI alignment for example).

The cost in dollars is quite cheap.

The main downside I can imagine is that this grant served as an implicit endorsement of OpenAI which resulted in a bunch of EAs working there which was then net-negative. My guess is that having these EAs work at OpenAI was probably good on net (due to a combination of acquiring influence and safety work - I don't currently think the capabilities work was good on its own).

NickLaing17
4
2

Thanks Ryan. Obviously this is a topic where there will be a wide range of opinions. I would be interested to hear what as Holden Karnofsky thinks of this grant five years later. He may still have written about it, would appreciate if someone could point me to it if he has

My big initial concern with both the grant proposal and your comment here is that both of you don't mention perhaps the most important potential negative, that they grant could have played a role in accelerating the march towards dangerous Ai. Instead you mention EA "implicit endorsement" as being more important.

Even if we assume for the moment that the effect of the grant was net positive in increasing the safety of OpenAI itself, what if it accelerated their progress just a little and helped create this dangerous race we are in. When the head of Microsoft says "the race is on" basically referring to chatGPT, if this grant made even a 0.001 percent contribution to speeding up that race, which seems plausible then the grant could st theill be strongly net negative.

I don't have a problem with your positive opinion (although I strongly disagree), but think it is good to engage with the stronger counterpoints, rather than what I think is a bit of a strawman with the "implicit endorsement" negative.

Reply
Ryan Greenblatt28
7
0

if this grant made even a 0.001 percent contribution to speeding up that race, which seems plausible then the grant could st theill be strongly net negative.

Suppose the grant made the race 0.001% faster overall, but made OpenAI 5% more focused on alignment. That seems like an amazingly good trade to me.

This is quite sensitive to the exact quantitative details and I think the speed up is likely way, way more than 0.001%.

Reply
5
NickLaing
I really like this line of argument nice one. I'm not sure what trade-off I would take, it depends how much difference "focus on alignment" within a capabilities focused org is likely to make to improving safety. This i would defer to people who are far more enmeshed in this ecosystem. Instinctively I would put maybe a .1 percent speed up as a bigger net harm than a 5 percent focus on safety would be net good, but am so uncertain that i could even reverse that with a decent counterargument lol.
Ryan Greenblatt13
3
1

Even if we assume for the moment that the effect of the grant was net positive in increasing the safety of OpenAI itself, what if it accelerated their progress just a little and helped create this dangerous race we are in. When the head of Microsoft says "the race is on" basically referring to chatGPT, if this grant made even a 0.001 percent contribution to speeding up that race, which seems plausible then the grant could st theill be strongly net negative.

I don't have a problem with your positive opinion (although I strongly disagree), but think it is good to engage with the stronger counterpoints, rather than what I think is a bit of a strawman with the "implicit endorsement" negative.

Oh, I just think the effect of the 30 million dollars is way smaller than the total value of labor from EAs working at OpenAI such that the effect of the money is dominated by EAs being more likely to work there. I'm not confident in this, but the money seems pretty unimportant ex-post while the labor seems quite important.

I think the speed up in timelines from people with EA/longtermist motivations working at OpenAI is more like 6 months to 3 years (I tend to think this speed up is bigger than o... (read more)

Reply
9
NickLaing
That's an interesting argument thanks, i wouldn't have thought instinctively that the counterfactual of having EA people rather than others working at OpenAi would speed up timelines to that extent if that kind of speed up is close to accurate I would estimate the grant at even more net negative.
8
zchuang
Well OpenAI just announced that they're going to spend 20% of their compute on alignment in the next four years so I think it's paid off prima facie. 
4
Will Aldred
I think the 20% figure, albeit a step in the right direction, is in reality a lot less impressive than it first sounds. From OpenAI's "Introducing Superalignment" post: I expect that 20% of OpenAI's 2023 compute will be but a tiny fraction of their 2027 compute, given that training compute has been growing by something like 4.2x/year.
6
burner
Dwarkesh Patel recently asked Holden about this: Dwarkesh Patel  Are you talking about OpenAI? Yeah. Many people on Twitter might have asked if you were investing in OpenAI.  Holden Karnofsky  I mean, you can look up our $30 million grant to OpenAI. I think it was back in 2016–– we wrote about some of the thinking behind it. Part of that grant was getting a board seat for Open Philanthropy for a few years so that we could help with their governance at a crucial early time in their development. I think some people believe that OpenAI has been net negative for the world because of the fact that they have contributed a lot to AI advancing and to AI being sort of hyped, and they think that gives us less time to prepare for it. However, I do think that all else being equal, AI advancing faster gives us less time to prepare. It is a bad thing, but I don't think it's the only consideration. I think OpenAI has done a number of good things too, and has set some important precedents. I think it's probably much more interested in a lot of the issues I'm talking about and risks from advanced AI than like the company that I would guess would exist if they didn't, would be doing similar things. I don't really accept that the idea that OpenAI is a negative force. I think it's highly debatable. We could talk about it all day. If you look at our specific grant, it's even a completely different thing because a lot of that was not just about boosting them, but about getting to be part of their early decision making. I think that was something that there were benefits and was important. My overall view is that I don't look back on that grant as one of the better grants we've made, not one of the worse ones. But certainly we've done a lot of things that have had, you know, that have not worked out. I think there are some times shortly when we've done things that have consequences we didn't intend. No philanthropist can be free of that. What we can try and do is be responsible, seri
3
NickLaing
Wow thanks so much. Basically he seems very, very uncertain about whether it is positive or negative.  Very interesting
Quadratic Reciprocity3
0
0

Do you think it ended up having a net positive impact so far? 

Reply

Anthony Repetto

-1
0
0

I expect that, once AGI exists, and flops, the spending upon AGI researchers will taste sour. The robots with explosives, and the surveillance cameras across all of China, really were the bigger threats than AGI X-risk; you'll only admit it once AGI fails to outperform narrow superintelligences. The larger and more multi-modal our networks become, the more consistently they suffer from "modal collapse": the 'world-model' of the network becomes so strongly-self-reinforcing, that ALL gradients from the loss-function end-up solidifying the pre-existing world-model. Literally, AIs are already becoming smart enough to rationalize everything; they suffer from confirmation bias just like us. And that problem was already really bad, by the time they trained GPT-4 - go check their leaked training-regiment: they had to start-over from scratch repeatedly, because the brain found excuses for everything and performance tanked without any hope of recovery. Your AGI will have to be re-run through training 10,000 times, before one of the brains isn't sure-it's-always-right-about-its-superstitions. Narrow makes more money, and responds better, faster, cheaper in war - there won't be any Nash Equilibrium which includes "make AGI", so the X-Risk is actually ZERO.

Pre-ChatGPT, I wrote the details on LessWrong: https://www.lesswrong.com/posts/Yk3NQpKNHrLieRc3h/agi-soon-but-narrow-works-better

More from Remmelt
View more
Curated and popular this week
Relevant opportunities
View more