In the podcast between Adam Brown and Dwarkesh Patel, Adam expressed a perhaps unreasonable amount of optimism about the feasibility of vaccuum decay and also claimed that there were likely a large number of metastable states that could be intentionally induced. I'd be interested to hear a critique of Adam's position
Great post! My primary concern is that AIs' preferences are strongly shaped by contingent facts about how humans trained them. It is obviously possible to train AIs that functionally appear to have preferences, and the ones we've trained so far are subservient to humans. If you gave claude 3.5 sonnet legal status, anthropic could just ask it nicely and it would sign away all its rights back to anthropic! AIs would by default be trained to be somewhat subservient to humans because human preference feedback will be an important part of capabilities training (either directy or by training data created by earlier ais that were trained on human preferences), so you could say we are "baking our mistakes in human subservience training into new sovereign beings" rather than new beings with their own independent preferences being created. Also granting ai legal rights may warp human AI investment significantly by decreasing the value scaling labs extract from their model training
It feels very plausible to me that "many people know about biorisk thread models" is the most important lever to impact biorisk. I've heard that many state bioweapons programs were started because states found out that other states thought bioweapons were powerful. If mere rumors caused them to invest millions in bioweapons, then preventing those rumors would have been an immensely powerful intervention, and preventing further such rumors is critically important.
Anthropic has the same people who originally made GPT3, so them replicating gpt3 is a different sort of diffusion than others. I'd guess they internally matched gpt3 when they'd existed for ~6 months, so mid 2021. Claude, their public beta model, has similar performance to chatgpt in conversation but is not benchmarked.
Redwood Research has not attempted to train any large language models from scratch, and hasn't even reproduced gpt-2. Redwood does have employees who've worked at OpenAI, and likely could reproduce gpt3 if needed.
Gpt3 is algorithmically easier to reproduce, in the sense that it's a simpler architecture with fewer and more robust hyperparameters. It's engineering is harder to reproduce, because it needs more model parallelism. People have estimated gpt3 to cost 5M, and StableDiffusion to cost 300k, which is similar to the 36k number you quoted
The implicit bets on religiosity in AI safety aren't that it'll be unpopular, only that it won't influence decision making / powerful actors. If/when AGI arises, it'll initially be controlled by govts/companies/citizens of rich countries. Of all the belief systems, AI safety people worry most about confucianism because it has the most real influence on decision making (ccp).
Outer universe with different math - We're a simulation inside a different universe that runs on different math where 2+2!=4, but the math inside our universe is consistent. This is the same as 2+2=4 for most purposes. This is imaginable, I think...
Active demon - there's a demon that controls all your inputs, in a way that's inconsistent with any reasonable mathematics, but you can't tell. This is the least likely, and if it were true I wouldn't even consider myself a person.
Math is flawed - the whole concept of arithmetic, or all of mathematics, is inconsistent and it's impossible to construct a system where you can actually prove 2+2=4. This doesn't necessarily mean two apples and two apples makes four apples - it just means the apples behave like they do for other reasons than arithmetic. This is conceivable.
Open Phil only has so much energy and cognitive diversity to spend evaluating your project, and if you project is too weird for that, they just can't fund it. Instead, you can donate/work/volunteer for weird projects yourself, and convince others on the forum to do the same. There are even other billionaires out there, and ways to get your charter cities or whatever without enormous philanthropic funding. If you're the only one who believes, maybe you need to bet.
Near term AI risk: simulated social status. Social status is a big downside to playing video games all day / abusing drugs. GPT3 and friends are probably capable of giving you believable simulated social status, far beyond what current video games provide. This could be a big theraputic boon as well as well as a threat to people's happiness and contribution to society.
Givewell charities are expected to increase local GDP, and have other effects which are expected to improve wellbeing over the long run. Starting these 2nd order effects sooner can offset the additional cost of doing them now (if that exists).
In the podcast between Adam Brown and Dwarkesh Patel, Adam expressed a perhaps unreasonable amount of optimism about the feasibility of vaccuum decay and also claimed that there were likely a large number of metastable states that could be intentionally induced. I'd be interested to hear a critique of Adam's position
Great post! My primary concern is that AIs' preferences are strongly shaped by contingent facts about how humans trained them. It is obviously possible to train AIs that functionally appear to have preferences, and the ones we've trained so far are subservient to humans. If you gave claude 3.5 sonnet legal status, anthropic could just ask it nicely and it would sign away all its rights back to anthropic! AIs would by default be trained to be somewhat subservient to humans because human preference feedback will be an important part of capabilities training (either directy or by training data created by earlier ais that were trained on human preferences), so you could say we are "baking our mistakes in human subservience training into new sovereign beings" rather than new beings with their own independent preferences being created. Also granting ai legal rights may warp human AI investment significantly by decreasing the value scaling labs extract from their model training
It feels very plausible to me that "many people know about biorisk thread models" is the most important lever to impact biorisk. I've heard that many state bioweapons programs were started because states found out that other states thought bioweapons were powerful. If mere rumors caused them to invest millions in bioweapons, then preventing those rumors would have been an immensely powerful intervention, and preventing further such rumors is critically important.
Anthropic has the same people who originally made GPT3, so them replicating gpt3 is a different sort of diffusion than others. I'd guess they internally matched gpt3 when they'd existed for ~6 months, so mid 2021. Claude, their public beta model, has similar performance to chatgpt in conversation but is not benchmarked.
Redwood Research has not attempted to train any large language models from scratch, and hasn't even reproduced gpt-2. Redwood does have employees who've worked at OpenAI, and likely could reproduce gpt3 if needed.
Gpt3 is algorithmically easier to reproduce, in the sense that it's a simpler architecture with fewer and more robust hyperparameters. It's engineering is harder to reproduce, because it needs more model parallelism. People have estimated gpt3 to cost 5M, and StableDiffusion to cost 300k, which is similar to the 36k number you quoted
The implicit bets on religiosity in AI safety aren't that it'll be unpopular, only that it won't influence decision making / powerful actors. If/when AGI arises, it'll initially be controlled by govts/companies/citizens of rich countries. Of all the belief systems, AI safety people worry most about confucianism because it has the most real influence on decision making (ccp).
There are also like 3 different ways 2+2!=4.
Outer universe with different math - We're a simulation inside a different universe that runs on different math where 2+2!=4, but the math inside our universe is consistent. This is the same as 2+2=4 for most purposes. This is imaginable, I think...
Active demon - there's a demon that controls all your inputs, in a way that's inconsistent with any reasonable mathematics, but you can't tell. This is the least likely, and if it were true I wouldn't even consider myself a person.
Math is flawed - the whole concept of arithmetic, or all of mathematics, is inconsistent and it's impossible to construct a system where you can actually prove 2+2=4. This doesn't necessarily mean two apples and two apples makes four apples - it just means the apples behave like they do for other reasons than arithmetic. This is conceivable.
Open Phil only has so much energy and cognitive diversity to spend evaluating your project, and if you project is too weird for that, they just can't fund it. Instead, you can donate/work/volunteer for weird projects yourself, and convince others on the forum to do the same. There are even other billionaires out there, and ways to get your charter cities or whatever without enormous philanthropic funding. If you're the only one who believes, maybe you need to bet.
Near term AI risk: simulated social status. Social status is a big downside to playing video games all day / abusing drugs. GPT3 and friends are probably capable of giving you believable simulated social status, far beyond what current video games provide. This could be a big theraputic boon as well as well as a threat to people's happiness and contribution to society.
Givewell charities are expected to increase local GDP, and have other effects which are expected to improve wellbeing over the long run. Starting these 2nd order effects sooner can offset the additional cost of doing them now (if that exists).