"I'd love to have a call and catch up in any case! I'm curious whether you already have an opinion on whether places like DeepMind will be interested in paying for evals like the two types mentioned here (character and backdoors)."
Quickly: 1. I'll schedule a call sometime, once I understand my schedule a bit better. 2. I have incredibly little info on DeepMind right now. As you may imagine, personally I'm excited about a lot of these areas, but I can't speak for DeepMind.
"Why did Guesstimate/Squiggle as for-profit not work out?" I've been asked this question a bunch of times before, happy to give some quick thoughts here. Most of my answer is here.
If it were the case that it seemed very easy to make these as startups, and ideally in which there were some clear cofounders who could handle the business side, I think the for-profit side would have been more promising.
In my case: 1. I realized this was a niche/small space, that getting a large market would likely mean having to pivot. 2. I wasn't sure what might take off. I think Squiggle has a lot going for it, but am not sure if the value proposition is strong enough for a large company now, especially as LLMs have been getting much more powerful (changing the trade-offs around decision tools). 3. My main goal was helping effective altruist / rationalist researchers, and I suspected that focusing much more on commercialization would have hurt that. 4. Related to (3), I just found myself much more motivated to go after altruists/nonprofits than to go after standard enterprise deals
Quickly (and to give context to others reading this):
I think that 'taking over QURI' makes the most sense for people who want to take over Guesstimate/Squiggle, and even then I'd default to being hesitant about that.
QURI doesn't really have much to take over, especially for research. There's a simple brand, there's a niche community that knows its history. There aren't staff or much IP at all. It's unclear to me what the value is of doing related work 'inside QURI' vs. making a new org.
"$1m in funding over ~2 years" -> I'd personally be excited to see what you do with this, but I'm unsure how easy it would be to raise this. In our experience fundraising was definitely not trivial, and we've mostly been asking for much less than $1m (though year by year). Most EA funders now are focused on other dedicated areas like technical AI safety or bio risk, I think our category of work has been more challenging. For a while the FTX Future Fund was interested in these areas, then later Coefficient Giving had a Forecasting fund, but both closed fairly quickly. It's quite possible that Anthropic money and similar will change the situation, but I haven't seen that yet. Related, I'd recommend very much deeply understanding the funding climate before making an incubator, as I think it's easy to create a bunch of orgs that basically have no where to go.
In terms of incubators, some reading this might be interested in Surplus! It's for-profit only, but definitely related to our work.
If you want a 501(c)(3) sponsor, I think there's a lot of good options. QURI was itself sponsored by Rethink Priorities for much of its life.
Multipolar worlds will compete away >90% of net value that would otherwise be preserved
If they're halfway-reasonable, they could use smart AIs to negotiate for them. Big question is who will control these worlds.
I think it's likely humans will settle on AI solutions that lose 90% of the value vs. my optimal solution, but that's very much a values question, not a multipolar vs. unipolar question.
AI alignment to humans will in practice avoid moral catastrophes to animals
I expect certain conservative/religious communities to lock-in values that could be really bad. But I'd expect that better tech can remove say ~90% of the damages? But this is very hand-wavy.
Shutting down one of these companies might cost, say, a trillion dollars and lost investor/employee value. And I think that the real risky 'frontier AI' might only be a portion of their work.
I could very much see an argument for them to stop a lot of the key frontier work and then move to more conservative engineering efforts, for instance. I think that there's a large variety of space to do around AI development and AI safety, it seems easy for me to imagine large changes in direction that could still have some market value but much less risks.
Sounds like arduous but important work. Good luck with the rest of it!
Thanks for doing this! I found this video very interesting. This area seems pretty critical to get right, quite happy to have more insight on it.
I had a bunch of unpublished drafts and notes in Google Docs. I just released them here, using Claude to help organize and summarize.
https://priors.quantifieduncertainty.org/
I imagine some here would find them interesting! Most are fairly old, starting from 2017, when I joined FHI.
"I'd love to have a call and catch up in any case! I'm curious whether you already have an opinion on whether places like DeepMind will be interested in paying for evals like the two types mentioned here (character and backdoors)."
Quickly:
1. I'll schedule a call sometime, once I understand my schedule a bit better.
2. I have incredibly little info on DeepMind right now. As you may imagine, personally I'm excited about a lot of these areas, but I can't speak for DeepMind.
"Why did Guesstimate/Squiggle as for-profit not work out?"
I've been asked this question a bunch of times before, happy to give some quick thoughts here. Most of my answer is here.
If it were the case that it seemed very easy to make these as startups, and ideally in which there were some clear cofounders who could handle the business side, I think the for-profit side would have been more promising.
In my case:
1. I realized this was a niche/small space, that getting a large market would likely mean having to pivot.
2. I wasn't sure what might take off. I think Squiggle has a lot going for it, but am not sure if the value proposition is strong enough for a large company now, especially as LLMs have been getting much more powerful (changing the trade-offs around decision tools).
3. My main goal was helping effective altruist / rationalist researchers, and I suspected that focusing much more on commercialization would have hurt that.
4. Related to (3), I just found myself much more motivated to go after altruists/nonprofits than to go after standard enterprise deals
Hi Dawn!
I'd be happy to discuss this.
Quickly (and to give context to others reading this):
Many congrats on the success! That seems fantastic.
If they're halfway-reasonable, they could use smart AIs to negotiate for them. Big question is who will control these worlds.
I think it's likely humans will settle on AI solutions that lose 90% of the value vs. my optimal solution, but that's very much a values question, not a multipolar vs. unipolar question.
I expect certain conservative/religious communities to lock-in values that could be really bad. But I'd expect that better tech can remove say ~90% of the damages? But this is very hand-wavy.
This seems like an very coarse take to me.
Shutting down one of these companies might cost, say, a trillion dollars and lost investor/employee value. And I think that the real risky 'frontier AI' might only be a portion of their work.
I could very much see an argument for them to stop a lot of the key frontier work and then move to more conservative engineering efforts, for instance. I think that there's a large variety of space to do around AI development and AI safety, it seems easy for me to imagine large changes in direction that could still have some market value but much less risks.