This is a special post for quick takes by tylermjohn. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
AIxBio looks pretty bad and it would be great to see more people work on it
- We're pretty close to having a country of virologists in a data center with AI models that can give detailed and accurate instructions for all steps of a biological attack — with recent reasoning models, we might have this already
- These models have safeguards but they're trivial to overcome — Pliny the Liberator manages to jailbreak every new model within 24 hours and open sources the jailbreaks
- Open source will continue to be just a few months behind the frontier given distillation and amplification, and these can be fine-tuned to remove safeguards in minutes for less than $50
- People say it's hard to actually execute the biology work, but I don't see any bottlenecks to bioweapon production that can't be done by a bio undergrad with limitless scientific knowledge; on my current understanding, the bottlenecks are not manual dexterity bottlenecks like playing a violin which require years of practice, they are knowledge bottlenecks
- Bio supply chain controls that make it harder to get ingredients aren't working and aren't on track to work
- So it seems like we're very close to democratizing (even bespoke) bioweapons. When I talk to bio experts about this they often reassure me that few people want to conduct a biological attack, but I haven't seen much analysis on this and it seems hard to be highly confident.
While we gear up for a bioweapon democracy it seems that there are very few people working on worst-case bio, and most of the people working on it are working on access controls and evaluations. But I don't expect access controls to succeed, and I expect evaluations to mostly be useful for scaring politicians, due in part to the open source issue meaning we just can't give frontier models robust safeguards. The most likely thing to actually work is biodefense.
I suspect that too many people working on GCR have moved into working on AI alignment and reliability issues and too few are working on bio. I suspect there are bad incentives, given that AI is the new technology frontier and working with AI is good career capital, and given that AI work is higher status.
When I talk to people at the frontier of biosecurity, I learn that there's a clear plan and funding available, but the work is bottlenecked by entrepreneurial people who can pick up a big project and execute on it autonomously — these people don't even need a bio background. On my current guess, the next 3-5 such people who are ambivalent about what to do should go into bio rather than AI, in part because AI seems to be more bottlenecked by less generalist skills, like machine learning, communications, and diplomacy.
I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I've seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people's value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren't empowered with AI tools (e.g. CIPs and CAIS'S alignment to the written output of citizens' assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it.
For deeper motivation see here.
Is the alignment motivation distinct from just using AI to solve general bargaining problems?
I don't know! It's possible that you can just solve a bargain and then align AI to that, like you can align AI to citizens assemblies. I want to be pitched.
I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I've seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people's value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren't empowered with AI tools (e.g. CIPs and CAIS'S alignment to the written output of citizens' assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it.
For deeper motivation see here.
I don't know! It's possible that you can just solve a bargain and then align AI to that, like you can align AI to citizens assemblies. I want to be pitched.