EA Forum Bot Site
EA Forum

Sophie Kim

AI Policy Researcher @ Cambridge Biosecurity Hub
17 karmaJoined thecounterfactual.substack.com

Bio

Hi! I'm Sophie, a gap year student at Stanford focusing on AI governance for catastrophic risk mitigation. I write a Substack, The Counterfactual -- https://thecounterfactual.substack.com/ -- and I'm currently an ERA Fellow on the AIxBiosecurity track working on tiered access controls for LLMs/cloud labs and interventions to stop China from open-weighting their models.

How others can help me

I'm in the early stages of co-founding a donor advisory initiative focused on neglected areas within AI safety, and would appreciate connections with anyone in the grantmaking/fundraising space!

How I can help others

Happy to chat / might be helpful if you're interested in learning more about:

  • AIxBiosecurity: tractable problems, current regulatory gaps, recent uplift studies, policy proposals
  • Mutually Assured AI Malfunction (MAIM) from Superintelligence Strategy
  • Risks associated with open-weight models (abliteration, safeguards, fine-tuning)

Comments
1

LLMs Didn’t Help Novices Build Viruses. That Might Be the Wrong Question.
Sophie Kim3
2
0

Thanks for your comment! The base rate argument is reasonable, and I agree that in absolute numbers there are far more potential novice threat actors than expert ones.

But I think timing matters. Expert uplift is a leading indicator-- by the time a model is good enough to meaningfully help a novice with no bio background complete a reverse genetics workflow, it's been helping people with partial expertise clear their specific bottlenecks for much longer. So I'd frame it less as "novice uplift doesn't matter" and more as "if we wait for novice uplift to show up in studies, we've already missed the window where expert uplift became dangerous." Measuring expert uplift first gives us an earlier warning signal.

Also worth noting that the "crazy expert" bucket isn't as empty as the simple model suggests; Aum Shinrikyo's bio program was led by people with graduate degrees, and the 2001 anthrax attacks were likely carried out by a senior USAMRIID researcher. The base rate of "expert with intent" may be low, but it's not zero, and the expected damage per attempt is much higher.

Reply