SOH

Sean_o_h

Research Programme Director @ University of Cambridge
3929 karmaJoined

Bio

I direct the AI:Futures and Responsibility Programme (https://www.ai-far.org/) at the University of Cambridge, which works on AI strategy, safety and governance. I also work on global catastrophic risks with the Centre for the Study of Existential Risk and AI strategy/policy with the Centre for the Future of Intelligence.

Comments
218

Yes, me too on Mick's points.

This is  a slow-burn solution, but the most effective support and rebuttals will come from people who aren't EAs, but are just fair/principled, and have had enough exposure to EA to know when attacks are unfair. E.g. See Dean Ball this week. So the more surface area EAs can create with those sorts of people, the better the position EA is in. For example, I think Andy Masley's datacenter water use posts created a lot of surface ara with such people and has been better for the EA 'brand' than any specific rebuttal.

(A part of this strategy involves, as a general principle, "behave with as much dignity, integrity and fairness as possible, even when others aren't). (Admittedly my own responses usually involve gently poking fun, but I do try to stay good natured).

Thank you for your important work! You're probably already speaking to them, but in this broader community and adjacent, I recommend the work of:

For 'role of Africa in frontier AI safety' isseues 

For more locally-focused safety work, the African Hub on AI Safety, peace and security

(among others, of course). 

This is appalling to read and I'm so sorry Fran. I don't know what else to say, but I felt I should at least say that.

Maybe, but this also seems like the kind of extremely broadly salient thing where it would be more difficult for EAs to make a big difference on the margins with their work and funding compared to 'regular' EA causes. (though people should also focus time and money on things important to them)

Seems worth someone tracking who the major shareholdrs are and how many voting rights they hold - e.g. I'd bet the house that Jaan Tallinn would be against this, so it'd be good to know if there are enough to support him to ward against possibilities like this.

I agree especially with the first of these (third is outside my scope of expertise). There's a lot of work that feels chronically underserved and unsupported in 'implementation-focused policy advocacy and communication'.

E.g. standards are correctly mentioned. Participation in standards development processes is an unsexy, frequently tedious and time-consuming processes. The big companies have people whose day jobs is basically sitting in standards bodies like ISO making sure the eventual standards reflect company priorities. AI safety/governance has a few people fitting it in alongside a million other priorities.

e.g. the EU AI office is still struggling to staff the AI Office, one of the most important safety implementation offices out there. e.g. I serve on an OECD AI working group, an international working group with WIC, have served on GPAI, PAI, and regularly advise UN and national governments. You can make a huge difference on these things especially if you have the time to develop proposals/recommendations and follow up between meetings. But I see the same faces from academia & cvil society at all of these - all of us exhausted, trying to fit in as much as we can, alongside research, management, fundraising, teaching+student mentorship + essays (for the academics). 

Some of this is that it takes time as an independent to reach the level of seniority/recognition to be invited to these working groups. But my impression from being on funding review panels that many of the people who are well-placed to do so still have an uphill battle to get funding for themselves or the support they need. It helps if you've had something flashy and obviously impactful (e.g. AI-2027) but there's a ton of behind-the-scenes work (that I think is much harder for funders to assess, and sometimes harder for philanthropists to get excited about) that an ecosystem with a chance of actually steering this properly and acting as the necessary counterweight to commercial pressures needs. Time and regular participation at national government level (a bunch of them!) plus US, EU, UN, OECD, G7/G20/G77, Africa, Belt&Road/ANSO, GPAI, ITU, ISO, things like Singapore SCAI & much more. Great opportunities for funders (including small funders).

Useful data and analysis thanks, though I'd note that from a TAI/AI risk-focused perspective I would expect the non-safety figures to overcount for some of these orgs. E.g. CFI (where I work) is in there at 25 FTE, but that covers a very broad range of AI governance/ethics/humanities topics, where only a subset (maybe a quarter?) would be specifically relevant to TAI governance (specifically a big chunk of Kinds of Intelligence that mainly does technical evaluation/benchmarking work, but advises policy on the basis of this, and the AI:FAR group). I would expect similar with some of the other 'broader' groups e.g. Ada.

Also in both categories I don't follow the rationale for including GDM but not the other frontier companies with safety/governance teams e.g. Anthropic, OpenAI, xAI (admittedly more minimal). I can see a rationale for including all or none of them.

The text of the plan is here:
http://hk.ocmfa.gov.cn/eng/xjpzxzywshd/202507/t20250729_11679232.htm

Features a section on AI safety:
"Advancing the governance of AI safety. We need to conduct timely risk assessment of AI and propose targeted prevention and response measures to establish a widely recognized safety governance framework. We need to explore categorized and tiered management approaches, build a risk testing and evaluation system for AI, and promote the sharing of information as well as the development of emergency response of AI safety risks and threats. We need to improve data security and personal information protection standards, and strengthen the management of data security in processes such as the collection of training data and model generation. We need to increase investment in technological research and development, implement secure development standards, and enhance the interpretability, transparency, and safety of AI. We need to explore traceability management systems for AI services to prevent the misuse and abuse of AI technologies. We need to advocate for the establishment of open platforms to share best practices and promote international cooperation on AI safety governance worldwide."

I will say though that I really enjoyed this - and it definitely imparts the, ah, appropriate degree of scepticism i might want potential applicants to have ;)

https://www.youtube.com/shorts/_DxM15ZuvG4 

Load more