The text of the plan is here:
http://hk.ocmfa.gov.cn/eng/xjpzxzywshd/202507/t20250729_11679232.htm

Features a section on AI safety:
"Advancing the governance of AI safety. We need to conduct timely risk assessment of AI and propose targeted prevention and response measures to establish a widely recognized safety governance framework. We need to explore categorized and tiered management approaches, build a risk testing and evaluation system for AI, and promote the sharing of information as well as the development of emergency response of AI safety risks and threats. We need to improve data security and personal information protection standards, and strengthen the management of data security in processes such as the collection of training data and model generation. We need to increase investment in technological research and development, implement secure development standards, and enhance the interpretability, transparency, and safety of AI. We need to explore traceability management systems for AI services to prevent the misuse and abuse of AI technologies. We need to advocate for the establishment of open platforms to share best practices and promote international cooperation on AI safety governance worldwide."

I will say though that I really enjoyed this - and it definitely imparts the, ah, appropriate degree of scepticism i might want potential applicants to have ;)

https://www.youtube.com/shorts/_DxM15ZuvG4 

I've a similar concern to Geoffrey's.

When I clicked on the video last week, there's a prominent link to careers, then jobs. At the time, 3 of the top 5 were at AGI companies (Anthropic, OpenAI, GDM). I eventually found the 'should you work at AGI labs?' link, but it was much less obvious. This is the funnel that tens of thousands of people will be following (assuming 1% of people watching the video consider a change of career).

80K has for a long time pushed safety & governance at AGI companies as a top career path. While some of the safety work may have been very safety-dominant, a lot of it has in practice helped companies ship product, and advanced capabilities in doing so (think RLHF etc - see https://arxiv.org/pdf/2312.08039 for more discussion). This is inevitably a higher likelihood in a commercial setting than in e.g. academia.

Policy and governance roles have done some good, but have in practice also contributed to misplaced trust in companies and greater support for e.g. self-governance than might otherwise have been the case. Anecdotally, I became more trusting of OpenAI after working with their researchers on Towards Trustworthy AI (https://arxiv.org/pdf/2004.07213), in light of them individually signing onto (and in some cases proposing) mechanisms such as whistleblower mechanisms and other independent oversight mechanisms. At the same time unbeknownst to them, OpenAI leadership were building clauses into their contracts to strip them of their equity if they criticised the company on leaving. I expect the act by safety-focused academics like myself of coauthoring the report with openAI policy people will have also had the effect of increasing the perceptions of trustworthiness of OpenAI. 

By now, almost everyone concerned about safety seems to have left OpenAI, often citing concerns over the ethics, safety-committedness and responsibilit-committedness of leadership. This includes everyone on Towards Trustworthy AI, and I expect many of the people funneled there by 80K. ( I get the impression from speaking to some of them that they feel they were being used as 'useful idiots'). One of the people who left over concerns was Daniel Kokatajlo himself - who indeed had to give up 85% of his family's net worth (temporarily, I believe) in order to be critical of OpenAI. 

Another consequence of this funnel is that it's contributed to the atrophy of the academic AI safety and governance pipeline, and loss of interest in supporting this part of the space by funders ('isn't the most exciting work happening inside the companies anyway?'). The most ethically-motivated people, who might otherwise have taken the hit of academic salaries and precarity, had a green light to go for the companies. This has contributed to the atrophy of the independent critical role, and government advisory role, that academia could have played in frontier AI governance.

There's a lot more worth reflecting on than is captured in the 'should you work at AI labs/companies' article. While I've focused on OpenAI to be concrete here, the underlying issues apply to some degree across frontier AI companies.

And there is a lot more that a properly reflective 80K could be doing here. E.g.

• Don't have the tech company roles coming up at the very top the week you release your video
• Have these high-risk roles flagged somehow, or in a separate category
• Make it so the roles are accessed through the 'should you work at tech companies' article or in conjunction with it, so that people are more likely to read it 

• Heck, you could even have a little survey to answer before accessing these high risk roles, like when you're investing in a high-risk asset

  (this is all just off the top of my head, I'm sure there are better suggestions).

It's long been part of 80k's strategy to put people in high-consequence positions in the hope they can do good, and exert influence around them. It is a high-risk strategy with pretty big potential downsides. There have now been multiple instances in which this plan has been shown not to survive contact with the kind of highly agentic, skilled-at-wielding-power individuals who end up in CEO-and-similar positions (I can think of a couple of Sams, for instance). If 80k is going to be pointing a lot of in-expectation young and inexperienced people in these directions, it might benefit from being a little more reflective about how it does it.

I don't think it's impossible to do good from within companies, but I do expect you need to be skilful,  sophisticated, and somewhat experienced. These are AGI companies. Their goal is to build AGI sooner than their competitors build AGI. Their leadership are extremely focused on this. Whether the role is in governance, or safety, it's reasonable to expect ultimately that you as an employee will be expected to help them do that (certainly not hinder them)

Really inspiring. Grateful to you for writing this up.

Thanks Rían, I appreciate it. And to be fair, this is from my perspective as much a me thing as it is an Oli thing. Like, I don't think the global optimal solution is an EA forum that's a cuddly little safe space for me. But we all have to make the tradeoffs that make most sense for us individually, and this kind of thing is costly for me.

One other observation that might explain some of the different perceptions on 'blame' here.

I don't think Oxford's bureaucracy/administration is good, and I think it did behave very badly at points*. But overall, I don't think Oxford's bureaucracy/behaviour was a long way outside what you would expect for the reference class of thousand-year-old-institutions with >10,000 employees. And Nick knew that was what it was, chose to be situated there, and did benefit (particularly in the early days) from the reputation boost. I think there is some reasonable expectation that having made that choice, he would put some effort into either figuring out how to operate effectively within its constraints, or take it somewhere else.

(*it did at point have the feeling of grinding inevitability of a failing marriage, where beyond a certain point everything one side did was perceived in the worst light and with maximal irritation by the other side, going in both directions, which contributed to bad behaviour I think).

It wasn't carefully chosen. It was the term used by the commenter I was replying to. I was a little frustrated, because it was another example of a truth-seeking enquiry by Milena getting pushed down the track of only-considering-answers-in-which-all-the-agency/wrongness-is-on-the-university side (including some pretty unpleasant options relating to people I'd worked with ('parasitic egregore/siphon money').

>Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?

So I just did copy and paste on the most relevant phrase, but flipped it. Bit blunter and more smart-arse than I would normally be (as you've presumably seen from my writing, I normally caveat to a probably-tedious degree), but I was finding it hard to challenge the simplistic fhi-good-uni-bad narrative. It was one line, I didn't think much about it.

I remain of the view that the claim is true/a reasonable interpretation, but de novo / in a different context I would have phrased differently.

Deleting this because on re-reading I think I'm just repeating myself, but in a more annoyed way. Thanks for checking with other people, I'll leave it at that.

Thank you. I’m grateful you checked with other people. Yes, I do think there is some history rewriting and mythologising going on here compared to my own memory of how things were, and this bothers me because I think the truth does matter.

There is a very real sense in which Nick had a pretty sweet setup at Oxford, in terms of having the power and influence to do an unusual thing. And there were a bunch of people around him working insanely hard to help make that happen. I also do think there is a degree to which, yes, Nick blew it. I don’t really want to dwell on this because it feels a bit like bad-mouthing FHI at its funeral. And it’s not the whole story. But it is a source of some frustration to me, because I did not have that position of power and influence in trying to do somewhat similar things, and have spent years banging my head against various walls, and I would have liked to see the FHI story go well. That is not to say i would have made all the right decisions had I had that power and influence (I'm sure I would not), or that I did make all the right decisions in the situations I was in. But I still think I am within my rights to have a view as someone who was actually there for 4 years doing the thing.

As well as all the good stuff, Nick was unusually pedantic and stubborn about a huge range of things, many of them (to my lights) relatively unimportant, from expensive cups to fonts to refusing to follow processes that would not realistically have impeded FHI’s intellectual activities to follow. And so many things would get framed as a battle to be won against the Faculty/University, where a bit of cooperation would have gone a long way. Play stupid games, win stupid prizes. It sucked up huge amounts of FHI time, huge amounts of Faculty time, and huge amounts of social capital, which made it harder to stand ground/get cooperation on the stuff that mattered. And it compounded over time. You don’t trust me, but do some digging and I think you’ll find it. There were two sides to this thing.

All of this I am saying based on a lot of context and experience at FHI. Rather than question or challenge me on my original point, your immediate reaction was two multi-paragraph posts seemingly aimed at publicly discrediting me in every way - repeatedly saying that you don’t consider me trustworthy; that my career’s work has been bad for the world and therefore my takes shouldn’t be listened to; that I am some sort of malign intellectual influence who is somehow responsible for intellectual attacks on other people*. To me this doesn't look like truth-seeking behaviour. It looks more like an effort to discredit a person who challenged the favoured narrative.

Even after being told by someone who actually was at FHI that I was a big part of making it work, your response seems to imply that if you did some digging into my time at FHI, you would find that actually my influence turned out to be negative and harmful. Well do that digging, see if that’s what you find. I worked damn hard there, took personal risks, and did good work. You want to claim that’s false, you can show some evidence.

*And no, I don’t think these things are equivalently harsh. I criticised Nick for bureaucratic mistakes. Nobody respects Nick primarily for his administrative/bureaucratic relationship skills. They respect him for other things, which I have praised on other occasions. Your personal go at me targeted pretty much every aspect of why people might respect me or consider me worth listening to. That is fundamentally different.

For what it's worth, I'm (at least partly) sympathetic to Oli's position here. If nothing else, from my end I'm not confident that the combined time usage of:

[Oli producing book-length critique of CSER/Leverhulme, or me personally, depending] +
[me producing presumably book-length response] +
[further back and forth] +
[a whole lot of forum readers trying to unpick the disagreements]

is overall worth it, particularly given (a) it seems likely to me there are some worldview/cultural differences that would take time to unpick and (b) I will be limited in what I can say on certain matters by professional constraints/norms.

 I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start:
https://forum.effectivealtruism.org/posts/an9GrNXrdMwBJpHeC/long-term-future-fund-august-2019-grant-recommendations-1#Addendum__Thoughts_on_a_Strategy_Article_by_the_Leadership_of_Leverhulme_CFI_and_CSER

And as to the claim "I also wouldn't be surprised if Sean's takes were ultimately responsible for a good chunk of associated pressure and attacks on people's intellectual integrity"  it seems like some of this is based on my online comments/writing. I don't believe I've ever deleted anything on the EA forum, LW, or very much on twitter/linkedin (the online mediums I use), my papers are all online, and so again a decent place to start is to search for my username and come to their own conclusions.

And I guess I should just say directly. I do wish it were possible to raise (specific) critical points on matter like faculty relations where I have some direct insight and discuss these, without immediate escalation to counterclaims that my career’s work has been bad for the world, that I am not to be trusted, and and that my influence is somehow responsible for attacks on people’s intellectual integrity. It’s very stressful and upsetting.

I suffer from (mild) social anxiety. That is not uncommon. This kind of very forceful interaction is valuable for some people but is difficult and costly for others to engage with. I am going to engage less with EA forum/LW as a result of this and a few similar interactions, and I am especially going to be more hesitant to be critical of EA/LW sacred cows. I imagine, given what you have said about my takes, that this will be positive from your perspective. So be it. But you might also consider the effect it will have on others who might be psychologically similar, and whose takes you might consider more valuable.