Heramb Podar's Quick takes

We as a movement do a terrible job of communicating just how hard it might be to get a job in AI Safety and honestly cause people to anchor/rely too much on EA resources which sets unrealistic expectations and also not being fair 

At this point, we need an 80k page on "What to do after leaving Open AI"

1. Don't start another AI safety lab

A lot of policy research seems to be written with an agenda in mind to shape the narrative. And this kind of destroys the point of policy research which is supposed to inform stakeholders and not actively convince or really nudge them.

This might cause polarization in some topics and is in itself, probably snatching legitimacy away from the space.

I have seen similar concerning parallels in the non-profit space, where some third-sector actors endorse/do things which they see as being good but destroys trust in the whole space.

This gives me scary unilaterist's curse vibes..

It's interesting how we also have scope neglect of key historical events:

Death Toll: Siege of Athens in 86 BC ~ People killed in Hiroshima <<< Battle of Stalingrad 

I find the Biden chip export controls a step in the right direction, and it also made me update my world model of compute governance being an impactful lever. However, I am concerned that our goals aren't aligned with theirs; US policymakers' incentive right now is to curb China's tech growth and fun trade war reasons, not pause AI.

This optimization for different incentives is probably going to create some split between US policymakers and AI safety folks as time goes on.

It also makes China more likely to treat this as a tech race which sets up interesting competitive race dynamics between the US and China which I don't see talked about enough. 

After talking and working for some time with non-EA organisations in the AI Policy space, I believe that we need to give more credence to the here-and-now of AI safety policy as well to get the attention of policymakers and get our foot in the door. That also gives us space to collaborate with other think tanks and organisations outside of the x-risk space that are proactive and committed to AI policy. Right now, a lot of those people also see X-risks as being fringe and radical(and these are people who are supposed to be on our side).

Governments tend to move slowly, with due process, and in small increments(think, "We are going to first maybe do some risk monitoring, only then auditing"). Policymakers are only visionaries with horizons until the end of their terms(hmm, no surprise). Usually, broad strokes in policy require precedents of a similar size for it to be feasible within a policymakers' agenda and the Overton window. 

Every group that comes to a policy meeting thinks that their agenda item is the most pressing because, by definition, most of the time, contacting and getting meetings with policymakers means that you are proactive and have done your homework.

I want to see more EAs respond to Public Voice Opportunities, for instance- something I rarely hear on the EA forum or via EA channels/material. 

I feel like being exposed early on to longer form GovAI-type reports has made me set the bar high for writing my thoughts out in short form, which really sucks in terms of an output standpoint.

People in EA end up optimizing for EA credentials so they can virtue signal to grantmakers, but grantmakers would probably like people to scope out non-EA opportunities because that allows us to introduce unknown people to the concerns we have

I see way too many people confusing movement with progress in the policy space. 

There can be a lot of drafts becoming bills with still significant room for regulatory capture in the specifics, which will be decided later on. Take risk levels, for instance, which are subjective - lots of legal leeway for companies to exploit. 

For policy recommendations, put forth things that actually build on or move the status quo.

For example, recommending a "National Youth Council" without a mandate can be a uniquely bad idea- instead of ignoring your (usually inactive) youth org, now, policymakers will ignore the Council of (usually inactive)youth orgs all while you(the actually proactive person), walk away with the false notion of a job well done.

The problem with AI safety policy is that if we don't specify and attempt to answer the technical concerns then someone else will and safety wash the concerns away.

CSOs need to understand what they themselves mean when they say "explainable" and "algorithmic transparency."

It's important to think about the policy space from a meta-level incentives/factors that might get in the way of having an impact, such as making AI safer.

One I heard today was that policy people thrive in moments of regulatory uncertainty, while this is bad for companies.

Communicating by keeping human rights at the centre of AI Policy discussion is extremely underappreciated.
For e.g., the UN Human Rights chief in 2021 called for a moratorium on the sale and use of artificial intelligence (AI) systems until adequate safeguards are put in place.

Respect for human rights is a well-established central norm; leverage it  

The real danger isn't just from AI getting better- it's from it getting good enough that humans start over-relying on it and offloading tasks to it. 

Remember that Petrov had automatic detection systems, too; he just independently came to the conclusion not to fire nukes back.

Everyone who seems to be writing policy papers/ doing technical work seems to be keeping generative AI at the back of their mind, when framing their work or impact. 

This narrow-eyed focus on gen AI might almost certainly be net-negative for us- unknowingly or unintentionally ignoring ripple effects of the gen AI boom in other fields (like robotics companies getting more funding leading to more capabilities, and that leads to new types of risks).

And guess who benefits if we do end up getting good evals/standards in place for gen AI? It seems to me companies/investors are clear winners because we have to go back to the drawing board and now advocate for the same kind of stuff for robotics or a different kind of AI use-case/type all while the development/capability cycles keep maturing.

We seem to be in whack-a-mole territory now because of the overton window shifting for investors.

I don't think we have a good answer to what happens after we do auditing of an AI model and find something wrong.

Given that our current understanding of AI's internal workings is at least a generation behind, it's not exactly like we can isolate what mechanism is causing certain behaviours. (Would really appreciate any input here- I see very little to no discussion on this in governance papers; it's almost as if policy folks are oblivious to the technical hurdles which await working groups)

At some point, one has to ask "Am I in the cause area because I am an EA or am in EA because I am in the cause area?" 

With open-source models being released and on ramps to downstream innovation lowering, the safety challenges may not be a single threshold but rather an ongoing, iterative cat-and-mouse game.

Just underscores the importance of people in the policy/safety field thinking far ahead

It's frankly quite concerning that usually technical specifications are only worked on by Working Groups after high-level qualitative goals are set by policymakers- seems to open a can of worms for different interpretations and safety washing.

Agreeing on building safe, trustworthy, and human-centric AI is akin to making an open call for a DIY definitions for different regulatory environments.