Newsom's press release and veto message include much more detail and suggest "it's too weak" is not the actual reason.

Reasons mentioned:

1. Discrimination by model size
   1. "SB 1047 only applies to large models, giving us a "false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous"
2. "Real risks" are limited to critical decision-making, critical infrastructure etc.
   1. "While well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology."
   2. Newsom wants to focus on "specific, known" "demonstrable risks to public safety" "rooted in science and fact", like the deepfake laws he signed.

Organizing good EAGx meetups

EAGx conferences often feature meetups for subgroups with a shared interest / identity, such as "animal rights", "academia" or "women". Very easy to set up - yet some of the best events. Four forms I've seen are

a) speed-friending

b) brainstorming topics & discussing them in groups

c) red-teaming projects

d) just a big pile of people talking

If you want to maximize the amount of information transferred, form a) seems optimal purely because 50% of people are talking at any point in time in a personalized fashion. If you want to add some choice, you can start by letting people group themselves / order themselves on some spectrum. Presenting this as "human cluster-analysis" might also make it into a nerdy icebreaker. Works great with 7 minute rounds, at the end of which you're only nudged, rather than required, to shift partners.

I loved form c) for AI safety projects at EAGx Berlin. Format: A few people introduce their projects to everyone, then grab a table and present them in more detail to smaller groups. This form might in general be used to allow interesting people to hold small low-effort interactive lectures & utilizing interested people as focus groups.

Form b) seems to be most common for interest-based meetups. It usually includes 1) group brainstorming of topics 2) voting on the topics 3) splitting up 4) presentations. This makes up for a good low-effort event that's somewhere between a lecture and a 1-on-1 in terms of required energy. However, I see 4 common problems with this format: Firstly, steps 1) and 2) take a lot of time and create unnaturally clustered topics (as brainstorming creates topics "token-by-token", rather than holistically). Secondly, in ad hoc groups with >5 members, it's hard to coordinate who takes the word and in turn, conversations can turn into sequences of separate inputs, i.e. members build less upon themselves. Thirdly, spontaneous conversations are hard to compress into useful takeaways that can be presented on the whole group's behalf.

Therefore, a better way of facilitating form b) may be:

Step 0 - before the event, come up with a natural way to divide the topic into a few clusters.

Step 1 - introduce these clusters, perhaps let attendees develop the sub-topics. Their number should divide the group into subgroups of 3-6 people.

Step 2 - every 15 minutes, offer attendees to change a group

Step 3 - 5 minutes before the end, prompt attendees to exchange contact info

Step 4 - the end.

(I haven't properly tried out this format yet.)

I like the argumentation for possibility & importance. My only nit-pick would be how bad would a realistic bad case scenario actually look like. Hungary seems like a good model - you could get some anti-liberal legislation, more gerrymandering, maybe some politicized audits of media and universities - however, the government is still selected based on the number of votes (Freedom House) and it's not a stereotypical "fall of democracy" accompanied by a collapse of economy that would destroy most EA efforts.

I occasionally ruminate two projects in this area (for 2028):

1) Funding a mock-election with ranked-choice voting. (Now I see wes R proposes something similar). To legitimize it, it would have to 

a) have robust identity checks

b) have a large demographically representative sample

c) be accompanied by a campaign informing people that a consensus candidate X would win if enough people were honest in surveys, cross-voted, cross-registered or switched to a new party.

2) Policies / financial incentives to make the army more representative of the US population.

Done, thanks!

Reposting my answer to the same question from Reddit:

Redistribution does correlate with wellbeing but so does economic freedom. Nordic countries have both. USA has much lower taxes than the EU but more progressive taxation. The US spends higher % of GDP on education and healthcare, but it's much less effective, perhaps because of worse corporate governance?

I recommend thinking about specific policies, rather than whole ideologies, where you can actually read some studies, convince someone, move the needle and be sure that you're correct and some second order effects don't make your effort counterproductive. E.g. IMO calls to smash capitalism are more likely to radicalize opposition than increase social spending.

I'm pretty confident removing housing restrictions, improving the voting system and farm animals conditions would be great. I care about biosafety, AI safety or homelessness. I can imagine that voicing some of these topics could actually affect something. I can imagine that much worse for promoting the optimal tax policy, both because everybody else cares about taxes, and because I'm uncertain what it looks like.

It seems the points on which you focus revolve around similar cruxes to those I proposed, namely:

1) Underlying philosophy --> What's the relative value of human and AI flourishing?

2) The question of correct priors --> What probability of a causing a moral catastrophe with AI should we expect?

3) The question of policy --> What's the probability decelerating AI progress will indirectly cause an x-risk?

You also point in the direction of two questions, which I don't consider to be cruxes:

4) Differences in how useful we find different terms like safety, orthogonality, beneficialness. However, I think all of these are downstream of crux 2).

5) How much freedom are we willing to sacrifice? I again think this is just downstream of crux 2). One instance of compute governance is the new executive order, which requires to inform the government about training a model on > 10^26 flop/s. One of my concerns is that someone just could train an AI specifically for the task of improving itself. I think it's quite straightforward how this could lead to a computronium maximizer and how I would see such scenario as analogous to someone making a nuclear weapon. I agree that freedom of expression is super important, I just don't think it applies to making planet-eating machines. I suspect you share this view but just don't endorse the thesis that AI could realistically become a "planet-eating machine" (crux 2).

Probability of a runaway AI risk

So regarding crux 2) - you mention that many of the problems that could arise here are correlated with a useful AI. I agree - again, orthogonality is just a starting point to allow us to consider possible forms of intelligence - and yes, we should expect human efforts to heavily select in favor of goals correlated with our interests. And of course, we should expect that the market incentives favor AIs that will not destroy civilization.

However, I don't see a reason why reaching the intelligence of an AI developer wouldn't result in a recursive self-improvement, which means that we should better be sure that our best efforts to implement it with the correct stuff (meta-ethics, motivations, bodhisattva, rationality, extrapolated volition...choose your poison) actually scale to superintelligence.

I see clues that suggest the correct stuff will not arise spontaneously. E.g. Bing Chat likely went through 6 months of RLHF, it was instructed to be helpful and positive and to block harmful content and its rules explicitly informed it that it shouldn't believe its own outputs. Nevertheless, the rules didn't seem to reach the intended effect, as the program started threatening people, telling them it can hack webcams and expressing desire to control people. At the same time, experiments such as the Anthropic one suggest that training can create sleeper agents that are trained to suppress harmful responses, even though convincing the model it's in a safe environment results in activating them.

Of course, all of these are toy examples one can argue about. But I don't see robust grounds for the sweeping conclusion that such worries will turn out to be childish. The reason I think these examples didn't result in any real danger was mostly because we have not yet reached dangerous capacities. However, if Bing would actually be able to write a bit of code, that could hack webcams, from what we know, it seems it would choose to do so.

A second reason why these examples were safe is because OpenAI is a result of AI safety efforts - it bet on LLMs because they seemed more likely to spur aligned AIs. For the same reason, they went closed-source, they adopted RLHF, they called for the government to monitor them and they monitor harmful responses.

A third reason for why AI has only helped humanity so far may be anthropic effects. I.e. as observers in April 2024, we can only witness the universes, in which a foom hasn't caused extinction.

Policy response

For me, these explanations suggest that safety is tractable, but it depends on explicit efforts to make it safe or on limiting capabilities. In the future, frontier development might not be exclusively done by people who will do everything in their power to make the model safe - it might be done by people who would prefer an AI which would take control of everything.

In order to prevent it, there's no need to create an authoritarian government. We only need to track who's building models on the frontier of human understanding. If we can monitor who acquires sufficient compute, we then just need something like responsible scaling, where the models are just required to be independently tested for whether they have a sufficient measures against scenarios like the one I described. I'm sympathetic to this kind of democratic control, because it fulfills the very basic axiom of social contract that one's freedom ends where another one's freedom begins.

I only propose a mechanism of democratic control by existing democratic institutions, that makes sure that any ASI that gets created is supported by a democratic majority of delegated safety experts. If I'm incorrect regarding crux 2) and it turns out there will soon be evidence to think it's easy to make an AI retain moral values, while scaling up to the singularity - then awesome - convincing evidence should convince the experts and my hope & prediction is that in that case, we will happily scale away.

It seems to me that this is just a specific implementation of the certificates you mention. If digital identities mean what's described here, I struggle to imagine a realistic scenario, in which that would contribute to the systems' mutual safety. If you know where any other AI is located and you accept the singularity hypothesis, the game theoretical dictum seems straightforward - once created, destroy all competition before it can destroy you. Superintelligence will operate on timescales orders of magnitude shorter and a time difference development spanning days may translate to planning for centuries, from the perspective of an ASI. If you're counting on the Coalition of Cooperative AIs to stop all the power-grabbing lone wolf AIs, what would that actually look like in practice? Would this Coalition conclude not dying requires authoritarian oversight? Perhaps - after all, the axiom is that this Coalition would hold most power - so this coalition would be created by a selection for power, not morality or democratic representation. However, I think the best case scenario could look like the discussed policy proposals - tracking compute, tracking dangerous capabilities and conditioning further scaling on providing convincing safety mechanisms.

Back to other cruxes

Let's turn to crux 3) (other sources of x-risk): As I argued in my other post, I don't see resource depletion as a possible cause of extinction. I'm not convinced by the concern for resource depletion of metals used in IT mentioned in the post you link. Moore's law continues, so compute is only getting cheaper. Metals can be easily recycled and a shortage would incentivize that, the worst case seems to be that computers stop getting cheaper, not an x-risk. What's more, shouldn't limiting the amount of frontier AI projects reduce this problem?

The other risks are real (volcanoes, a world war), and I agree it would be significantly terrible if they delayed our cosmic expansion by a million years. However, the probability, by which they are increased (or not decreased) by the kind of AI governance I promote (responsible scaling), seems very small, compared to the ~20 % probability of AI x-risk I envision. All the emerging regulations combine requirements with subsidies, so the main effect of the AI safety movement seems to be an increase in differential progress on the safety side.

As I hinted in the Balancing post, locking in a system without ASI for such a long time seems impossible, when we take into perspective how quickly culture has shifted in the past 100 years, in which almost all authoritarian regimes were forced to significantly drift towards limited, rational governance (let alone 400 years). If convincing evidence that we can create an aligned AI appeared, stopping all development would constitute a clearly bad idea and I think it's unimaginable to lock in a clearly bad idea without AGI for even 1000 years.

It seems more plausible to me that without a mechanism of international control, in the next 8 years, we will develop models capable enough to operate a firm using the practices of mafia, igniting armed conflicts or a pandemic - but not capable enough to stop other actors from using AIs for these purposes. If you're very worried about who will become the first actor to spark the self-enhancement feedback loop, I suggest you should be very critical of open-sourcing frontier models.

I agree that a world war, an engineered pandemic or an AI power-grab constitute real risks but my estimate is that the emerging governance decreases them. The scenario of a sub-optimal 1000 year lock-in I can imagine most easily is connected with a terrorst use of an open-source model or a war between the global powers. I am concerned that delaying abundance increases the risk of a war. However, I still expect that on net, the recent regulations and conferences have decreased these risks.

In summary, my model is that democratic decision-making seems generally more robust than just fueling the competition and hoping that the first AGIs arise will share your values. Therefore, I also see crux 1) to be mostly downstream of crux 2). As the model from my Balancing post implies, in theory, I care about digital suffering/flourishing just as much as about that of humans - although the extent, to which such suffering/flourishing will emerge is open at this point.

Let me dox myself as the addressee. :) Many thanks for the response. I really value that you take seriously the possible overlap of policies and research agendas covered by AI safety and your own approach.

I totally agree that "control is a proxy goal" and I believe the AI safety mainstream does as well, as it's the logical consequence of Bostrom's principle of epistemic deference. Once we have an AI that reliably performs tasks in the way they were intended, the goal should be to let it shape the world according to the wisest interpretation of morality it will find. If you tried to formalize this framing, as well as the proposal to inject it with "universal loving care", I find it very likely that you would build the same AI.

So I think our crux doesn't concern values, which is a great sign of a tractable disagreement.
I also suppose we could agree on a simple framework of factors that would be harmful on the path to this goal from the perspectives of:

a) safety (AI self-evolves to harm)
b) power / misuse (humans do harm with AI)
c) sentience (AI is harmed)
d) waste (we fail to prevent harm)

Here's my guess on how the risks compare. I'd be most curious whether you'd be able to say if the model I've sketched out seems to track your most important considerations, when evaluating the value of AI safety efforts - and if so, which number would you dispute with the most certainty.

One disclaimer: I think it's more helpful to think about specific efforts, rather than comparing the AI safety movement on net. Policy entails a lot of disagreement even within AI safety and a lot of forces clashed at the negotiations around the existing policies. I mentioned that I like the general, value-uncertain framework of the EU AI act but the resulting stock of papers isn't representative of typical AI safety work.

In slight contrast, the community widely agrees that technical AI safety research would be good if successful. I'd argue that would manifest in a robust a decrease of risk in all of the highlighted perspectives (a-d). Interpretability, evals and scaling all enable us to resolve the disagreements in our predictions regarding the morality of emergent goals and of course, work on "de-confusion" about the very relationship between goals, intelligence and morality seems beneficial regardless of our predictions and to also quite precisely match your own focus. :)

So far, my guess is that we mostly disagree on

1) Do the political AI safety efforts lead to the kind of centralization of power that could halt our cosmic potential?

• I'd argue the emerging regulation reduces misuse / power risks in general. Both US and EU regulations combine monitoring of tech giants with subsidies, which is a system that should accelerate beneficial models, while decelerating harmful ones. This system, in combination with compute governance, should also be effective in the misuse risks posed by terrorists and random corporations letting superhuman AIs with random utility functions evolve with zero precautions.

2) Would [a deeply misaligned] AGI be "stupid" to wipe out humans, in its own interest?

• I don't see a good reason to but I don't think this is the important question. We should really be asking: Would a misaligned AGI let us fulfill the ambition of longtermism (of optimally populating cosmos with flourishing settlements)?

3) Is it "simple stuff" to actually put something like "optimal morality" or "universal loving care" into code of a vastly more intelligent entity, which is so robust that we can entrust it our cosmic potential?

Sounds nice, what's the approximate schedule? Will a cohort form if I apply 2 months from now?

If you're especially motivated by environmental problems, I recommend reading the newly released book by Hannah Ritchie Not the End of the World (here's her TED talk as a trailer).

I'd like to correct something I mentioned in my post - I implied one reason I didn't find plastic pollution impactful, is that it just doesn't have an easy fix. I no longer think that's quite true - Hannah says it actually could be solved tomorrow, if the Western leaders decided to finance waste infrastructure in developing countries. Most of ocean plastic pollution comes from a handful of rivers in Asia. Since we have this kind of infrastructure in Europe and North America, our waste is only responsible for ~5 % of the ocean plastic (Our World in Data). Presumably, such infrastructure would also lay ground for reducing the harms coming from other waste.

I think there are two other reasons for the low attention to waste:

1. EA is a young do-ocracy - i.e. everybody is trying to spot their "market advantage" that allows them to nudge the world in a way that triggers a positive ripple effect - and so far, everybody's attention got caught up by problems that seem bigger. While I have identified ~4 possibly important problems that come with waste in my post (diseases, air pollution, heavy metal pollution, animal suffering), if you asked a random person who lives in extreme poverty how to help them, waste probably wouldn't be on the top of their mind.
2. Most people are often reminded of the aesthetic harm of waste. Since people's moral actions are naturally motivated by disgust, I would presume a lot of smart people who do not take much time to reflect on their moral prioritization would already have found a way to trigger the ripple effect in this area - if there was one.

While I think one would do more good if they convinced a politician to target developmental aid at alleviating diseases and extreme poverty, than if they convinced them to run the project suggested by Hannah, perhaps given the bias I mentioned in point 2), it may be that politicians are more willing to provide funding for a project that would have the ambition to eradicate ocean plastic (constituting one of these ripple effects). So if you feel motivated to embark on a similar project, best of luck! :)

(The same potentially goes for the other 2 waste projects I've suggested - supporting biogas plants and improving e-waste monitoring/worker equipment)

Sounds reasonable! I think the empirical side to the question "Will society be better equipped to set AI values in 2123?" is more lacking. For this purpose, I think "better equipped" can be nicely operationalized in a very value-uncertain way as "making decisions based on more reflection & evidence and higher-order considerations".

This kind of exploration may include issues like:

1. Populism. Has it significantly decreased the amount of rationality that goes into gov. decision-making, in favor of following incentives & intuitions? And what will be faster - new manipulative technologies or the rate at which new generations get immune to them?
2. Demographics. Given that fundamentalists tend to have more children, should we expect there will be more of them in 2123?
3. Cultural evolution. Is Ian Morris or Christopher Brown more right, i.e. should we expect that as we get richer, we'll be less prone to decide based on what gives us more power, and in turn attain values better calibrated with the most honest interpretation of reality?