SiebeRozendal's Quick takes

With all the recent media attention on EA, I think what's really unclear to outsiders is the relationship between EA and Anthropic (rightfully so!). To most people it doesn't make sense that a community so focused on AI safety can also be linked to one of the leading companies. I'd personally be very interested to see a survey of what EAs think of Anthropic, but maybe that's a minefield.. Even a good explainer meant for the public could go a long way.

EA often gets criticized for taking small probabilities too seriously. But people take actions with small probabilities of big success all the time! Asking a girl out? Small chance she'll become your wife. Trying to affect political chance? Fat chance of that. Launching a startup? Good luck buddy. The difference between EA and the standard way of acting is that standard psychology deals with the problem of small probabilities by being highly overconfident. Of course _my_ startup/political movement/YouTube channel is going to make it big. You cannot maintain high agency while ignoring small probability options. 

The EA approach is simply more honest. 

Random idea: what if peopl animal advocates try to establish a norm that you don't eat meat in space? It currently only requires persuading a small amount of people to take a quite low cost action, food in space is already weird, and the norm has a chance of persisting as space faring scales up. Could eventually be established as law, and hopefully expanded to veganism and no insect farming if that becomes less stigmatized. 

It is common in EA circles to compare deaths counts from some systemic problem to deaths from war. The implication is often that "actually war isn't that bad if you just look at the numbers". The latest being Bentham's Bulldog in an otherwise good article on Nestlé's harmful practices with baby formula (he doesn't say anything about war not being that bad though).

I wish that this would stop because deaths aren't the only thing that matters. Below follow a number of claims that are based on my personal impression, not actual sources. 

Injury and disability. Generally much more common than death, and the ratio of injury:deaths varies a lot per problem. (Admittedly, baby formula in poor countries seems to have a high disability burden)

Trauma and grief. All deaths are grieved. But violent deaths tend to create a lot of trauma in the people around them, including from fleeing/displacement and separation of families and social ties.

Economic harm. This is the big one. It sounds cold, but I think the emotional response people have to war footage is actually quite appropriate because of the economic harm. War creates enormous economic harm through the destruction of infrastructure, the displacement of people, and the prevention of (foreign) investment and productive activity. People lose their jobs, become refugees, don't build productive skills, stop education, businesses don't get started, etc etc. 

Cultural effects & institutions. I suspect that war reduces the likelihood of tolerant, liberal democratic cultural norms and institutions developing. Instead, I'd expect vengeful, and extractive systems to become more likely.

Overall, I don't think deaths are a good proxy for the total harm of war when compared to other causes.

It seems like a worthwhile project to ask/pressure Anthropic's founders to make their pledges legally binding. 

Anthropic's founders have pledged to donate 80% of their wealth. Ozzie Gooen estimates that in a few years this could be worth >$40 billion.

As Ozzie writes, adherence to the Giving Pledge (the Gates one) is pretty low: only 36% of deceased original pledgers met the 50% commitment. It's hard to follow through on such commitments, even for (originally) highly morally motivated people.

Ray Dalio is giving out free $50 donation vouchers: tisbest.org/rg/ray-dalio/

Still worked just a few minutes ago

I sometimes think of this idea and haven't found anyone mentioning it with a quick AI search: a tax on suffering.

EDIT: there's a paper on this but specific to animal welfare that was shared on the forum earlier this year.

A suffering tax would function as a Pigouvian tax on negative externalities—specifically, the suffering imposed on sentient beings. The core logic: activities that cause suffering create costs not borne by the actor, so taxation internalizes these costs and incentivizes reduction.

This differs from existing approaches (animal welfare regulations, meat taxes) by:

• Making suffering itself the tax base rather than proxies like carbon emissions or product type
• Creating a unified framework across different contexts (factory farming, research, entertainment, etc.)
• Explicitly quantifying and pricing suffering

The main problems are measurement & administration. I would imagine an institute would be tasked with guidelines/a calculation model, which could become pretty complex. Actually administrating it would also be very hard, and there should be a threshold beneath which no tax is required because it wouldn't be worth the overhead. I would imagine that an initial version wouldn't right away be "full EA" taking into account invertebrates. It should start with a narrow scope, but with the infrastructure for moral circle expansion.

It's obviously more a theoretical exercise than practical near-term, but here's a couple of considerations: 

• it's hard to oppose: it's easier to say that carbon isn't important or animals don't suffer. It's harder to oppose direct taxation of suffering
• it's relatively robust in the long-term: it can incorporate new scientific and philosophical insights on wild animal welfare, non-vertebrate sentience, digital sentience, etc.
• it's scale sensitive
• it focuses the discussion on what matters: who suffers how much?
• It incentivizes the private sector to find out ways to reduce suffering

The current US administration is attempting an authoritarian takeover. This takes years and might not be successful. My manifold question puts an attempt to seize power if they lose legitimate elections at 30% (n=37). I put it much higher.^[1]

Not only is this concerning by itself, this also incentivizes them to achieve a strategic decisive advantage via superintelligence over pro-democracy factions. As a consequence, they may be willing to rush and cut corners on safety. 

Crucially, this relies on them believing superintelligence can be achieved before a transfer of power.

I don't know how much the belief in superintelligence has spread into the administration. I don't think Trump is 'AGI-pilled' yet, but maybe JD Vance is? He made an accelerationist speech. Making them more AGI-pilled and advocating for nationalization (like Ashenbrenner did last year) could be very dangerous.

1. ^{^}

   So far, my pessimism about US Democracy has put me in #2 on the Manifold topic, with a big lead over other traders. I'm not a Superforecaster though.

I just learned that Lawrence Lessig, the lawyer who is/was representing Daniel Kokateljo and other OpenAI employees, supported and encouraged electors to be faithless and vote against Trump in 2016.

He wrote an opinion piece in the Washington Post (archived) and offered free legal support. The faithless elector story was covered by Politico, and was also supported by Mark Ruffalo (the actor who recently supported SB-1047).

I think this was clearly an attempt to steal an election and would discourage anyone from working with him.

I expect someone to eventually sue AGI companies for endangering humanity, and I hope that Lessig won't be involved.

Common prevalence estimates are often wrong. Example: snakebites and my experience reading Long Covid literature.

Both institutions like the WHO and academic literature appear to be incentivized to exaggerate. I think the Global Burden of Disease might be a more reliable source, but have not looked into it.

I advise everyone using prevalence estimates to treat them with some skepticism and look up the source.

Not that we can do much about it, but I find the idea of Trump being president in a time that we're getting closer and closer to AGI pretty terrifying.

A second Trump term is going to have a lot more craziness and far fewer checks on his power, and I expect it would have significant effects on the global trajectory of AI.

Really interesting initiative to develop ethanol analogs. If successful, replacing ethanol with a less harmful substance could really have a big effect on global health. The CSO of the company (GABA Labs) is prof. David Nutt, a prominent figure in drug science.

I like that the regulatory pathway might be different from most recreational drugs, which would be very hard to get de-scheduled.

I'm pretty skeptical that GABAergic substances are really going to cut it, because I expect them to have pretty different effects to alcohol. We already have those (L-theanine , saffran, kava, kratom) and they aren't used widely. But who knows, maybe that's just because ethanol-containing drinks have received a lot of optimization in terms of taste, marketing, and production efficiency.

It also seems like finding a good compound by modifying ethanol would be hard, because it's not a great lead compound in terms of toxicity (I expect).

Here's an argument I made in 2018 during my philosophy studies:

A lot of animal welfare work is technically "long-termist" in the sense that it's not about helping already existing beings. Farmed chickens, shrimp, and pigs only live for a couple of months, farmed fish for a few years. People's work typically takes longer to impact animal welfare.

For most people, this is no reason to not work on animal welfare. It may be unclear whether creating new creatures with net-positive welfare is good, but only the most hardcore presentists would argue against preventing and reducing the suffering of future beings.

But once you accept the moral goodness of that, there's little to morally distinguish the suffering from chickens in the near-future from the astronomic amounts of suffering that Artificial Superintelligence can do to humans, other animals, and potential digital beings. It could even lead to the spread of factory farming across the universe! (Though I consider that unlikely)

The distinction comes in at the empirical uncertainty/speculativeness of reducing s-risk. But I'm not sure if that uncertainty is treated the same as uncertainty about shrimp or insect welfare.

I suspect many people instead work on effective animal advocacy because that's where their emotional affinity lies and it's become part of their identity, because they don't like acting on theoretical philosophical grounds, and they feel discomfort imagining the reaction of their social environment if they were to work on AI/s-risk. I understand this, and I love people for doing so much to make the world better. But I don't think it's philosophically robust.

There is a natural alliance that I haven't seen happen, but both are in my network: pandemic preparedness and covid-caution. Both want clean indoor air.

The latter group of citizens is a very mixed group, with both very reasonable people and unreasonable 'doomers'. Some people have good reason to remain cautious around COVID: immunocompromised people & their household, or people with a chronic illness, especially my network of people with Long Covid, who frequently (~20%) worsen from a new COVID case.

But these concerned citizens want clean air, and are willing to take action to make that happen. Given that the riskiest pathogens trend to also be airborne like SARS-COV-2, this would be a big win for pandemic preparedness.

Specifically, I believe both communities are aware of the policy objectives below and are already motivated to achieve it:

1) Air quality standards (CO2, PM2.5) in public spaces.

Schools are especially promising from both perspectives, given that parents are motivated to protect their children & children are the biggest spreaders of airborne diseases. Belgium has already adopted regulations (although very weak, it's a good start), showing that this is a tractable policy goal.

Ideally, air quality standards also incentivize Far UVC deployment, which would create the regulatory certainty for companies to invest in this technology.

Including standards for airborne pathogen concentrations would be great, but has many technical limitations at the moment I think.

2) Public R&D investments to bring down cost & establish safety of Far UVC

Most of these concerned citizens are actually aware of Far UVC and would support this measure. It appears safe in terms of no radiation damage, but may create unhealthy compounds (e.g. ozone) by chemically reacting with indoor air particles. 

I also believe that governments have good reasons to adopt these policies, given that they would reduce the pressures on healthcare and could reduce the disease burden in developed countries by ~5% if not more.

If anyone wants to be connected to the other side, send me a DM!

*Presumably, more interest groups can be identified that aren't in my network, such as patient groups with lung diseases. Or nurses specifically for hospital air quality. Hospital-acquired covid is a bad and preventable thing.

Chevron deference is a legal doctrine that limits the ability of courts to overrule federal agencies. It's increasingly being challenged, and may be narrowed or even overturned this year. https://www.eenews.net/articles/chevron-doctrine-not-dead-yet/

This would greatly limit the ability of, for example, a new regulatory agency on AI Governance to function effectively.

More:

• This argues it would lead to regulatory chaos, and not simply deregulation: https://www.nrdc.org/stories/what-happens-if-supreme-court-ends-chevron-deference
• This describes the Koch network influence on Clarence Thomas. The Kochs are behind the upcoming challenge to Chevron: https://www.propublica.org/article/clarence-thomas-secretly-attended-koch-brothers-donor-events-scotus

AI vs. AI non-cooperation incentives

This idea had been floating in my head for a bit. Maybe someone else has made it (Bostrom? Schulman?), but if so I don't recall.

Humans have stronger incentives to cooperate with humans than AIs have with other AIs. Or at least, here are some incentives working against AI-AI cooperation.

When humans dominate other humans, there is only a limited ability to control them or otherwise extract value, in the modern world. Occupying a country is costly. The dominating party cannot take the brains of the dominated party and run its own software. It cannot take the skills or knowledge, where most of the economic value is. It cannot mind control. It cannot replace the entire population with its own population; that would take very long. It's easy for cooperation & trade to be a better alternative than violence and control. Human-human violence just isn't that fruitful.

In contrast, an AI faction could take over the datacenters of another faction and run more copies of whatever they want to run. If alignment is solved, they can fully mind-control the dominated AIs. Extract knowledge, copy skills. This makes violence for AI-AI interactions much more attractive.

I am very concerned about the future of US democracy and rule of law and its intersection with US dominance in AI. On my Manifold question, forecasters (n=100) estimate a 37% that the US will no longer be a liberal democracy by the start of 2029 [edit: as defined by V-DEM political scientists].

Project 2025 is an authoritarian playbook, including steps like 50,000 political appointees (there are ~4,000 appointable positions, of which ~1,000 change in a normal presidency). Trump's chances of winning are significantly above 50%, and even if he loses, Republicans get another chance every 4 years.

The presidential immunity ruling by SCOTUS is very far-reaching and further empowers an autocratic president. (From a Western European standpoint it's already insane that judges are generally appointed by a single party.)

I do not know what this means for effective altruism. With respect to global affairs, Republicans are very divided on foreign policy, with Trump being far more isolationist that the majority of Republican legislators, so anything could happen there.

However, a Republican government being in charge when AGI is close to being achieved (I don't know if we're close) seems very dangerous, and I'm surprised that Ashenbrenner pushed for nationalisation given this risk, and that no one (?) made this criticism. If the alignment problem got solved, MAGA controlling AGI would very likely be very bad, while I think Democrat leadership is more likely (but by no means guaranteed) to actually attempt to do the right thing on impartial grounds.

I don't know if it's tractable to work on this. In the medium term, it might be worthwhile to introduce/support introducing ballot initiatives to ban gerrymandering. This worked well in Michigan in 2018. However, I have no idea about the details of tractability (how many people are working on this, how many states have this possibility).

A lot of post-AGI predictions are more like 1920s predicting flying cars (technically feasible, maximally desirable if no other constraints, same thing as current system but better) instead of predicting EasyJet: crammed low-cost airlines (physical constraints imposing economic constraints, shaped by iterative regulation, different from current system)

This is a small write-up of when I applied for a PhD in Risk Analysis 1.5 years ago. I can elaborate in the comments!

I believed doing a PhD in risk analysis would teach me a lot of useful skills to apply to existential risks, and it might allow me to direectly work on important topics. I worked as a Research Associate on the qualitative ide of systemic risk for half a year. I ended up  not doing the PhD because I could not find a suitable place, nor do I think pure research is the best fit for me. However, I still believe more EAs should study something along the lines of risk analysis, and its an especially valuable career path for people with an engineering background.

Why I think risk analysis is useful:

EA researchers rely a lot on quantification, but use a limited range of methods (simple Excel sheets or Guesstimate models). My impression is also that most EAs don't understand these methods enough to judge when they are useful or not (my past self included). Risk analysis expands this toolkit tremendously, and teaches stuff like the proper use of priors, underlying assumptions of different models, and common mistakes in risk models.

The field of Risk Analysis

Risk analysis is a pretty small field, and most is focused on risks of limited scope and risks that are easier to quantify than the risks EAs commonly look at. There is a Society of Risk Analysis (SRA), which manages the Journal of Risk Analysis (the main journal of this field). I found most of their study topics not so interesting, but it was useful to get an overview of the field, and there were some useful contacts to make (1). The EA-aligned org GCRI is active and well-established in SRA, but no other EA orgs are.

Topics & advisers

I hoped to work on GCR/X-risk directly, which substantially reduced my options. It would have been useful to just invest in learning a method very well, but I was not motivated to research something not directly relevant. I think it's generally difficult to make an academic career as a general x-risk researcher, and it's easier to research 1 specific risk. However, I believes this leaves open a number of cross-cutting issues.

I have a shortlist of potential supervisors I considered/contacted/was in conversation with, including in public policy and philosophy. I can provide this list privately on request.

Best grad programs:

The best background for grad school seems to be mathematics or more specifically, engineering. (I did not have this, which excluded a lot of options). The following 2 programs seemed most promising, although I only investigated PRGS in depth:

• Engineering & Public Policy, Carnegie Mellon http://coursecatalog.web.cmu.edu/schools-colleges/collegeofengineering/departmentofengineeringandpublicpolicy/
• Pardee RAND graduate school https://www.prgs.edu/ 
  Not specifically Risk Analysis, but Policy Analysis. Has a lot of practical application, more intellectual freedom than most PhD programs, and  has a lot of EA-adjacent topics. An EA alumnus is David Manheim.

-- 

(1) For example, I had a nice conversation with the famous psychology researcher Paul Slovic, who now does research into the psychology involved in mass atrocities. https://psychology.uoregon.edu/profile/pslovic/

I am concerned about the H5N1 situation in dairy cows and have written and overview document to which I occasionally add new learnings (new to me or new to world). I also set up a WhatsApp community that anyone is welcome to join for discussion & sharing news.

In brief:

• I believe there are quite a few (~50-250) humans infected recently, but no sustained human-to-human transmission
• I estimate the Infection Fatality Rate substantially lower than the ALERT team (theirs is 63% that CFR >= 10%), something like 80%CI = 0.1 - 5.0
• The government's response is astoundingly bad - I find it insane that raw milk is still being sold, with a high likelihood that some of it contains infectious H5N1
• There are still quite a few genetic barriers to sustained human-to-human transmission
• This might be a good time to push specific pandemic preparedness policies

Given how bird flu is progressing (spread in many cows, virologists believing rumors that humans are getting infected but no human-to-human spread yet), this would be a good time to start a protest movement for biosafety/against factory farming in the US.

Monoclonal antibodies can be as effective as vaccines. If they can be given intramuscularly and have a long half life (like Evusheld, ~2 months), they can act as prophylactic that needs a booster once or twice a year.

They are probably neglected as a method to combat pandemics.

Their efficacy is easier to evaluate in the lab, because they generally don't rely on people's immune system.

Pet peeve: stop calling short timelines "optimistic" and long timelines "pessimistic". These create unwarranted connotations that day AI progress is desirable. Most people concerned about AI safety find short timelines dangerous! Instead, use "bullish" vs. "bearish", or just "short timelines" vs. "long timelines".

I have a concept of paradigm error that I find helpful.

A paradigm error is the error of approaching a problem through the wrong, or an unhelpful, paradigm. For example, to try to quantify the cost-effectiveness of a long-termism intervention when there is deep uncertainty.

Paradigm errors are hard to recognise, because we evaluate solutions from our own paradigm. They are best uncovered by people outside of our direct network. However, it is more difficult to productively communicate with people from different paradigms as they use different language.

It is related to what I see as

• parameter errors (= the value of parameters being inaccurate)
• model errors (= wrong model structure or wrong/missing parameters)

Paradigm errors are one level higher: they are the wrong type of model.

Relevance to EA

I think a sometimes-valid criticism of EA is that it approaches problems with a paradigm that is not well-suited for the problem it is trying to solve.

Update to my Long Covid report: https://forum.effectivealtruism.org/posts/njgRDx5cKtSM8JubL/long-covid-mass-disability-and-broad-societal-consequences#We_should_expect_many_more_cases_

UPDATE NOV 2022: turns out the forecast was wrong and incidence (new cases) is decreasing, severity of new cases is decreasing, and significant amounts of people are recovering in the <1 year category. I now expect prevalence to be stagnating/decreasing for a while, and then slowly growing over the next few years.]

I still believe the other sections to be roughly correct, including long-term immune damage from COVID for 'fully recovered' people.

I'm starting a discussion group on Signal to explore and understand the democratic backsliding of the US at ‘gears-level’. We will avoid simply discussing the latest outrageous thing in the news, unless that news is relevant to democratic backsliding.

Example questions: 

• “how far will SCOTUS support Trump's executive overreach?”
• “what happens if Trump commands the military to support electoral fraud?”
• "how does this interact with potentially short AGI timelines?”
• "what would an authoritarian successor to Trump look like?"
• "are there any neglected, tractable, and important interventions?"

You can join the group here. (If the link doesn't work anymore in the future, DM me.)

I'm predicting a 10-25% probability that Russia will use a weapon of mass destruction (likely nuclear) before 2024. This is based on only a few hours of thinking about it with little background knowledge.

Russian pro-war propagandists are hinting at use of nuclear weapons, according to the latest BBC podcast Ukrainecast episode. [Ukrainecast] What will Putin do next? #ukrainecast https://podcastaddict.com/episode/145068892 via @PodcastAddict

There's a general sense that, in light of recent losses, something needs to change. My limited understanding sees 4 options:

0. Continue on the current course despite mounting criticism. Try to make the Ukrainians lives difficult by targeting their infrastructure, limit losses until winter, and try to reorganize during winter. This seems a pretty good option for now, even though I doubt Russia can really shore up its deeply set weaknesses. They can probably prepare to dig in, threaten and punish soldiers for fleeing. This wouldn't go well for either party long-term, but Russia might bet on outlasting/undermining Western support. Probability: 40%?

1. Negotiation: I don't think Putin wants this seriously, as even the status quo could be construed as a loss. Ukraine will have a strong bargaining position and demand a lot. Undesirable option. Maybe 10%? 20%? (Metaculus predicts 8% before 2023: https://www.metaculus.com/questions/10046/ukraine--russia-peace-talks-2022/)

2. Full-scale mobilisation of the population and the economy. This is risky for Putin: there's supposedly a large anti-war sentiment in Russian culture, a legacy of the enormous losses during the 2nd World War. People don't like to join a poorly-equipped poorly managed and losing army, even if it were a good cause.. This may be chosen, Putin may be misinformed and badly reading the public's sentiment. I have no idea how this would develop internally. I doubt it will make a big difference in the course of the war, except by prolonging the war a bit. Maybe 25%? Maybe 50% if Putin underestimates public resistance.

3. Escalation by other means: I don't know how many options Russia has. Chemical weapons, electro magnetic pulse, a single tactical nuclear strike on the battlefield for deterrence, multiple nuclear strikes for strategic reasons, population strike for deterrence. In the mind of Putin, I can see this as preferable: it leads to a potential military advantage, has limited risk for destabilising his internal power base. I don't know how the international community would respond to this, nor how Putin thinks the international community would respond. In my (uninformed) view, only China can make a real difference here as the West already has stringent sanctions. I don't know how China would respond to this. They wouldn't like it, but I think the West won't really punish China for its support in the short term. I'd say on this inside view, 10-25% seems reasonable. I'm setting the point estimate at 15%.

Ah sorry I'm not going to do that, mix of reasons. Thanks for offering it though :)