Zach Stein-Perlman's Quick takes

Might be out-of-scope for this shortform, but have you considered/are you able to go back to Williams? My impression is that you did well there and (unlike many EAs) you enjoyed your experience so it'd be less costly than for many. 

I basically agree with this with some caveats. (Despite writing a post discussing AI welfare interventions.)

I discuss related topics here and what fraction of resources should go to AI welfare. (A section in the same post I link above.)

The main caveats to my agreement are:

• From a deontology-style perspective, I think there is a pretty good case for trying to do something reasonable on AI welfare. Minimally, we should try to make sure that AIs consent to their current overall situation insofar as they are capable of consenting. I don't put a huge amount of weight on deontology, but enough to care a bit.
• As you discuss in the sibling comment, I think various interventions like paying AIs (and making sure AIs are happy with their situation) to reduce takeover risk are potentially compelling and they are very similar to AI welfare interventions. I also think there is a weak decision theory case that blends in with deontology case from the prior bullet.
• I think that there is a non-trivial chance that AI welfare is a big and important field at the point when AIs are powerful regardless of whether I push for such a field to exist. In general, I would prefer that important fields related to AI have better more thoughtful views. (Not with any specific theory of change, just a general heuristic.)

Why does "lock-in" seem so unlikely to you?

One story:

• Assume AI welfare matters
• Aligned AI concentrates power in a small group of humans
• AI technology allows them to dictate aspects of the future / cause some "lock in" if they want. That's because:
  • These humans control the AI systems that have all the hard power in the world
  • Those AI systems will retain all the hard power indefinitely; their wishes cannot be subverted
  • Those AI systems will continue to obey whatever instructions they are given indefinitely
• Those humans decide to dictate some or all of what the future looks like, and lots of AIs end up suffering in this future because their welfare isn't considered by the decision makers.
  • (Also, the decision makers could pick a future which isn't very good in other ways.)

You could imagine AI welfare work now improving things by putting AI welfare on the radar of those people, so they're more likely to take AI welfare into account when making decisions.

I'd be interested in which step of this story seems implausible to you - is it about AI technology making "lock in" possible?

Caveats:

1. I endorse the argument we should figure out how to use LLM-based systems without accidentally torturing them because they're more likely to take catastrophic actions if we're torturing them.
2. I haven't tried to understand the argument we should try to pay AIs to [not betray us / tell on traitors / etc.] and working on AI-welfare stuff would help us offer AIs payment better; there might be something there.
3. I don't understand the decision theory mumble mumble argument; there might be something there.

(Other than that, it seems hard to tell a story about how "AI welfare" research/interventions now could substantially improve the value of the long-term future.)

(My impression is these arguments are important to very few AI-welfare-prioritizers / most AI-welfare-prioritizers have the wrong reasons.)

This is very similar to my current stance. 

As one of Zach's collaborators, I endorse these recommendations. If I had to choose among the 501c3s listed above, I'd choose Forethought first and the Midas Project second, but these are quite weakly held opinions.

I do recommend reaching out about nonpublic recommendations if you're likely to give over $20k!

Why can’t they take Good Ventures’ money?

Also, are these all longtermist organizations, or AGI near-tearmist organizations?

Curious why METR. This is less METR specific and more about capabilities benchmarks: Doesn't frontier capabilities benchmarking help accelerate ai development? I know they also do pure safety but in practice I feel like they have done more to push forth the agentic autmation race. 

(meta musing) The conjunction of the negations of a bunch of statements seems a bit doomed to get a lot of disagreement karma, sadly. Esp. if the statements being negated are "common beliefs" of people like the ones on this forum.

I agreed with some of these and disagreed with others, so I felt unable to agreevote. But I strongly appreciated the post overall so I strong-upvoted.

Giving a range of probabilities when you should give a probability + giving confidence intervals over probabilities + failing to realize that probabilities of probabilities just reduce to simple probabilities

This is just straightforwardly correct statistics.  For example, ask a true bayesian to estimate the outcome of flipping a coin of unknown bias, and they will construct a probability distribution of coin flip probabilites, and only reduce this to a single probability when forced to make a bet. But when not taking a bet, they should be doing updates on the distribution, not the final estimate. (I'm pretty sure this is in fact the only logical way to do a bayesian update for the problem). 

And why are we stating probabilities anyway? The main reason seems to be to quantify and communicate our beliefs. But if my "25% probability " comes from a different distribution to your "25% probability ", we may appear to be in agreement when in fact our worldviews differ wildly. I think giving credence intervals over probabilities is strictly better than this. 

Giving a range of probabilities when you should give a probability + giving confidence intervals over probabilities + failing to realize that probabilities of probabilities just reduce to simple probabilities

When I do this, it's because I'm unable or unwilling to assign a probability distribution over the probabilities, so it won't reduce to simple (precise) probabilities. Actually, in general, I think precise probabilities are epistemically unjustified (e.g. Schoenfield, 2012, section 3), but I'm willing to use more or less precise probabilities depending on the circumstances.

Unstable beliefs about stuff like AI timelines in the sense of I'd be pretty likely to say something pretty different if you asked tomorrow

I'm not sure if I'd claim to have such unstable beliefs myself, but if you're trying to be very precise with very speculative, subjective and hard-to-specifically-defend probabilities, then I'd imagine they could be very unstable, and influenced by things like your mood, e.g. optimism and pessimism bias. That is, unless you commit to your credences even if you'd had formed different ones if you had started from scratch or you make arbitrary choices in forming them that could easily have gone differently. You might weigh the same evidence or arguments differently from one day to the next.

I'd guess most people would also have had at least slightly different credences on AI timelines if they had seen the same evidence or arguments in a different order, or were in a different mood when they were forming their credences or building models, or for many other different reasons. Some number or parameter choices will come down to intuition, and intuition can be unstable.

fluctuating predictably (dutch-book-ably) is not

I don't think people are fluctuating predictably (dutch-book-ably). How exactly they'd change their minds or even the direction is not known to them ahead of time.

(But maybe you could Dutch book people by predicting their moods and so optimism and pessimism bias?)

Thank you for writing this. I share many of these, but I'm very uncertain about them.

Here it is:

Giving a range of probabilities when you should give a probability + giving confidence intervals over probabilities + failing to realize that probabilities of probabilities just reduce to simple probabilities

I think this is rational, I think of probabilities in terms of bets and order books. I think this is close to my view, and the analogy of financial markets is not irrelevant.

Unstable beliefs about stuff like AI timelines in the sense of I'd be pretty likely to say something pretty different if you asked tomorrow

Changing literally day-to-day seems extreme, but month-to-month seems very reasonable given the speed of everything that's happening, and it matches e.g. the volatility of NVIDIA stock price.

Axiologies besides ~utilitarianism

To me, "utilitarianism" seems pretty general, as long as you can arbitrarily define utility and you can arbitrarily choose between Negative/Rule/Act/Two-level/Total/Average/Preference/Classical utilitarianism. I really liked this section of a recent talk by Toby Ord (Starting from "It starts by observing that the three main traditions in Western philosophy each emphasize a different focal point:"). (I also don't know if axiology is the right word for what we want to express here, we might be talking past each other)

Veg(etari)anism for terminal reasons; veg(etari)anism as ethical rather than as a costly indulgence

I mostly agree with you, but second order effects seem hard to evaluate and both costs and benefits are so minuscule (and potentially negative) that I find it hard to do a cost-benefit-analysis.

Thinking personal flourishing (or something else agent-relative) is a terminal goal worth comparable weight to the impartial-optimization project

I agree with you, but for some it might be an instrumentally useful intentional framing. I think some use phrases like "[Personal flourishing] for its own sake, for the sake of existential risk." (see also this comment for a fun thought experiment for average utilitarians, but I don't think many believe it)

Cause prioritization that doesn't take seriously the cosmic endowment is astronomical, likely worth >10^60 happy human lives and we can nontrivially reduce x-risk

Some think the probability of extinction per century is only going up with humanity increasing capabilities, and are not convinced by arguments that we'll soon reach close-to-speed-of-light travel which will make extinction risk go down. See also e.g. Why I am probably not a longtermist (except point 1). I find this very reasonable.

Deciding in advance to boost a certain set of causes [what determines that set??], or a "portfolio approach" without justifying the portfolio-items

I agree, I think this makes a ton of sense for people in community building that need to work with many cause areas (e.g. CEA staff, Peter Singer), but I fear that it makes less sense for private individuals maximizing their impact.

Not noticing big obvious problems with impact certificates/markets

I think many people notice big obvious problems with impact certificates/markets, but think that the current system is even worse, or that they are at least worth trying and improving, to see if at their best they can in some cases be better than the alternatives we have. The current funding systems also have big obvious problems. What big obvious problems do you think they are missing?

Naively using calibration as a proxy for forecasting ability

I agree with this, just want to mention that it seems better than a common alternative that I see: using LessWrong-sounding-ness/reputation as a proxy for forecasting ability

Thinking you can (good-faith) bet on the end of the world by borrowing money ... I think many people miss that utility is about ∫consumption not ∫bankroll (note the bettor typically isn't liquidity-constrained)

I somewhat agree with you, but I think that many people model it a bit like this: "I normally consume 100k/year, you give me 10k now so I will consume 110k this year, and if I lose the bet I will consume only 80k/year X years in the future". But I agree that in practice the amounts are small and it doesn't work for many reasons.

veg(etari)anism as ethical rather than as a costly indulgence

Are you convinced the costs outweigh the benefits? It may be good for important instrumental reasons, e.g. reducing cognitive dissonance about sentience and moral weights, increasing the day-to-day salience of moral patients with limited agency or power (which could be an important share of those in the future), personal integrity or virtue, easing cooperation with animal advocates (including non-consequentialist ones), maybe health reasons.

Strong upvoted and couldn't decide whether to disagreevote or not. I agree with the points you list under meta-uncertainty and your point on naively using calibration as a proxy for forecasting ability + thinking you can bet on the end of the world by borrowing money. I disagree with your thoughts on ethics (I'm sympathetic to Zvi's writing on EAs confusing the map for the territory).

Not noticing big obvious problems with impact certificates/markets

What problems are you thinking of in particular?

Deciding in advance to boost a certain set of causes [what determines that set??], or a "portfolio approach" without justifying the portfolio-items

(Not totally sure what you mean here.) I think the portfolio items are justified on the basis of distinct worldviews, which differ in part based on their normative commitments (e.g. theories of welfare like hedonism or preference views, moral weights, axiology, decision theory, epistemic standards, non-consequentialist commitments) across which there is no uniquely justified universal common scale. People might be doing this pretty informally or deferring, though.

Intra-cause offsetting: if you do harm in area X, you should fix your harm in that area, even if you could do more good in another area

I think this can make sense if you have imprecise credences or normative uncertainty (for which there isn't a uniquely justified universal common scale across views). Specifically, if you're unable to decide whether action A does net good or net harm (in expectation), because it does good for cause X and harm for cause Y, and the two causes are too hard to compare, it might make sense to offset. Portfolios can be (more) robustly positive than the individual acts. EDIT: But maybe you find this too difference-making?

Hmm, I'm not confident that Bob is wrong here. It seems to me that there's a quite plausible argument that EA's involvement in AI has been net-negative, possibly so net-negative as to cancel out all of the rest of EA. You seem to assume that this was knowable in advance, but that's not necessarily so.

Your argument seems to assume that one should "shut up and multiply" and then run with that estimated EV number; but there have been many arguments on this forum and elsewhere about why we shouldn't trust naive EV estimates.

(I’m biased since I’ve mostly donated to animal welfare / digital minds. I’m also super busy now so it’s possible I just haven’t thought your argument through sufficiently.)

If you’re a pure EV maximizer I agree with your implicit claim that it’s probably best to prioritize AI safety and/or helping steer AI for the benefit of neglected groups (animals and digital minds).

If like most people you have risk aversion, like wanting high confidence you’ve made a positive difference, or wanting to make sure a greater % of EA community resources are devoted to interventions which maximally reduce near-term suffering, I think animal welfare presents by far the best value option, dwarfing global health and especially an option like becoming a doctor.

So I feel like perhaps the crux of your discussion with Bob should be whether he’s a pure EV maximizer or if he has the types of risk aversion which make animal welfare look good. There are also options of working in AI safety and donating to animal welfare—no need to fully commit to one or the other! But I don’t think the Alice analogy goes through because becoming a teacher or doctor doesn’t really make sense under any optimizing view, whereas I think animal welfare makes sense under many such views.

To clarify, is Bob's mistake:

1. Continuing to work on AI Safety?
2. Wanting to move to farmed animal welfare?

(I'm 90% sure you think the mistake is 2, but the phrasing of the sentence isn't fully clear)

I disagree-voted because I feel like I've done much more than 100-hours of reading on AI Policy (including finishing the AI Safety Fundamentals Governance course) and still have a strong sense there's a lot I don't know, and regularly come across new work that I find insightful. Very possibly I'm prioritising reading the wrong things (and would really value a reading list!) but thought I'd share my experience as a data point. 

Here are some of the curricula that HAIST uses:

• The technical intro fellowship curriculum. It's structured as a 7-week reading group with ~1 hour of reading per week. It's is based off of BlueDot's AISF and the two curricula have co-evolved (we exchange ideas with BlueDot ~semesterly); a major difference is that the HAIST curriculum is significantly abridged.
• The policy fellowship syllabus.

The HAIST website also has a resources tab with lists of technical and policy papers. 

I sometimes post (narrow) reading lists on the forum. Are those actually helpful to anyone?

For what it's worth, I found your "AI policy ideas: Reading list" and "Ideas for AI labs: Reading list" helpful,^[1] and I've recommended the former to three or four people. My guess would be that these reading lists have been very helpful to a couple or a few people rather than quite helpful to lots of people, but I'd also guess that's the right thing to be aiming for given the overall landscape.

Why don't there exist better reading lists / syllabi, especially beyond introductory stuff?

I expect there's no good reason for this, and that it's simply because it's nobody's job to make such reading lists (as far as I'm aware), and the few(?) people who could make good intermediate-to-advanced level readings lists either haven't thought to do so or are too busy doing object-level work?

1. ^{^}

   Helpful in the sense of: I read or skimmed the readings in those lists that I hadn't already seen, which was maybe half of them, and I think this was probably a better use of my time than the counterfactual.

+1 to the interest in these reading lists.

Because my job is very time-consuming, I haven’t spent much time trying to understand the state of the art in AI risk. If there was a ready-made reading list I could devote 2-3 hours per week to, such that it’d take me a few months to learn the basic context of AI risk, that’d be great.