Zach Stein-Perlman

@ ailabwatch.org
5473 karmaJoined Working (0-5 years)Berkeley, CA, USA
ailabwatch.org

Bio

Participation
1

AI strategy & governance. ailabwatch.org.

Comments
475

Topic contributions
1

I mostly agree with the core claim. Here's how I'd put related points:

  • Impact is related to productivity, not doing-your-best.
  • Praiseworthiness is related to doing-your-best, not productivity.
  • But doing-your-best involves maximizing productivity.
  • Increasing hours-worked doesn't necessarily increase long-run productivity. (But it's somewhat suspiciously convenient to claim that it doesn't, and for many people it would.)

I haven't read all of the relevant stuff in a long time but my impression is Bio/Chem High is about uplifiting novices and Critical is about uplifting experts. See PF below. Also note OpenAI said Deep Research was safe; it's ChatGPT Agent and GPT-5 which it said required safeguards.

I haven't really thought about it and I'm not going to. If I wanted to be more precise, I'd assume that a $20 subscription is equivalent (to a company) to finding a $20 bill on the ground, assume that an ε% increase in spending on safety cancels out an ε% increase in spending on capabilities (or think about it and pick a different ratio), and look at money currently spent on safety vs capabilities. I don't think P(doom) or company-evilness is a big crux.

fwiw I think you shouldn't worry about paying $20/month to an evil company to improve your productivity, and if you want to offset it I think a $10/year donation to LTFF would more than suffice.

The thresholds are pretty meaningless without at least a high-level standard, no?

One problem is that donors would rather support their favorite research than a mixture that includes non-favorite research.

I'm optimistic about the very best value-increasing research/interventions. But in terms of what would actually be done at the margin, most work that people would do for "value-increasing" reasons would be confused/doomed, I expect (and this is less true for AI safety).

I think for many people, positive comments would be much less meaningful if they were rewarded/quantified, because you would doubt that they're genuine. (Especially if you excessively feel like an imposter and easily seize onto reasons to dismiss praise.)

I disagree with your recommendations despite agreeing that positive comments are undersupplied.

Given 3, a key question is what can we do to increase P(optimonium | ¬ AI doom)?

For example:

  • Averting AI-enabled human-power-grabs might increase P(optimonium | ¬ AI doom)
  • Averting premature lock-in and ensuring the von Neumann probes are launched deliberately would increase P(optimonium | ¬ AI doom), but what can we do about that?
  • Some people seem to think that having norms of being nice to LLMs is valuable for increasing P(optimonium | ¬ AI doom), but I'm skeptical and I haven't seen this written up.

(More precisely we should talk about expected fraction of resources that are optimonium rather than probability of optimonium but probability might be a fine approximation.)

One key question for the debate is: what can we do / what are the best ways to "increas[e] the value of futures where we survive"?

My guess is it's better to spend most effort on identifying possible best ways to "increas[e] the value of futures where we survive" and arguing about how valuable they are, rather than arguing about "reducing the chance of our extinction [vs] increasing the value of futures where we survive" in the abstract.

Load more