This is a special post for quick takes by yefreitor. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Sorted by Click to highlight new quick takes since:

Claim: Credible plans for a "pivotal act" may drive AI race dynamics

(Epistemic status: I had mathematica do all the grunt work and did not check the results carefully)

Consider a simple normal-form game with two equally capable agents A and B, each of which is deciding whether to aggressively pursue AI development, and three free parameters:

  • the probability  that accelerating AI development results in an existential catastrophe (with utility -1 for both agents, versus a utility 0 status quo). 
  • the utility  of developing the first friendly AI
  • the utility  of the other agent developing friendly AI

We'll first assume the coin only gets flipped once: developing a friendly AI lets you immediately control all other AI development.

Since our choice of parameterization was in retrospect one that requires a lot of typing, we'll define  and then rescale to get something more readable

  • (Accelerate, Accelerate) is always a Nash equilibrium, no matter how trivial the differences  captures are.
  • (Don't, Don't) is a Nash equilibrium when , as you would expect
  • (Don't, Don't) is never a trembling-hand equilibrium, since (Don't) does not weakly dominate (Accelerate) for either player.
  • When  (Accelerate) weakly dominates (Don't) and (Accelerate, Accelerate) is a trembling-hand equilibrium.

Now consider the case where (Accelerate, Accelerate) instead flips two coins.

This is potentially a much safer situation:

  • (Accelerate, Accelerate) is only a Nash equilibrium when 
  • (Don't, Don't) is still a Nash equilibrium when 
  • (Don't, Don't) is a trembling-hand equilibrium if it's a Nash equilibrium and (Accelerate, Accelerate) is not.
Curated and popular this week
Relevant opportunities