Reframing LessWrong-style decision theory as "commitment theory"

Elias Schmied

Reframing LessWrong-style decision theory as "commitment theory"

Elias Schmied

13 min readJul 8

Comments 1

Sorted by

New & upvoted

Elias Schmied

(If you want to immediately skip to why I believe past attempts to incorporate LessWrong-style intuitions into standard decision theory are insufficient, go to footnote 16)

Comments

More from the author

A brief list of ways AI safety efforts could be net negative

Elias Schmied·2w ago·2m read

A simple argument for trying less hard

Elias Schmied·3w ago·5m read

Curated and popular this week

Cultivating hope: calibrating the expectations for cultivated meat to end factory farming

PabloAMC 🔸·1w ago·Curated 6d ago·22m read

137

Maybe do the thing you wish CEA would do

alejoacelas 🔸·5d ago·2m read

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts: * Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do * NEST is an opinionated ideas-fi...

The first video from Giving What We Can's new channel is out now!

JustinPortela·22h ago·1m read

Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job. My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI. ...

Recent opportunities to take action

New round of digital minds funding opportunities at Longview

zdgroff, Longview Philanthropy·2d ago·2m read

173

Possible mistake EAs are making and shout out to Pause AI UK

Michelle_Hutchinson·1w ago·4m read

Inspiring colleagues in Luxembourg on Effective Giving + identifying infrastructural gaps

Lorenzo Fong Ponce 🔸·19h ago·12m read

^{^}

To be clear, it’s not my personally preferred framing, and in my opinion it ultimately fails for conceptual/metaphysical reasons, and LessWrong thought is much closer to correct - but I think it goes quite far, as I’ll argue in this post.

^{^}

Those about updatelessness, specifically, not those about counterfactuals (which are obviously part of academic decision theory).

^{^}

I’ve rewritten it to make it clearer, since the original is a bit confusing and underspecified (see this Stuart Armstrong comment), and including that Omega specifically simulates you to predict you is unnecessary. My version preserves the structure and should meet the core desideratum of its proponents (that FDT chooses to burn to death for seemingly no reason) - but maybe call it Bomb* (with a star at the end) for clarity, if you reference it.

It’s kind of like the Transparent Newcomb’s Problem, but optimized for counterintuitiveness - for example, that you are in an “impossible reality” and not in a “possible reality that you just need to make real”.

^{^}

at least with extremely high probability. (also, for convenience this paragraph is taken from Will MacAskill and modified for my context - I couldn’t find a better way of explaining it than his, but I also needed to make too many small changes to make it a direct quote.)

^{^}

This is a more UDT-style framing.

^{^}

e.g. there are a variety of arguments in the comments on Will’s post.

^{^}

This is the academic discussion on “binding” and dynamic choice.

^{^}

One might also gesture to some intuition that this is central to what “being an agent” means - you only have the ability to make plans if you can lock your future self into a set of actions, if you can expect your future self to “cooperate” with your current self’s commitments. Decision theory and commitment theory seem like two sides of the coin of ideal decisionmaking - how to maximize for your preferences (an adversarial position towards the world) versus how to intentionally ignore and blind yourself to your preferences (to cooperate with the world). (This framing is inspired by Richard Ngo).

^{^}

This is the famous “Non-FDT agents self-modify to be FDT agents” argument, put into practice.

^{^}

This is one of the core ways that UDT was originally characterized. The clearest statement of that I’ve been able to find is Demski and Garrabrant (2019), who straight-up define it as: “UDT [recommends] that the agent do whatever would have seemed wisest before—whatever your earlier self would have committed to do”.

(and FDT is an umbrella term for “UDT-ish approaches to decision theory”. Although there are some terminological subtleties - there’s a longer comment by Rob Bensinger in that post that’s quite clarifying. But it also comes up several times in the FDT paper as a core property, just ctrl-F “committed”.)

Of course, you still need the correct counterfactuals - e.g. when deciding what to precommit to in a situation with an agent that is similar but not identical to you. But counterfactuals are the “decision theory dimension” (EDT vs CDT), not the “commitment theory dimension” (updateless vs updateful), so they are a separate issue. (But I think e.g. commitment theory EDT just works).

^{^}

It’s buried pretty deep in the post, so I’ll quote it - Christiano says: “I’m not sure if I’m thinking about worlds that don’t exist, or if it’s us who don’t exist and there is some real world somewhere thinking about us.”

There’s one caveat here (this is complicated and not well-argued, feel free to skip): When agents throughout the universe realize that they should make the great commitment, they might already be in the midst of executing strategies that they might together have precommitted to not doing in advance (e.g. cooperating with other value systems in the universe too little - ECL), if they were making the great commitment from the perspective “outside of time” of the simple agent structure that is inside of them (cf Nate Soares’ discussion of being “wrapped around The Algorithm” here). This is the debate about “how far back” to be updateless in UDT - there’s an analogous debate in commitment theory about how simple of a version of yourself you imagine making the great commitment (and therefore how much what you commit to is determining outcomes all throughout the universe). So you can’t avoid weird metaphysics after all, (“outside of time” is obviously metaphysical reasoning), no matter how hard you try. That’s part of why I think the commitment theory framing is ultimately insufficient. But it should at least make clear that people shouldn’t be disagreeing about stuff like Newcomb’s problem and Bomb* (anything other than ECL, really) - you don’t even need to get into metaphysics to see the right answer there!

^{^}

Not all of it - Some of it is better classified as EDT vs CDT stuff (i.e. the “decision theory dimension”), and some as updateless vs updateful stuff (i.e. the “commitment theory dimension”).

^{^}

If this makes you disconnect from the scenario because your past self wouldn’t be this utilitarian, just imagine some other less bad consequence. Something like this works no matter what your preferences are - the point is just that FDT can make you choose a locally bad outcome for seemingly no reason.

^{^}

Of course, if you face such an unlikely situation in real life, you should completely rethink your assumptions and seriously wonder if you’ve gone crazy. But as it’s a hypothetical decision problem, we’ve fixed the agent’s ex hypothesi beliefs about the situation he’s in - so I’ve written this snippet like you just happen to not think of that possible response.

^{^}

Once again, this is the academic discussion on binding.

^{^}

I want to briefly examine other attempts to do something similar, and explain why I think they fail. First, Will MacAskill’s “Global CDT”, as it’s closest in the social graph:

Even as it succeeds at replicating a lot of FDT’s recommendations, it leaves less of its conceptual attractiveness intact. Global CDT is framed as simply evaluating with CDT many different “evaluative focal points”: different personalities, dispositions, rules, and so on. For example, it gets as specific as “[the] sort of person… who cooperates in prisoner’s dilemmas”. But we’d prefer to retain the intuitive sense that the “FDT insight” is one singular thing (“act as you would have precommitted to”), and still avoid the associated metaphysical and normative claims - which we can, with commitment theory.

More importantly, I think it fails on counterfactual mugging with a deterministic coin, since the reality where the coin lands heads is not real and could never have been real, and so doesn’t get taken into account in any updateful decision theory after you learn it lands tails (even if we evaluate bigger things like dispositions or rules). (You can understand dispositions and rules as leaky human approximations of theoretical “true” updatelessness - which just evaluates anew the correct precommitment to have taken for every situation)

For attempts from academia (this is going to be uninteresting if you’re not in the weeds on this stuff, so feel free to skip, and I haven’t fully read any of these papers so take it with a grain of salt): I expect the worries above to also apply to Fisher’s and Gauthier’s disposition-based decision theories, Parfit’s “rational irrationality”, and Spohn (”Reversing 30 Years of Discussion”, 2012). McClennen’s “resolute choice” (“it is rational to follow through on plans even when they become locally harmful”) is insufficient because it fails on situations that you didn’t anticipate, like Bomb* - the thing you need to be “resolute” about is much deeper than plans. The same goes for Bratman (Intentions, Plans, and Practical Reason, 1987). Poellinger (“Unboxing the Concepts in Newcomb's Paradox”, 2013) and Hedden (”Counterfactual Decision Theory”, 2023) are about counterfactuals (the “decision theory dimension”), and not updatelessness (the “commitment theory dimension“), so they’re orthogonal to this discussion.

Meacham’s cohesive decision theory (“do as you would have bound yourself to do”, quite similar) is the closest academic relative to FDT (footnote 34 is genuinely fascinating as an early discussion of the idea of updatelessness) and has seemingly the exact same recommendations (modulo a theory of counterfactuals) - but as it also has acts as its evaluative focal point, it incurs the same intuitive drawback of calling it “rational” to choose to burn to death in Bomb*. Also, since it lacks the intuition pumps I’ve built up around it, and tries to fit it into the unnatural frame of decision theory, it ends up looking a bit unmotivated (which is maybe why it didn’t get much traction). It also doesn’t explicitly open up the question of what exactly it means for an action to “have been good to have been bound to” (which ends up being a rich field of study).

^{^}

Especially because, in my opinion, you inevitably run into metaphysical questions (changing the past, alternate realities, platonism about computations) if you really try to justify why e.g. taking the Left box and burning to death in Bomb* is the “correct” decision, in the decision-theoretic sense of maximizing your utility. (Not sure how much of a hot take that is, I will justify it more in a future post). And it’s understandable that people are reluctant to wade into that.

Finally, a brief and dense sidenote (don’t worry if this doesn’t make sense to you): Commitment theory is actually slightly better than naive TDT-style FDT (which is the way many people conceive of FDT), because it performs better on counterfactual mugging with a deterministic coin. This is the sense in which FDT lacks a theory of anthropics, in the “UDT = FDT + a theory of anthropics” way of carving up the space - another way of seeing that metaphysics comes into play.

(Of course, commitment theory doesn’t fully avoid metaphysics at the end of the day either, as I argue in footnote 11)

Reframing LessWrong-style decision theory as "commitment theory"

Reframing LessWrong-style decision theory as "commitment theory"

Introduction

The great commitment

Commitment theory

FDT

A detailed example

Conclusion