I agree with a lot of this, and generally think people should act "as if":
There is at least one of them in some 'basement' universe.
They "control all the the copies" of themself
rather than reason/act as if "I might be in a simulation". In practice I think the two don't come apart massively, but I think there are risks in just adoping this latter approach (e.g. psychological).
On the two claims I think people should follow as if they were "true":
If we reject (1) then we can't rule out the case of our observations being non-veridical[1].
This would make it hard for us to influence the 'base' reality (which for utilitarians might be where most of the potential value is).
In a large enough multiverse, (1) seems likely. (Though if (1) is false and we're in a non-veridical simulation, it's unclear how much evidence we can have about whether we're in a large multiverse[2]).
Even if (1) did not hold, we might hope that some of observations are still useful in helping us influence the basement reality e.g. mathematical/logical/ philosophical arguments [3]. Of course simulators could be be arbitrarily Cartesian-demon like, but these seem safe to me to bracket-out.
I think (2) is not too tricky to accept and widelt accepted in these parts (though I think many academic decision theories would reject). Of the range of views the axes described in this post, one can be an evidentialist/EDT-like or use the algorithmic ontology.
Regarding veridicity, I think there's some risk of circularity here (and am not certain I've thread the needle):
With non-veridical simulations, I want to say something like ruling out simulations that some smart intelligence would say are of "unreasonable counterfactuals" (or are just complete fiction - i.e. they don't aim to capture some of the same dynamics as they believe they have in their own universe/multiverse)
So for example, if the totality of what exists ("the universe") was determined by some arbitrary (to some highly intelligent observer who understood physics as much as possible ) initial conditions, them I would class the simulation that runs "the universe but with different intial conditions" as reasonable.
With respect to an intelligent observer that believed in (and had good reason to believe) in a sufficiently large multiverse, then it's plausible that many counterfactuals are "reasonable".
Conditional on being in a non-veridical simulation, beyond our prior we have an anthropic update (if we are non-updateless) but little else to inform us about the world. (Perhaps our prior has expectations over which simulations that basement beings run)
Without (1), I don't think we can necessarily rule out simulators being able to run "counterlogical simulations" (this may only seem impossible to us because of the world they are happening to simulate).
Regarding Jackson's comment, I agree that 'dumping' money last-minute is a bit silly. Spending at a higher rate (and saving less) doesn't seem so crazy - which is what it seems you were considering.
Why do you believe that capital won't be useful?
My guess the modal outcome from AGI (and eventual ASI) is human disempowerment/extinction. Less confidently, I also suspect that most worlds where things go 'well' look weird and not not much like business-as-normal. For example, if we eventually have a sovereign ASI implement some form of coherent extrapolated volition I'm pretty unsure how (we would want) this to interact with individuals' capital. [Point 2. of this recent shortform feel adjacent - discussing CEV based on population rather than wealth].
Thanks for the list! Napkin AI is particularly cool.
Your LibGen link isn't working for me. Its Wikipedia page usually has live (& safe) links in the infobox. Same for Sci-Hub.
One recent service I've found is Readwise. It's similar to Pocket/Instapaper in adding web-pages to read later, but you can also upload your own EPUB and PDFs.
I thought about this a few years ago and have a post here.
I agree with Caleb's comment on the necessity to consider what a post-superintelligence world would look like, and whether capital could be usefully deployed. This post might be of interest.
My own guess is that it's most likely that capital won't be useful and that more aggressive donating makes sense.
The consequence of this for the "spend now vs spend later" debate is crudely modeled in The optimal timing of spending on AGI safety work, if one expects automated science to directly & predictably precede AGI. (Our model does not model labor, and instead considers [the AI risk community's] stocks of money, research and influence)
We suppose that after a 'fire alarm' funders can spend down their remaining capital, and that the returns to spending on safety research during this period can be higher than spending pre-fire alarm (although our implementation, as Phil Trammell points out, is subtly problematic, and I've not computed the results with a corrected approach).
Thanks again Phil for taking the read this through and for the in-depth feedback.
I hope to take some time to create a follow-up post, working in your suggestions and corrections as external updates (e.g. to the parameters of lower total AI risk funding, shorter Metaculus timelines).
I don't know if the “only one big actor” simplification holds closely enough in the AI safety case for the "optimization" approach to be a better guide, but it may well be.
This is a fair point.
The initial motivator for the project was for AI s-risk funding, of which there's pretty much one large funder (and not much work is done on AI s-risk reduction outside of people and organizations and people outside the effective altruism community) though this result is entirely on AI existential risk, which is less well modeled as a single actor.
My intuition is that the "one big actor" does work sufficiently well for the AI risk community given the shared goal (avoid an AI existential catastrophe) and my guess that a lot of the AI risk done by the community doesn't change the behaviour of AI labs much (i.e. it could be that they choose to put more effort into capabilities over safety because of work done by the AI risk community, but I'm pretty sure this isn't happening).
For example, the value of spending after vs. before the "fire alarm" seems to depend erroneously on the choice of units of money. (This is the second bit of red-highlighted text in the linked Google doc.) So I'd encourage someone interested in quantifying the optimal spending schedule on AI safety to start with this model, but then comb over the details very carefully.
To comment on this particular error (though not to say that other errors Phil points to are not also unproblematic - I've yet to properly go through them), for what it's worth, the main results of the post suppose zero post fire alarm spending[1] and (fortunately) since in our results we use units of millions of dollars and take the initial capital to be on the order of 1000 $m, I don't think we face this problem of smaller η having the reverse than desired effect for
In a future version I expect I'll just take the post-fire alarm returns to spending to use the same returns exponent η from before the fire alarm but have some multiplier - i.e. xη returns to spending before the fire-alarm and kxη afterwards.
Strong agreement that a global moratorium would be great.
I'm unsure if aiming for a global moratorium is the best thing to aim for rather than a slowing of the race-like behaviour -- maybe a relevant similar case is whether to aim directly for the abolition of factory farms or just incremental improvements in welfare standards.
Thanks for the transcript and sharing this. The coverage seems pretty good, and the airplane crash analogy seems pretty helpful for communicating - I expect to use it in the future!
I agree. This lines with models of optimal spending I worked on which allowed for a post-fire alarm "crunch time" in which one can spend a significant fraction of remaining capital.
Nice post :-)
I agree with a lot of this, and generally think people should act "as if":
rather than reason/act as if "I might be in a simulation". In practice I think the two don't come apart massively, but I think there are risks in just adoping this latter approach (e.g. psychological).
On the two claims I think people should follow as if they were "true":
If we reject (1) then we can't rule out the case of our observations being non-veridical[1].
This would make it hard for us to influence the 'base' reality (which for utilitarians might be where most of the potential value is).
In a large enough multiverse, (1) seems likely. (Though if (1) is false and we're in a non-veridical simulation, it's unclear how much evidence we can have about whether we're in a large multiverse[2]).
Even if (1) did not hold, we might hope that some of observations are still useful in helping us influence the basement reality e.g. mathematical/logical/ philosophical arguments [3]. Of course simulators could be be arbitrarily Cartesian-demon like, but these seem safe to me to bracket-out.
I think (2) is not too tricky to accept and widelt accepted in these parts (though I think many academic decision theories would reject). Of the range of views the axes described in this post, one can be an evidentialist/EDT-like or use the algorithmic ontology.
Regarding veridicity, I think there's some risk of circularity here (and am not certain I've thread the needle):
With non-veridical simulations, I want to say something like ruling out simulations that some smart intelligence would say are of "unreasonable counterfactuals" (or are just complete fiction - i.e. they don't aim to capture some of the same dynamics as they believe they have in their own universe/multiverse)
So for example, if the totality of what exists ("the universe") was determined by some arbitrary (to some highly intelligent observer who understood physics as much as possible ) initial conditions, them I would class the simulation that runs "the universe but with different intial conditions" as reasonable.
With respect to an intelligent observer that believed in (and had good reason to believe) in a sufficiently large multiverse, then it's plausible that many counterfactuals are "reasonable".
Conditional on being in a non-veridical simulation, beyond our prior we have an anthropic update (if we are non-updateless) but little else to inform us about the world. (Perhaps our prior has expectations over which simulations that basement beings run)
Without (1), I don't think we can necessarily rule out simulators being able to run "counterlogical simulations" (this may only seem impossible to us because of the world they are happening to simulate).
Hey Bulat! Tristan from the Center on Long-Term Risk (CLR) here.
I don't use Telegram and don't know of an existing Telegram group. There are already several existing spaces across different platforms:
The Forum is also used for some discussion in this area.
Hope this helps!
Regarding Jackson's comment, I agree that 'dumping' money last-minute is a bit silly. Spending at a higher rate (and saving less) doesn't seem so crazy - which is what it seems you were considering.
My guess the modal outcome from AGI (and eventual ASI) is human disempowerment/extinction. Less confidently, I also suspect that most worlds where things go 'well' look weird and not not much like business-as-normal. For example, if we eventually have a sovereign ASI implement some form of coherent extrapolated volition I'm pretty unsure how (we would want) this to interact with individuals' capital. [Point 2. of this recent shortform feel adjacent - discussing CEV based on population rather than wealth].
Thanks for the list! Napkin AI is particularly cool.
Your LibGen link isn't working for me. Its Wikipedia page usually has live (& safe) links in the infobox. Same for Sci-Hub.
One recent service I've found is Readwise. It's similar to Pocket/Instapaper in adding web-pages to read later, but you can also upload your own EPUB and PDFs.
I thought about this a few years ago and have a post here.
I agree with Caleb's comment on the necessity to consider what a post-superintelligence world would look like, and whether capital could be usefully deployed. This post might be of interest.
My own guess is that it's most likely that capital won't be useful and that more aggressive donating makes sense.
The consequence of this for the "spend now vs spend later" debate is crudely modeled in The optimal timing of spending on AGI safety work, if one expects automated science to directly & predictably precede AGI. (Our model does not model labor, and instead considers [the AI risk community's] stocks of money, research and influence)
We suppose that after a 'fire alarm' funders can spend down their remaining capital, and that the returns to spending on safety research during this period can be higher than spending pre-fire alarm (although our implementation, as Phil Trammell points out, is subtly problematic, and I've not computed the results with a corrected approach).
Thanks again Phil for taking the read this through and for the in-depth feedback.
I hope to take some time to create a follow-up post, working in your suggestions and corrections as external updates (e.g. to the parameters of lower total AI risk funding, shorter Metaculus timelines).
This is a fair point.
The initial motivator for the project was for AI s-risk funding, of which there's pretty much one large funder (and not much work is done on AI s-risk reduction outside of people and organizations and people outside the effective altruism community) though this result is entirely on AI existential risk, which is less well modeled as a single actor.
My intuition is that the "one big actor" does work sufficiently well for the AI risk community given the shared goal (avoid an AI existential catastrophe) and my guess that a lot of the AI risk done by the community doesn't change the behaviour of AI labs much (i.e. it could be that they choose to put more effort into capabilities over safety because of work done by the AI risk community, but I'm pretty sure this isn't happening).
To comment on this particular error (though not to say that other errors Phil points to are not also unproblematic - I've yet to properly go through them), for what it's worth, the main results of the post suppose zero post fire alarm spending[1] and (fortunately) since in our results we use units of millions of dollars and take the initial capital to be on the order of 1000 $m, I don't think we face this problem of smaller η having the reverse than desired effect for
In a future version I expect I'll just take the post-fire alarm returns to spending to use the same returns exponent η from before the fire alarm but have some multiplier - i.e. xη returns to spending before the fire-alarm and kxη afterwards.
Though if one thinks there will many good opportunities to spend after a fire alarm, our main no-fire-alarm results would likely be an overestimate
Strong agreement that a global moratorium would be great.
I'm unsure if aiming for a global moratorium is the best thing to aim for rather than a slowing of the race-like behaviour -- maybe a relevant similar case is whether to aim directly for the abolition of factory farms or just incremental improvements in welfare standards.
This post from last year - What an actually pessimistic containment strategy looks like - has some good discussion on the topic of slowing down AGI research.
Thanks for the transcript and sharing this. The coverage seems pretty good, and the airplane crash analogy seems pretty helpful for communicating - I expect to use it in the future!
I agree. This lines with models of optimal spending I worked on which allowed for a post-fire alarm "crunch time" in which one can spend a significant fraction of remaining capital.