New title is better!

After making sure we don't all die, this should be the first priority

Please do not write clickbait titles like this. Put the subject matter in the title.

But I think expecting returns something like 50-100% p.a. would be reasonable, based on historic performance of these funds.

You can't have it both ways. Either you can extrapolate from historical performance, in which case you should use long-run market returns of ~5%. Or you have a specific view on what's going to outperform, in which case it's a question of what your expectations are. You can't just take a fund that performed extremely well over the last 3–5 years and then say that level of performance will continue. If some thesis did really well in the past, then everyone else can also see that it did well, and you should assume it's now priced in unless you have some marginal thesis for why it's not priced in.

In fact, in the long run, stocks have exhibited 3–5 year reversals—that is, stocks that perform particularly well over the last 3–5 years tend to underperform the market going forward.[1] Mutual funds haven't exhibited reversals, but funds that outperform tend to regress to average performance.[2] At minimum, your thesis needs to have some explanation of why that's not going to happen this time.

I'm not saying you're wrong. AI stocks may well be a good investment. But I am pretty unimpressed by most arguments I've seen. "The last 5 years saw 50–100% return, therefore that same level of return will continue" is not a good argument—by (weak) default, it's evidence that future returns will be lower than the market, not higher. (I wrote more on my thoughts in a recent post, especially this section.)

[1] Fama, E. F., & French, K. R. (1988). Permanent and Temporary Components of Stock Prices. Journal of Political Economy, 96(2), 246–273. http://www.jstor.org/stable/1833108

[2] Berk, J. B., & Green, R. C. (2004). Mutual Fund Flows and Performance in Rational Markets. Journal of Political Economy, 112(6), 1269–1295. https://doi.org/10.1086/424739

How would you model the expected investment rate of return? I generally assume a 5% real return, which is small enough that it wouldn't affect the model in OP very much. If want to incorporate a large expected return on AI investments, then I don't know how to do that anymore.

I basically agree, although "dangerous approaches are tamped down" is doing most of the work here IMO. By default (i.e. no tamping-down), I expect the situation with a weakly-superhuman Scientist AI to be:

1. a small number of sane people ask "are we doomed if we go with the following alignment strategy", and when it says yes, they don't do it
2. a lot of people don't bother to ask at all, they just ask the Scientist AI how to build ASI
3. a lot of people say "we have to build ASI before the reckless people in group 2", they build ASI using their best-guess alignment strategy that has an 88.5% chance of failing, and we die with 88.5% probability

(I think Bengio would agree that this is a concern, and would agree that we need global coordination on AI safety to make this work.)

According to a survey of Congressional staffers, they care about personalized emails more than phone calls, and phone calls (much) more than form emails:

https://mdickens.me/assets/images/Congress-influence.png

(the way my brain works is that I'd much rather spend 5 minutes writing a personalized email than spend 5 minutes looking up the name of my representative and a relevant law to reference. YMMV)

I realize this comment makes me sound lazy (that's because I am lazy), but the way the doc is presented gives me too much work to do, in a way that's avoidable.

I'm not going to call my representative because I hate phone calls, so I'll just focus on the email part. The email template requires me to look up my senator's name and look up relevant laws from a list. I'm too lazy to do that. I would send a letter if there were a website where I could put in my state and it would auto-fill my senator's name and the list of relevant laws. It would not be very hard to build a web form to do that.

ASPCA has a web form that requires less work on my part. It still require me to fill out a bunch of personal info but I believe that part is a requirement. Although I'm not sure it auto-fills the rep's name, and it doesn't give any option to customize the message.

https://ifanyonebuildsit.com/act/letter (unrelated to farm bill) is a good example of a page that auto-fills as much information as possible.

Here are three other responses that I didn't see addressed. I only skimmed the original article so apologies if these were addressed there.

1. The saturation view does not obviously avoid the repugnant conclusion. (To be clear, I don't think the repugnant conclusion should be avoided; I think people's intuitions are misled by scope insensitivity.) The number of morally distinct experiences is probably very large—perhaps proportional to 2 to the power of the number of neurons, which would mean there are vastly more possible experiences than will ever realize. You could easily have a world with (say) 10^20 barely-worth-living lives that are all sufficiently distinct such that this world, on the saturation view, is better than a world with only (say) 10^7 extremely good lives.
2. I strongly suspect that the difference between the total view and the saturation view is a factual question, not a moral one. I suspect that if we had a solution to the hard problem of consciousness, then it would follow from that solution whether we should care about repeated experiences equally.
3. I have an intuition that the value of my own life is innate, and not related to who else might have a similar life. If someone offers me a good experience, I don't say, "Wait, let me check if anyone else has had an identical experience at some point in the past, because if so, my experience won't have any value." Or if I have a really good experience and then after the fact someone tells me that it doesn't count because a clone of me already had the same experience last week, I'd say: "No, you're wrong; I just felt that experience, and it was good, and I know it was good because I experienced it being good, and you can't take that away from me."

A fourth point, which the article did address:

First, you could reinterpret the intuition as entirely instrumental. Perhaps variety matters only because it tends to support exploration, learning, and other welfare-promoting goods. If so, then our intuition that Variety is better than Homogeneity is a misfire, attributing intrinsic value to something that has only instrumental value.

[the article's reply:]

Imagine some truly wonderful moment — of falling in love, or a flash of creative insight, or communing with the spiritual. Suppose that this moment is far more wonderful than anything that humanity has experienced to date: you or I would give up years of ordinary happy life just to experience such a peak of joy. But now suppose that this moment is just ever so slightly less good than some other moment that could be produced, with the same resources. So, instead, the world consists of a near-endless reliving of that other moment, and that first moment is never experienced at all.

To me, this feels like a tremendous loss. The world has omitted something wonderful that could have been created.

This doesn't seem like a satisfying response to me. I could say that your feeling of tremendous loss stems from the instrumental value of novel experiences; you haven't done anything to dispel that argument.

I would consider it from a first person POV. Suppose I'm given the choice to experience something amazing, or something a bit less amazing. If I experienced either thing in the past, then I have no memory of it (to guarantee that it's a proper replication of the amazing experience, and not a version altered by memory). I would pick the amazing experience, and not the lesser one.

IMO a "Scientist AI" is more promising in a world where we first get a global ban on superintelligent AI, or something else that prevents anyone from building "high-impact AI". Then AI developers coalesce around "Scientist AI" as a safe approach and develop it carefully.

(I still think a Scientist AI would result in everyone dying, but at least it's a better starting point)

Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don't expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can't deal with the sharp left turn.

(I would make an exception for CaML-style alignment-to-animals work, but that's not about AI safety as it's normally construed.)