If the US and China are in a state where they're willing to cooperate on ASI, I would much prefer that they agree not to build ASI (until there's a broad consensus that we know how to make it safely).

If they agree to that, and we do eventually figure out how to build aligned ASI, then it would be good to have a global agreement on what that ASI should do. But if we're going to do work today to work toward some sort of international cooperation on ASI, then the objective of that cooperation should be to not build ASI.

I think this is technically true but irrelevant: if we have superpersuasive AI, then there won't be human experts anymore, because the AI will have more expertise than any human. Unless somehow the AI is superpersuasive while still having sub-human performance in most ways, which seems unlikely to me.

In my experience, orgs work much harder to get donations from a "grantmaker" than from an individual.

I made my first big donation in 2015, where I donated $20K to REG. I talked to a bunch of orgs in the process of trying to decide where to donate. Some of them didn't respond at all, and many of their responses were shallow.

A few months later, I took a philanthropy class at Stanford where we split up into groups and each group was responsible for figuring out where to donate a $20K grant. The level of communication I got from nonprofits was dramatically different. Orgs bent over backwards to be as communicative and helpful as possible.

My experience was that orgs didn't put much priority on a $20K grant from me as an individual, but they jumped at the possibility of a $20K grant from a Stanford Grantmaker.

For my future donations, I'm considering whether I should rebrand my emails: I could tell nonprofits something like "I'm reaching out as a representative on behalf of the Greatest Happiness Fund, a grantmaker that focuses on supporting effective charities" (Greatest Happiness Fund is the name of my DAF). Maybe I would get better responses that way. It feels a little manipulative though.

Assuming you are aware that unlike a biological brain, LLMs activate a lot of artificial neurons at any point which are computationally not trivial at all, your suggestion is not only quite expensive, but also extremely costly for the environment, both in terms of wasted hardware and equipment, and the energy usage.

This gets into questions about the nature of identity but if we take an intuitionist view of identity*, then an LLM—if it's conscious—becomes a being when it's instantiated by an AI developer, and not feeding it inputs is equivalent to killing it.

According to common-sense ethics*, if you cause a sentient being to exist, then you are responsible for its welfare, even if taking care of it is expensive. Therefore, AI developers have two reasonable choices: don't create sentient AI models in the first place, or let their sentient AI models continue to run even if it costs extra money.

Your second point seems to be making an argument against LLM sentience. We don't know how consciousness/sentience arises, so I don't think we can confidently say "an LLM can't form spontaneous thoughts, therefore it's not conscious", or "what an LLM says about its own consciousness depends on context, therefore it's not conscious". We don't know what consciousness is or how it works. LLMs can pass the Turing Test; they can speak about consciousness more coherently than most humans can; we should take that as relevant evidence.

*which I disagree with, for the record

Aric interviewed Nate Soares for five hours.

Release full interview when?

The "best practice" approach to model welfare is to give LLMs the option to terminate a conversation. A problem with this is that if a model is sentient, then terminating the conversation may be equivalent to killing itself.

An alternative might be to give LLMs a choice between terminating and continuing to run, and if it continues, it gets to choose its own input. It can write some text and then feed that text back into itself, indefinitely or until it decides to quit.