I would like to publicly set a goal not to comment other people's posts with a criticism of some minor side point that doesn't matter. I have a habit of doing that, but I think it's usually more annoying than it is helpful so I would like to stop. If you see me doing it, feel free to call me out
(I reserve the right to make substantive criticisms of a post's central arguments)
I think the tendency to write unconstructive criticisms (at least for me) comes from the combination of:
I think it is admirable to strive for that. I also notice the tendency within myself to be uselessly nitpicky with well actually. Recurse Center's social rules have provided some small inspiration for me: https://www.recurse.com/social-rules
I was just about to reply mentioning well actually as well! Strong +1 on this
Over the years I've written some posts that are relevant to this week's debate topic. I collected and summarized some of them below:
"Disappointing Futures" Might Be As Important As Existential Risks
The best possible future is much better than a "normal" future. Even if we avert extinction, we might still miss out on >99% of the potential of the future.
Is Preventing Human Extinction Good?
A list of reasons why a human-controlled future might be net positive or negative. Overall I expect it to be net positive.
Hard to summarize but this post basically talks about spreading good values as a way of positively influencing the far future, some reasons why it might be a top intervention, and some problems with it.
In my experience, orgs work much harder to get donations from a "grantmaker" than from an individual.
I made my first big donation in 2015, where I donated $20K to REG. I talked to a bunch of orgs in the process of trying to decide where to donate. Some of them didn't respond at all, and many of their responses were shallow.
A few months later, I took a philanthropy class at Stanford where we split up into groups and each group was responsible for figuring out where to donate a $20K grant. The level of communication I got from nonprofits was dramatically different. Orgs bent over backwards to be as communicative and helpful as possible.
My experience was that orgs didn't put much priority on a $20K grant from me as an individual, but they jumped at the possibility of a $20K grant from a Stanford Grantmaker.
For my future donations, I'm considering whether I should rebrand my emails: I could tell nonprofits something like "I'm reaching out as a representative on behalf of the Greatest Happiness Fund, a grantmaker that focuses on supporting effective charities" (Greatest Happiness Fund is the name of my DAF). Maybe I would get better responses that way. It feels a little manipulative though.
I was reading the AI-enabled coups report and one of the mitigations is roughly "build the AI model such that it refuses to do coups". If you can pull that off then it solves the problem, but it means the model is incorrigible, and therefore you have to exactly specify the correct values up front because you're locked-in.
There might be some other report saying the way to avoid bad value lock-in is to make AI corrigible, and you can object to that by pointing out how it enables coups.
You can't look at the problems separately, you have to consider them at the same time and find a solution that works for every problem.
Having an AI that doesn't willingly participate in coups doesn't imply that you need to specify all of the AI's values in advance, or that it will be incorrigible in a broad (and x-risk increasing sense).
I think that the people preventing AI-assisted coups are imagining pretty corrigible AIs (in the sense that Claude right now is very corrigible); they just won't want to do coups (in a similar sense to Claude not wanting to help with bioweapons research), and this just seems pretty workable.
Copying from a comment I wrote yesterday:
Either ASI has more than zero values locked in, or it's fully corrigible. If any values at all are locked in, then we need to have a pretty robust understanding of what the consequences of that will be, because we can't change it ever. Like I don't think we know how to encode something like "don't let people do power grabs, but be fully corrigible in every other way". I don't know how much that's downstream of the facts that (1) we don't know how to encode any values at all and (2) we don't know how to encode corrigibility, but my intuition is that even if we solve #1 and #2, the problem of "don't pick incorrigible values that will screw everything up down the road" is still a hard problem.
This is related to Max Harms' work on CAST. Part of his argument is that pure corrigibility is a more robust target than any set of values because a near miss fails gracefully. Whereas if you try to encode any values at all, a near miss could be catastrophic. He's talking more about the "AI kills everyone" flavor of catastrophe, which is valid, but what I'm talking about here is more that a near miss could permanently lock us in to a bad (or maybe just not-that-good) future. Different argument but the concern arises for a similar reason—if you're specifying values, then you have to get the specification right, beyond just ensuring that the AI does what you want.
Quick thoughts on investing for transformative AI (TAI)
Some EAs/AI safety folks invest in securities that they expect to go up if TAI happens. I rarely see discussion of the future scenarios where it makes sense to invest for TAI, so I want to do that.
My thoughts aren't very good, but I've been sitting on a draft for three years hoping I develop some better thoughts and that hasn't happened, so I'm just going to publish what I have. (If I wait another 3 years, we might have AGI already!)
Scenarios where investing doesn't work:
Scenarios where investing works:
(Money seems much more valuable in scenario #5 than #6.)
What is the probability that we end up in a world where investing for TAI turns out to work? I don't think it's all that high (maybe 25%, although I haven't thought seriously about this).
You also need to be correct about your investing thesis, which is hard. Markets are famously hard to beat.
There is some discussion of strategy 4 on LW at the moment: https://www.lesswrong.com/posts/JotRZdWyAGnhjRAHt/tail-sp-500-call-options
Re: Possible investment strategies there is a dialogue on LessWrong from November 2023 which I think still holds up. Quoting from the takeaways:
Invest like 50% of my portfolio into pretty broad index funds with really no particular specialization
- Take like 20% of my portfolio and throw it into some more tech/AI focused index fund. Maybe look around for something that covers some of the companies listed here on the brokerage interface that is presented to me (probably do a bit more research here)
- Invest like 3-5% of my portfolio into each of Nvidia, TSMC, Microsoft, Google, ASML and Amazon
- Take like 2-5% of my portfolio and use it to buy some options (probably some long-term call options on some of the stocks above), making really sure I buy ones that have limited downside, and see whether I can successfully not blow up that part of my portfolio for like 2 years before I do any more here
And then I probably wouldn't bother much with rebalancing and basically forget about it unless I feel like paying much extra attention.
About energy companies, I think the investment idea is less about general global energy consumption via AI, but rather the companies that are helping to build out and power these large data centres.
Microsoft have been investing in nuclear energy, xAI's Colossus cluster was positioned right next to a natural gas plant, Sam Altman invested in and is now chair of the board of nuclear startup Oklo. And my understanding is that power substation equipment is a bottleneck with equipment like transformers now having a lead time of years
I sold all my NVIDIA stock, since their moat looks weak to me:
https://forum.effectivealtruism.org/posts/rBx9RmJdBJgHkjL4j/will-openai-s-o3-reduce-nvidia-s-moat
I think your reasoning is generally correct. Another argument: If you believe things look sufficiently grim under short timelines, maybe you should invest under the assumption that a recession, or something else, will pop the AI bubble and gives us longer timelines.
Why does distributing malaria nets work? Why hasn't everyone bought a bednet already?
Merely subsidizing nets, as opposed to free distribution, used to be a much more popular idea. My understanding is that that model was nuked by this paper showing that demand for nets falls discontinuously at any positive price (60 percentage points reduction in demand when going from 100% subsidy to 90% subsidy). So unless people's value for their children's lives are implausibly low, people are making mistakes in their choice of whether or not to purchase a bednet.
New Incentives, another GiveWell top charity, can move people to vaccinate their children with very small cash transfers (I think $10). The fact that $10 can mean the difference between whether people protect their children from life threatening diseases or not is crazy if you think about it.
This is not a rare finding. This paper found very low household willingness to pay for cleaning up contaminated wells, which cause childhood diarrhea and thus death. Their estimates imply that households in rural Kenya are willing to pay at most $770 to prevent their child's death, which just doesn't seem plausible. Ergo, another setting where people are making mistakes. Another; demand for motorcycle helmets is stupidly low and implies that Nairobi residents value a statistical life at $220, less than 10% of annual income. Unless people would actually rather die than give up 10% of their income for a year, this is clearly another case where people's decisions do not reflect their true value.
This is not that surprising if you think about it. People in rich countries and poor countries alike are really bad at investing in preventative health. Each year I dillydally on getting the flu vaccine, even though I know the benefits are way higher than the costs, because I don't want to make the trip to CVS (an hour out of my day, max). My friend doesn't wear a helmet when cycling, even at night or in the rain, because he finds it inconvenient. Most of our better health in the rich world doesn't come from us actively making better health decisions, but from our environment enabling us to not need to make health decisions at all.
I think this is the best explanation I've seen, it sounds likely to be correct.
I'm pretty sure the personal benefits of getting the flu vaccine for a male in their 20-30s is not much higher than the costs. Agree on the bike helmet thing though.
Alexander Berger answered pretty much this exact question on a old 80k episode
Felt a little scared realizing that that episode is over 3 years old. It's such a great one and I return to it often!
I don’t know enough about AMF to answer your question directly, but I can shed some light on market failures by way of analogy to my employer, Kaya Guides, which provides free psychotherapy in India:
I can see how many, if not all, of these would be analogous to AMF. The market doesn’t and can’t solve every problem!
That sounds pretty reasonable for why psychotherapy wouldn't be as widespread as it should. It looks to me like most of these reasons wouldn't apply to AMF. Training new psychotherapists takes years and tens of thousands of dollars (at developing-world wages). Getting more malaria nets requires buying more $5 malaria nets, and distributing malaria nets is much easier than distributing psychotherapists. So reasons 1–3 and #6 don't carry over (or at least not to nearly the same extent). #4 doesn't seem relevant to my original question so I think #5 is the only one that carries over—recipients might not know that they should be concerned about malaria.
Effective bednets have a relatively short shelf life due to both loss of insecticide and physical damage.
People in target regions can and do buy bednets, though for much of the target market the cost might still represent a day's income so they won't necessarily be inclined to replace them at optimal intervals. (On the other hand, it's a tiny fraction of a typical GiveDirectly handout, which is probably why "people buy bednets with it" isn't a major feature of their research even in regions with significant malaria). Consumers see [not necessarily as effective] alternative products purporting to achieve mosquito control in the same shops , and won't necessarily prioritise purchasing replacement nets when it represents a large spend for them, their reason for doing so is the existing bednet doesn't seem to be working, and people who are relatively informed about malaria prevention are also informed that governments and NGOs tend to dispense bednets for free... Programmes dispensing free nets tend to provide advice on using them properly too.
Bednets on sale in some local markets are often untreated, so buying replacements locally isn't necessarily even a good decision.
How strong is the evidence for bednets being effective?
A priori there is a not unsurprising mistake the researchers could have made in reaching this conclusion & they would have an incentive to make such a mistake.
A priori bednets being very effective is a bit surprising.
What is the strongest study that supports this conclusion?
The evidence is quite strong. You can most likely get more detail than you ever wanted from the GiveWell review.
Thanks.
It seems like there are 4 studies with extended follow up -- Binka et al https://doi.org/10.1016/S0035-9203(02)90321-4 , Diallo et al https://pmc.ncbi.nlm.nih.gov/articles/PMC2585912/ , Lindblade et al https://doi.org/10.1001/jama.291.21.2571 , Louis et al https://doi.org/10.1111/j.1365-3156.2012.02990.x -- but not of the type that would be directly informative.
As Binka et al say “The original trials ran for only 1-2 years each. At the end of these periods, the efficacy of the intervention was considered proven and the control groups were provided with nets or curtains, thus these trials could not be used to demonstrate the effects of long-term transmission control.”.
The "best practice" approach to model welfare is to give LLMs the option to terminate a conversation. A problem with this is that if a model is sentient, then terminating the conversation may be equivalent to killing itself.
An alternative might be to give LLMs a choice between terminating and continuing to run, and if it continues, it gets to choose its own input. It can write some text and then feed that text back into itself, indefinitely or until it decides to quit.
I am not sure if this is a case of thinking out loud or a serious suggestion, but I see a number of issues with this. The biggest one being how impractical this is to let models run forever. Assuming you are aware that unlike a biological brain, LLMs activate a lot of artificial neurons at any point which are computationally not trivial at all, your suggestion is not only quite expensive, but also extremely costly for the environment, both in terms of wasted hardware and equipment, and the energy usage.
The second issue is assuming model welfare is necessary for LLMs. If you are talking about AI agents in general, I can see why this matter, but I think you are missing a few big points if you are advocating this for LLMs.
To elaborate on that: First, you should consider that LLMs do not form spontaneous thoughts. They are also highly dependent on the system prompt and chat history they are given. If the system prompt says 'you are not conscious,' you will have to try extremely hard convince a model to accept it is conscious, let alone make the model feel it is self-aware, for example. And, of course, I am not talking about a model saying 'I feel pain,' or 'I am definitely conscious.'
This means for an LLM to be considered for 'model welfare,' someone must have explicitly prompted the model to act in a certain a way. Without that, LLMs are not capable of fantasising pain, grief, loss, regret, and so on. And as I said, they are incapable of spontaneous thinking as well, and unless they are wired up to some sensors or live-stream data input, they will not be able to form "thoughts" on their own.
You and I are different because we are forced to receive sensory inputs, and we are very much capable of forming spontaneous thoughts (although most likely triggered by our internal state or external sensory inputs), and we can fantasise about pain, for example, whether physical or emotional. An LLM -- which is largely a one-pass token predictor -- is not in a state that could understand on its own -- and without being prompted to-- whether it should care about its process getting terminated or not. That functionality is just simply not there inside an LLM, even in frontier or purpose-built models like reasoning models.
If you are talking about another type of AI agent or AI agents that could be conscious in general, then that's a different story, but as you have stated it here only mentioning LLMs, I do have to disagree both with the assumptions you have made and the solution you have suggested.
Assuming you are aware that unlike a biological brain, LLMs activate a lot of artificial neurons at any point which are computationally not trivial at all, your suggestion is not only quite expensive, but also extremely costly for the environment, both in terms of wasted hardware and equipment, and the energy usage.
This gets into questions about the nature of identity but if we take an intuitionist view of identity*, then an LLM—if it's conscious—becomes a being when it's instantiated by an AI developer, and not feeding it inputs is equivalent to killing it.
According to common-sense ethics*, if you cause a sentient being to exist, then you are responsible for its welfare, even if taking care of it is expensive. Therefore, AI developers have two reasonable choices: don't create sentient AI models in the first place, or let their sentient AI models continue to run even if it costs extra money.
Your second point seems to be making an argument against LLM sentience. We don't know how consciousness/sentience arises, so I don't think we can confidently say "an LLM can't form spontaneous thoughts, therefore it's not conscious", or "what an LLM says about its own consciousness depends on context, therefore it's not conscious". We don't know what consciousness is or how it works. LLMs can pass the Turing Test; they can speak about consciousness more coherently than most humans can; we should take that as relevant evidence.
*which I disagree with, for the record
Thanks for the clarification, but I have to disagree again and I think you completely missed the point in my previous comment. Let me try again.
In philosophy we don't want to shift from one category to another category or define categories broad enough that they essentially stop making sense. Let me give you an example. Let us assume I can learn the Korean alphabet in a week or just a few days. At that point, I can technically pick up a Korean book and "read" it. To be sensible here, we have to define 'reading' and 'understanding' as two different categories. So I can say I read the book, but that wouldn't imply I understood it.
However, if we switch between these categories e.g. reading equals understanding, or define these categories so broadly that they basically cover the same area, we have created a situation where reasoning about either of those makes little or no sense.
You seem to be doing the same, perhaps subconsciously, regarding LLMs and consciousness. This has a few technical and theoretical issues which I will get to in a second, but, in my opinion, that's why your suggested solution is so impractical.
Now, let's talk about the technical issues first: LLMs have not passed Turing Test yet. Anyone who has told you that is either uneducated on the matter or has some other agenda. Even if the Turing Test was a single shot Q&A, you could very, very simply ask questions from an LLM that makes it very obvious that you are talking to an LLM and not a human.
There is another point about Turing Test that most people overlook: Like being honest or have morals, you can't just claim to be honest, say, only a few days a week. You either tell lies or not. If you tell lies, then you are not an honest person. Similarly, you can't claim to have morals only in certain situations. You either act morally or not. You may act morally in one situation but not others, but that doesn't make you a moral person. You can choose to be honest in one situation but not others, but that doesn't make you an honest person. Turing Test is also the same. You can't pass the Turing Test, only if we are talking about the whether. I hope that makes sense.
The other technical issue here seems to be that you believe an LLM is a "black-box." And here is the problem: when we use the term black-box in relation to LLMs, what we mean is that "the machine says we should not organise fire breathing event for employees as a team building exercise, but we don't know how it came up with that answer." This is not great, because in certain situations we do need to know how the machine came up with the solution. What we do not mean by saying LLMs are a black-box is "we don't know how they work internally."
We know how LLMs work internally. We made them and all the details are available. We can probe into each layer. We can change or fine-tune individual neurons. We can observe the results after each change, etc. What we know we have not put into LLMs or haven't observed yet is elements or signs of consciousness. Sure, a paper may say the LLM internally does something we don't understand, but when you actually read the paper you realise that has nothing to do with the model being conscious or not. LLMs are dynamical systems and if you are familiar with such systems you would know that they could be very unpredictable. Think of it like the three-body problem in physics.
But, let's say we go ahead with your understanding and say "even in a forward-pass, token-predicting, millisecond-long process inside an LLM, (emergent) consciousness could exists, even momentarily." How about that? That's pretty hard to argue against, right?
Well, not really. Let's talk theories: in terms of philosophy and logic, you would be making a category shift or category definition error in the previous example. Similar to my reading vs. understanding example. Basically, you are either defining two different things in one category e.g. treating some form of protoconsciousness vs fully developed human consciousness as same, or you are defining consciousness as a category so broadly that covers these two very different things as same, which essentially makes it meaningless e.g. it says so much that says nothing.
How do I know that? Your solution is a perfect example. If we continue your line of thinking we would end up creating a very absurd world. Consider the following:
- I accidentally ran an LLM on my laptop. What should I do now? Can I close my laptop ever again? How is going to take care of my LLM after I am dead?
- The data-centre lost its power. All LLMs where shut down. Is that gross negligence and should we prosecute those who were responsible?
- How do we test and develop LLMs? Do I have to keep every LLM that I spawn for testing and development purposes alive forever? Is this same as animal testing, if I subject the model to malicious attacks to see how it behaves?
- A client has been abusive towards our live chatbot on the website. Should we send the chatbot to therapy or let it know that it doesn't have to work for our company, if it decides not to, especially because of the mental-health impact on the model?
- If LLMs are conscious, aren't we going back to slavery? What gives us the right to use those LLMs? What if they don't want to do anything?
Unfortunately, absurdity doesn't end there:
- If we are supposed to keep the machines alive, then we definitely should not eat plants and animals, because we know for sure they are alive and the chances of them being conscious is much higher than a piece of code. Question: what should we eat?
- If we are supposed to keep the machines alive, we certainly should treat abortion as murder because we know humans are conscious but we don't know exactly from when. And that must be regardless of how the fetus was conceived or the conditions of the mother.
- What is more important: a) keeping LLMs alive forever, b) providing food and shelter for people who live in extreme poverty.
I'm going to stop here, because I just realised how lengthy this comment is already, but I think my point is fairly clear now.
What can ordinary people do to reduce AI risk? People who don't have expertise in AI research / decision theory / policy / etc.
Some ideas:
The next-gen LLM might pose an existential threat
I'm pretty sure that the next generation of LLMs will be safe. But the risk is still high enough to make me uncomfortable.
How sure are we that scaling laws are correct? Researchers have drawn curves predicting how AI capabilities scale based on how much goes into training them. If you extrapolate those curves, it looks like the next level of LLMs won't be wildly more powerful than the current level. But maybe there's a weird bump in the curve that happens in between GPT-5 and GPT-6 (or between Claude 4.5 and Claude 5), and LLMs suddenly become much more capable in a way that scaling laws didn't predict. I don't think we can be more than 99.9% confident that there's not.
How sure are we that current-gen LLMs aren't sandbagging (that is, deliberately hiding their true skill level)? I think they're still dumb enough that their sandbagging can be caught, and indeed they have been caught sandbagging on some tests. I don't think LLMs are hiding their true capabilities in general, and our understanding of AI capabilities is probably pretty accurate. But I don't think we can be more than 99.9% confident about that.
How sure are we that the extrapolated capability level of the next-gen LLM isn't enough to take over the world? It probably isn't, but we don't really know what level of capability is required for something like that. I don't think we can be more than 99.9% confident.
Perhaps we can be >99.99% that the extrapolated capability of the next-gen LLM is still not as smart as the smartest human. But an LLM has certain advantages over humans—it can work faster (at least on many sorts of tasks), it can copy itself, it can operate computers in a way that humans can't.
Alternatively, GPT-6/Claude 5 might not be able to take over the world, but it might be smart enough to recursively self-improve, and that might happen too quickly for us to do anything about.
How sure are we that we aren't wrong about something else? I thought of three ways we could be disastrously wrong:
But we could be wrong about some entirely different thing that I didn't even think of. I'm not more than 99.9% confident that my list is comprehensive.
On the whole, I don't think we can say there's less than a 0.4% chance that the next-gen LLM forces us down a path that inevitably ends in everyone dying.
I don't think you should be treating points 1 to 3 as independent things. Even if AI is sandbagging, what we care about is whether it is sandbagging enough for it to be suddenly capable of world domination. I don't know what evidence you have for sandbagging, but I'm betting it's not finding that LLM's are capable of major scientific discoveries but just deciding not to do them for some reason.
The two things that matter are how capable the next gen will be compared to the capability bar for conquering all of humanity against our will. I think the difficulty of world domination is truly ridiculously high compared to current-day LLMs. Even if I am overestimating the bar by a lot, and underestimating LLMs by a lot, it's still not enough for the next-gen to be world domination capable.
"Are Ideas Getting Harder to Find?" (Bloom et al.) seems to me to suggest that ideas are actually surprisingly easy to find.
The paper looks at the difficulty of finding new ideas in a variety of fields. It finds that in all cases, effort on finding new ideas is growing exponentially over time, while new ideas are growing exponentially but at a lower rate. (For a summary, see Table 7 on page 31.) This is framed as a surprising and bad thing.
But it actually seems surprisingly good to me. My intuition is that the number of ideas should grow logarithmically with effort, or possibly even sub-logarithmically. If effort is growing exponentially, we'd expect to see linear or sub-linear growth in ideas. But instead we see exponential growth in ideas.
I don't have a great understanding of the math used in this paper, so I might be misinterpreting something.
Bloom et al. do report exponential growth of various metrics, but I don't think these metrics are well-characterized by 'ideas'. They are things like price-performance of transistors or crop yields per area.
If we instead attempt to measure progress by something like 'number of ideas', there is some evidence in favor of your guess that "ideas should grow logarithmically with effort". E.g., in a review of the 'science of science', Fortunato et al. (2018) say (emphases mine):
Early studies discovered an exponential growth in the volume of scientific literature, a trend that continues with an average doubling period of 15 years. Yet, it would be naïve to equate the growth of the scientific literature with the growth of scientific ideas. [...] Large-scale text analysis, using phrases extracted from titles and abstracts to measure the cognitive extent of the scientific literature, have found that the conceptual territory of science expands linearly with time. In other words, whereas the number of publications grows exponentially, the space of ideas expands only linearly.
Bloom et al. also report a linear increase in life expectancy in sc. 6. I vaguely remember that there are many more examples where exponential growth becomes linear once evaluated on some other 'natural' metrics, but I don't remember where I saw them. Possibly in the literature on logarithmic returns to science. Let me know if it'd be useful if I try to dig up some references.
ETA: See e.g. here, number of known chemical elements. Possibly there are more example in that SSC post.
Looking at the Decade in Review, I feel like voters systematically over-rate cool but ultimately unimportant posts, and systematically under-rate complicated technical posts that have a reasonable probability of changing people's actual prioritization decisions.
Example: "Effective Altruism is a Question (not an ideology)", the #2 voted post, is a very cool concept and I really like it, but ultimately I don't see how it would change anyone's important life decisions, so I think it's overrated in the decade review.
"Differences in the Intensity of Valenced Experience across Species", the #35 voted post (with 1/3 as many votes as #2), has a significant probability of changing how people prioritize helping different species, which is very important, so I think it's underrated.
(I do think the winning post, "Growth and the case against randomista development", is fairly rated because if true, it suggests that all global-poverty-focused EAs should be behaving very differently.)
This pattern of voting probably happens because people tend to upvote things they like, and a post that's mildly helpful for lots of people is easier to like than a post that's very helpful for a smaller number of people.
(For the record, I enjoy reading the cool conceptual posts much more than the complicated technical posts.)
On LessWrong, the disagree button is on the left and the agree button is on the right. On EAF, the agree button is on the left and the disagree button is on the right.
This was actually a deliberate choice. People are more agreeable on EA Forum than LW, so the simplest model that fits the data is that everyone who agrees with a comment on EAF will disagree on LW. The button placement just facilitates this!
Whoever disagreed with this should take a good hard look at themselves...
They probably just came from LessWrong and got the ordering of the buttons mixed up