MichaelDickens's Quick takes

This was actually a deliberate choice. People are more agreeable on EA Forum than LW, so the simplest model that fits the data is that everyone who agrees with a comment on EAF will disagree on LW. The button placement just facilitates this!

Whoever disagreed with this should take a good hard look at themselves...

Having an AI that doesn't willingly participate in coups doesn't imply that you need to specify all of the AI's values in advance, or that it will be incorrigible in a broad (and x-risk increasing sense).

I think that the people preventing AI-assisted coups are imagining pretty corrigible AIs (in the sense that Claude right now is very corrigible); they just won't want to do coups (in a similar sense to Claude not wanting to help with bioweapons research), and this just seems pretty workable. 

I think the tendency to write unconstructive criticisms (at least for me) comes from the combination of:

1. I have a strong urge to comment on anything that looks incorrect
2. Writing substantive criticisms of a post (often) requires grokking the whole post and thinking deeply about it, which is hard. Criticizing some specific sentence is easy because my brain instantly surfaces the criticism when I read the sentence

There is some discussion of strategy 4 on LW at the moment: https://www.lesswrong.com/posts/JotRZdWyAGnhjRAHt/tail-sp-500-call-options

Re: Possible investment strategies there is a dialogue on LessWrong from November 2023 which I think still holds up. Quoting from the takeaways:

Invest like 50% of my portfolio into pretty broad index funds with really no particular specialization

• Take like 20% of my portfolio and throw it into some more tech/AI focused index fund. Maybe look around for something that covers some of the companies listed here on the brokerage interface that is presented to me (probably do a bit more research here)
• Invest like 3-5% of my portfolio into each of Nvidia, TSMC, Microsoft, Google, ASML and Amazon
• Take like 2-5% of my portfolio and use it to buy some options (probably some long-term call options on some of the stocks above), making really sure I buy ones that have limited downside, and see whether I can successfully not blow up that part of my portfolio for like 2 years before I do any more here

And then I probably wouldn't bother much with rebalancing and basically forget about it unless I feel like paying much extra attention.

About energy companies, I think the investment idea is less about general global energy consumption via AI, but rather the companies that are helping to build out and power these large data centres.

Microsoft have been investing in nuclear energy, xAI's Colossus cluster was positioned right next to a natural gas plant, Sam Altman invested in and is now chair of the board of nuclear startup Oklo. And my understanding is that power substation equipment is a bottleneck with equipment like transformers now having a lead time of years

I sold all my NVIDIA stock, since their moat looks weak to me:

https://forum.effectivealtruism.org/posts/rBx9RmJdBJgHkjL4j/will-openai-s-o3-reduce-nvidia-s-moat

I think your reasoning is generally correct. Another argument: If you believe things look sufficiently grim under short timelines, maybe you should invest under the assumption that a recession, or something else, will pop the AI bubble and gives us longer timelines.

I am not sure if this is a case of thinking out loud or a serious suggestion, but I see a number of issues with this. The biggest one being how impractical this is to let models run forever. Assuming you are aware that unlike a biological brain, LLMs activate a lot of artificial neurons at any point which are computationally not trivial at all, your suggestion is not only quite expensive, but also extremely costly for the environment, both in terms of wasted hardware and equipment, and the energy usage.

The second issue is assuming model welfare is necessary for LLMs. If you are talking about AI agents in general, I can see why this matter, but I think you are missing a few big points if you are advocating this for LLMs.

To elaborate on that: First, you should consider that LLMs do not form spontaneous thoughts. They are also highly dependent on the system prompt and chat history they are given. If the system prompt says 'you are not conscious,' you will have to try extremely hard convince a model to accept it is conscious, let alone make the model feel it is self-aware, for example. And, of course, I am not talking about a model saying 'I feel pain,' or 'I am definitely conscious.' 

This means for an LLM to be considered for 'model welfare,' someone must have explicitly prompted the model to act in a certain a way. Without that, LLMs are not capable of fantasising pain, grief, loss, regret, and so on. And as I said, they are incapable of spontaneous thinking as well, and unless they are wired up to some sensors or live-stream data input, they will not be able to form "thoughts" on their own.

You and I are different because we are forced to receive sensory inputs, and we are very much capable of forming spontaneous thoughts (although most likely triggered by our internal state or external sensory inputs), and we can fantasise about pain, for example, whether physical or emotional. An LLM -- which is largely a one-pass token predictor -- is not in a state that could understand on its own -- and without being prompted to-- whether it should care about its process getting terminated or not. That functionality is just simply not there inside an LLM, even in frontier or purpose-built models like reasoning models.

If you are talking about another type of AI agent or AI agents that could be conscious in general, then that's a different story, but as you have stated it here only mentioning LLMs, I do have to disagree both with the assumptions you have made and the solution you have suggested.

Merely subsidizing nets, as opposed to free distribution, used to be a much more popular idea. My understanding is that that model was nuked by this paper showing that demand for nets falls discontinuously at any positive price (60 percentage points reduction in demand when going from 100% subsidy to 90% subsidy). So unless people's value for their children's lives are implausibly low, people are making mistakes in their choice of whether or not to purchase a bednet.

New Incentives, another GiveWell top charity, can move people to vaccinate their children with very small cash transfers (I think $10). The fact that $10 can mean the difference between whether people protect their children from life threatening diseases or not is crazy if you think about it.

This is not a rare finding. This paper found very low household willingness to pay for cleaning up contaminated wells, which cause childhood diarrhea and thus death. Their estimates imply that households in rural Kenya are willing to pay at most $770 to prevent their child's death, which just doesn't seem plausible. Ergo, another setting where people are making mistakes. Another; demand for motorcycle helmets is stupidly low and implies that Nairobi residents value a statistical life at $220, less than 10% of annual income. Unless people would actually rather die than give up 10% of their income for a year, this is clearly another case where people's decisions do not reflect their true value.

This is not that surprising if you think about it. People in rich countries and poor countries alike are really bad at investing in preventative health. Each year I dillydally on getting the flu vaccine, even though I know the benefits are way higher than the costs, because I don't want to make the trip to CVS (an hour out of my day, max). My friend doesn't wear a helmet when cycling, even at night or in the rain, because he finds it inconvenient. Most of our better health in the rich world doesn't come from us actively making better health decisions, but from our environment enabling us to not need to make health decisions at all.

Alexander Berger answered pretty much this exact question on a old 80k episode

I don’t know enough about AMF to answer your question directly, but I can shed some light on market failures by way of analogy to my employer, Kaya Guides, which provides free psychotherapy in India:

1. Our beneficiaries usually can’t afford psychotherapy outright
2. They sometimes live rurally, and can’t travel to places that do psychotherapy in person
3. There are not enough psychotherapists in India for everyone to receive it
4. The government, equally, don’t have the capacity or interest to develop the mental health sector enough (against competing health priorities) to make free treatment available
5. Our beneficiaries usually don’t know what psychotherapy is, or that they have a problem at all, nor that it can be treated
6. We are incentivised to make psychotherapy as cheap as possible to reach the worst-served portion of the market, while for-profits are incentivised to compete in more lucrative parts of the market

I can see how many, if not all, of these would be analogous to AMF. The market doesn’t and can’t solve every problem!

Effective bednets have a relatively short shelf life due to both loss of insecticide and physical damage.

People in target regions can and do buy bednets, though for much of the target market the cost might still represent a day's income so they won't necessarily be inclined to replace them at optimal intervals. (On the other hand, it's a tiny fraction of a typical GiveDirectly handout, which is probably why "people buy bednets with it" isn't a major feature of their research even in regions with significant malaria). Consumers see [not necessarily as effective] alternative products purporting to achieve mosquito control in the same shops , and won't necessarily prioritise purchasing replacement nets when it represents a large spend for them, their reason for doing so is the existing bednet doesn't seem to be working, and people who are relatively informed about malaria prevention are also informed that governments and NGOs tend to dispense bednets for free...      Programmes dispensing free nets tend to provide advice on using them properly too.

Bednets on sale in some local markets are often untreated, so buying replacements locally isn't necessarily even a good decision.

How strong is the evidence for bednets being effective?

A priori there is a not unsurprising mistake the researchers could have made in reaching this conclusion & they would have an incentive to make such a mistake.

A priori bednets being very effective is a bit surprising.

What is the strongest study that supports this conclusion?

I don't think you should be treating points 1 to 3 as independent things. Even if AI is sandbagging, what we care about is whether it is sandbagging enough for it to be suddenly capable of world domination. I don't know what evidence you have for sandbagging, but I'm betting it's not finding that LLM's are capable of major scientific discoveries but just deciding not to do them for some reason. 

The two things that matter are how capable the next gen will be compared to the capability bar for conquering all of humanity against our will. I think the difficulty of world domination is truly ridiculously high compared to current-day LLMs. Even if I am overestimating the bar by a lot, and underestimating LLMs by a lot, it's still not enough for the next-gen to be world domination capable. 

Bloom et al. do report exponential growth of various metrics, but I don't think these metrics are well-characterized by 'ideas'. They are things like price-performance of transistors or crop yields per area.

If we instead attempt to measure progress by something like 'number of ideas', there is some evidence in favor of your guess that "ideas should grow logarithmically with effort". E.g., in a review of the 'science of science', Fortunato et al. (2018) say (emphases mine):

Early studies discovered an exponential growth in the volume of scientific literature, a trend that continues with an average doubling period of 15 years. Yet, it would be naïve to equate the growth of the scientific literature with the growth of scientific ideas. [...] Large-scale text analysis, using phrases extracted from titles and abstracts to measure the cognitive extent of the scientific literature, have found that the conceptual territory of science expands linearly with time. In other words, whereas the number of publications grows exponentially, the space of ideas expands only linearly.

Bloom et al. also report a linear increase in life expectancy in sc. 6. I vaguely remember that there are many more examples where exponential growth becomes linear once evaluated on some other 'natural' metrics, but I don't remember where I saw them. Possibly in the literature on logarithmic returns to science. Let me know if it'd be useful if I try to dig up some references.

ETA: See e.g. here, number of known chemical elements. Possibly there are more example in that SSC post.