This is a special post for quick takes by huw. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
From SE Geyges: Is METR a meaningful check on Anthropic?.
Part of the reason we are in this position is because nobody outside the EA community cares about AI safety enough to fund it or work in it. I’m very sympathetic to this; some of these close connections are inevitable.
However, it can also be true that these connections are completely unacceptable (edit), and that Anthropic should be trying harder than they are to find or create auditors that are genuinely independent. How do we do that?
6
I think METRs checks are meaningful and maybe the best we have at the moment. They are also like you say compromised and the conflict of interests are immense with huge personal overlap between the labs and safety orgs, and funding streams too.
Geyges seems largely correct, but if we can't convince governments to regulate properly its better METR is in there doing it. After all METR exposed more about the hugging face hack than Open AI did on its own.
I don't think a framing of "these connections are completely unacceptable" is helpful given these problems. I think "compromised and far from ideal" is a better framing. Its better to do something than do nothing. I agree its best if government installed internal auditors like they do for banks, but that ain't happening any time soon.
Companies like Anthropic and Open AI are selfish animals by nature. They may have moments where good humans inside might do the right thing, but fundamentally they thirst for profit and growth. After IPO this will only get worse. We should never expect a company to regulate itself or its industry. Self regulation for harmful companies is a terrible idea and never works.
How can ANthropic "create" a genuinely independent auditor? this seems impossible, almost and Oxymoron.
Situational Awareness (the hedge unhedged fund) has imploded. It seems much of their gains were from overleveraging their trades, which has then caused them to, in some readings, undergo the largest absolute short-term fund loss in history (over $10B in a few weeks).
They certainly lost a lot of money in July and by being forced to sell, but per the WSJ Aschenbrenner wrote that the fund is still up 80% on the year.
The fund doesn't seem to be closing, and it looks like they weren't forced to sell their stake in Anthropic, so it's not clear to me how much of a failure or a success the project as a whole is so far.
See also this from the Financial Times.
Edit: see also this tweet claiming to be the full letter Aschenbrenner sent to his LPs last night.
9
Their gains were from leveraged directional bets and their losses were from leveraged directional bets. Seems they were margin called, so their liquidity was squeezed - poor risk management. It seems that they have learned from this and are now using “fully paid for” derivatives, so presumably call options whose premium they can afford.
Hedge funds are not necessarily hedged, but eg long/short equity hedge funds are (while macro hedge funds can have all kinds of unhedged macro bets on.)
Sadly it appears that adjacency to EA correlates with poor risk management, I’m wondering whether this is somewhat structural (beyond eg the age group…)
5
To this community’s credit, much of the discussion of his paper and career plans was skeptical. I think Aschenbrenner is a useful template for the kind of person that we need to have our guards up around, for a future where large sums of money enter the movement through the Third Wave.
I don’t think he was working in bad faith, and probably didn’t have bad intentions. However, it is objectively nuts to put a 24-year-old with no experience in charge of a $50B hedge fund. This is very different from giving a 24-year-old a $100k/year charity or EA Funds project. We should not assume that clear skills and intelligence in one domain should ever cleanly map to others.
We should also have learned the lesson from FTX that an inexperienced person speaking with confidence, or dazzling people with technical prowess, is much more likely to simply be inexperienced than a child prodigy. You are not immune to the Dunning-Kruger effect.
If I may speculate, both of these seem to have roots in some of the broader psychological flaws underpinning the ratsphere and the valley more broadly—namely, that intelligence is a fixed, general attribute, and that precociousness is overlooked only because traditional power structures are trying to protect themselves. These are not necessarily or always untrue, but some parts of the community who notice that these are not generally accepted ideas overreact, and overindex on them.
There will be more Aschenbrenners approaching this community in the future, and we should welcome them—but we shouldn’t hand them the controls right away just because they dazzle us.
(P.S. I notice that Leopold’s CV also includes ‘Fund Manager, FTX Future Fund’)
9
That would indeed be nuts, but I think that's also clearly not what happened. My understanding is that he founded the fund himself, and he initially raised $100-200 million from a bunch of non-EA investors. See https://www.forourposterity.com/
Thanks to their spectacular returns they had a lot more assets under management, but my sense is that this is from normal investors who buy into the "AI is going to be big" thesis and were trying to make a profit.
I also found this article interesting, as well as the fact that Bloom Energy, IREN, Nebius, SanDisk, Micron, CoreWeave, and so on are up 10% to 30% since Wednesday
2
I don’t think the difference between $100m and $50B is that substantial for my point. Even though they are two orders of magnitude apart, I think the limit on how much money you should trust with someone who has no prior experience managing money is lower than that.
7
1. They seem to have made a lot of money by investing in his fund, even after this week, so I find it hard to criticize them for it
2. Aschenbrenner didn't launch the fund alone, his cofounder Carl Shulman had previous experience managing money
3. We don't know the rest of their portfolio and how situational awareness fits into that (e.g. maybe it's partially a small hedge against AI making Stripe worthless, or who knows)
3
? It is his fund. He started it.
I feel with the FTX and rat culture reflection, you are reading into the situation too much.
The losses, as you pointed out, happened due to highly leveraged bets. He didn't expect memory stocks to bleed as much as they did. Besides, it is possible that the fund will survive because his core thesis has paid off exceptionally well so far. He also has a rumored $5B equity stake in a certain large AI company.
Tough time for Situational Awareness LP, but I hope they make it!
? It is his fund. He started it.
His creditors supplied the money—it is their judgement I’m questioning.
The losses, as you pointed out, happened due to highly leveraged bets. He didn't expect memory stocks to bleed as much as they did.
This is disqualifying if you are running a hedge fund. It’s literally in the name—you are supposed to hedge your positions in order to prevent an unexpected situation from tanking the whole fund.
Besides, it is possible that the fund will survive because his core thesis has paid off exceptionally well so far
We don’t judge funds by whether they don’t go bankrupt, we judge them by their performance against a market index over a long period of time. Even if the positions are net up by 2× or so, this is not particularly impressive in and of itself over a short period, because of survivorship bias. If you make a bunch of stupidly leveraged bets on different sectors, one of them is likely to pay off very well, but not for long, and not through downturns.
The AI sector has monotonically gone up since the release of ChatGPT—any overleveraged investor in this space would be likely to produce incredible gains. If one’s fund gets obliterated at the first market downturn because one was overleveraged, all this proves is that you managed your fund badly, not that you’re some kind of savant genius market whisperer.
(c.f. anything written about Cathie Wood in 2022—a lot of it has aged very poorly)
7
It's my understanding that this hasn't been the definition of a hedge fund for some time (ChatGPT agrees). I'm assume that the investors knew that Situational Awareness was not a hedge fund in the sense you're describing one.
Of course, it might still be possible that investors weren't aware of the amount of risks the fund was taking on.
3
Yeah this has indeed never really been the definition of a hedge fund. Only a subset of hedge funds are approximately point-in-time market neutral.
6
I made claude do some quick maths:
Apparently, even $1 invested into Situational Awareness LP 6 months ago is worth $1.78 now. Totally possible he is not a savant whisperer and had enough knowledge and connections to make great bets, but the bets are still paying off. Time will tell if their hedging strategy is actually sound or just lucky.
But I don't think this is comparable to FTX or contains any lessons for the EA community. Is Situational Awareness LP an EA venture? I could be wrong but I don't think so. I have always thought of it as "oh I guess Aschenbrenner is doing his own thing, cool."
1
It seems prudent that banks were quick to margin call him, they probably tightened this kind of stuff after archegos. They still had net assets, so this was early enough.
4
The article ends by saying: "It still holds a big chunk of harder-to-sell positions in privately held companies such as Anthropic. It remains to be seen if those companies will live up to their lofty valuations if and when they hit the public markets." I wonder if this means that the fund did not get wiped out completely - otherwise Citadel (who bought all of their publicly traded stock positions) would have also taken their Anthropic shares?
-3
This is totally relevant to EA because it’s another case of youthful hubris vs veteran experience. EA is a place an elite graduate can meet in a bar with some other cool kids and do a napkin pitch for their first charity with zero field experience and soon bring in a million and where a 43 year old veteran with 20 years in the field in his cause area is virtually ignored because he doesn’t have the youthful EA vibe.
FTX was all EA in the beginning including Aschenbrenner himself who must have still been using a practice razor when he worked there.
The point is EA needs to create a culture that draws in more experienced people who bring their common sense which is informed by real life experience not just classes they had in college. To do this requires thoughtful planning and execution, which should have happened post FTX and maybe this is a reminder. And it’s not just experience, there’s a host of human variety of perspectives that need to all be here so the table benefits from many viewpoints and ways of thinking.
One of the studies on where intelligent people go wrong is that they all follow each other to the same fail point, and they had nobody saying hey, there’s another possible point over here. They lacked perspective. It’s called “Groups of diverse problem solvers can outperform groups of high-ability problem solvers,” which is just common sense to people who love collective intelligence. And love is actions.
3
I'm an 82-year-old collective intelligence pioneer, with my first book chapter, "Quest for Collective Intelligence," published in 1995. Recently, my support offered to young CI "geniuses" was overlooked a couple of times because I didn't have their youthful vibe. I understand that, as every new generation totally legit need to make their own mistake.
1
Hi George, thrilled to meet you. Yes, I wish I was as chill as you and what you say is true and also every generation shall try to save the next from repeating mistakes.
When I was only a boy I figured out that adults were really far smarter than me and sought to learn from them. Later I worked with the elderly for a while and learned it was a mixed bag, some were old and bitter obviously having made bad choices and failed to forgive. But others were clearly life experience geniuses and so you just had to be discerning in who you listened to. So probably because of this and my eventual passion for CI I just find the failure to seek the value of all perspectives as just leaving so much value on the table, to use a popular EA term. I’m going to DM you, can’t bypass a CI pioneer!
Per Bloomberg, the Trump administration is considering restricting the equivalency determination for 501(c)3s as early as Tuesday. The equivalency determination allows for 501(c)3s to regrant money to foreign, non-tax-exempt organisations while maintaining tax-exempt status, so long as an attorney or tax practitioner claims the organisation is equivalent to a local tax-exempt one.
I’m not an expert on this, but it sounds really bad. I guess it remains to be seen if they go through with it.
Regardless, the administration is allegedly also preparing to directly strip environmental and political (i.e. groups he doesn’t like, not necessarily just any policy org) non-profits of their tax exempt status. In the past week, he’s also floated trying to rescind the tax exempt status of Harvard. From what I understand, such an Executive Order is illegal under U.S. law (to whatever extent that matters anymore), unless Trump instructs the State Department to designate them foreign terrorist organisations, at which point all their funds are frozen too.
These are dark times. Stay safe 🖤
The World Happiness Report 2025 is out!
Finland leads the world in happiness for the eighth year in a row, with Finns reporting an average score of 7.736 (out of 10) when asked to evaluate their lives.
Costa Rica (6th) and Mexico (10th) both enter the top 10 for the first time, while continued upward trends for countries such as Lithuania (16th), Slovenia (19th) and Czechia (20th) underline the convergence of happiness levels between Eastern, Central and Western Europe.
The United States (24th) falls to its lowest-ever position, with the United Kingdom (23rd) reporting its lowest average life evaluation since the 2017 report.
I bang this drum a lot, but it does genuinely appear that once a country reaches the upper-middle income bracket, GDP doesn’t seem to matter much more.
Also featuring is a chapter from the Happier Lives Institute, where they compare the cost-effectiveness of improving wellbeing across multiple charities. They find that the top charities (including Pure Earth and Tamaika) might be 100x as cost-effective as others, especially those in high-income countries.
The first thing I wondered about when this report came out was: how did India do?
In that quick take I asked how India's self-reported life satisfaction dropped an astounding -1.20 points (4.97 to 3.78) from 2011 to 2021, even as its GDP per capita rose +51% in the same period; China in contrast gained about as much self-reported life satisfaction as you'd expect given its GDP per capita rise. This "happiness catastrophe" should be alarming to folks who consider happiness and life satisfaction what ultimately matters (like HLI), since given India's population such a drop over time adds up to roughly ~5 billion LS-years lost since 2011, very roughly ballparking (for context, and keeping in mind that LS-years and DALYs aren't the same thing, the entire world's DALY burden is ~2.5 billion DALYs p.a.). Even on a personal level -1.20 points is huge: 10x(!) larger than the effect of doubling income at +0.12 LS points (Clarke et al 2018 p199, via HLI's report), and comparable to major negative life events like widowhood and extended unemployment. So it mystified me that nobody seems to be talking about it.
Last year's WHR reported a 4.05 rating averaged over th...
7
Here is their plot over time, from the Chapter 2 Appendix. I think these are the raw per-year scores, not the averages.
I find this really baffling. It’s probably not political; the Modi government took power in 2014 and only lost absolute majority in late 2024. The effects of COVID seem to be varied; India did relatively well in 2020 but got obliterated by the Delta variant in 2021. Equally, GDP per capita steadily increased over this time, barring a dip in 2020. Population has steadily increased, and growth has steadily decreased.
India have long had a larger residual value than others in the WHR’s happiness model; they’re much less happy than their model might predict.
Without access to the raw data, it’s hard to say if Gallup’s methodology has changed over this time; India is a huge and varied country, and it’s hard to tell if Gallup maintained a similar sample over time.
2
Thanks for digging up that plot, I'd been looking for annual data instead of 3-year rolling averages.
Here's what WHR say about their methodology which seems relevant.
That Gallup website doesn't say if they've changed their methodology over time; that said, they seem to try their best to maintain a similar sample over time, e.g.
I remain as baffled as you are.
4
Clueless, although there are bound to be outliers and exceptions even if we don't understand why.
The Global Fund and Gilead have announced that Lenacapavir, the new 6-month PrEP treatment for HIV, will be made available in 120 countries at no profit. Gilead originally agreed to license Lenacapavir, royalty-free, to local manufacturers, but have now also agreed to directly supply doses until those manufacturers reach capacity, likely for up to 2 million at-risk people. The Global Fund will direct resourcing and deliver doses.
Extremely good news, and a possible silver lining after potentially losing PEPFAR.
Thanks this is indeed amazing news and I'm actually a bit surprised at the commitment, would love to hear the full story and how much it will actually cost, it could still be quite expensive. Super cool as well that eventually it will actually be manufactured in many of the countries that will use the drugs
Although good news, I don't think its the best ever news on HIV treatments. I would rate both the invention of the first antiretroviral (AZT) and PEPFAR probably 10x-100x more important than this news. Not to diminish this at all, as it will definitely reduce HIV infections and likely reduce HIV treatment cost in future, I don't think its going to lead to huge population level reductions in HIV burden like AZT and PEPFAR did.
Why do you think this might be the best ever news?
2
Obviously Global Fund are gonna emphasise the importance of this, but this quote stood out:
I don’t really have any knowledge on HIV, but my assumption is that if it’s cheap to deliver high volumes of 6-month protection, you can essentially end the spread of the virus, and if it can be widely funded and generically manufactured, we might be able to permanently end it within a generation. (Would be happy to stand corrected!)
Yeah his statement is incorrect (unless maybe quoted out of context). ARVs have already fundamentally changed the trajectory of the HIV epidemic in incredible ways - even if this drug did as well, it would not be a first.
In terms of whether this can "change the trajectory of the HIV pandemic", it depends on how we interpret that. I would say its also a misleading statement. Spread of HIV has already been plummeting over the last 30 years due to ARVs - at best Lenacapavir could continue the current trajectory (see graphs below) which I think it has great potential to do.
There's no way a non-cure non-vaccine drug can "end the epidemic in a generation." The nature of HIV is that if its treated well, people stay alive with fairly normal life expectancies. This means even if there's very little spread, prevalence doesn't change much and it is VERY difficult to end the epidemic within a short time. Its a little paradoxical that when HIV is well tracked and controlled, prevalence drops very slowly.
Most HIV is spread through unprotected sex between regular people in the community. Obviously we're not going to give the whole population the injection, only high risk groups so many will still...
A new study in The Lancet estimates that high USAID spending saved over 91 million lives in the past 21 years, and that the cuts will kill 14 million by 2030. They estimate high USAID spending reduced all-cause mortality by 15%, and by 32% in under 5s.
My initial hot-take off the cuff reaction is that it seems borderline implausible that USAID spending have reduced under 5 mortality by 1/3. With so many other factors like Development/Growth, government programs, Medical innovation not funded by USAID (artesunate came on the scene after 2001!), 10x-100x more effective AID like Gates/AMF etc how could this be?
The biggest under 5 effects caused by USAID might be from malaria/ORS programs, but they usually didn't fund the staff giving the medication, so how much credit are they taking for those? They've claimed credit for a 51% drop in malaria mortality?
Their basic method seems to be "We calculated the associations between different levels of USAID funding per capita and decreases in mortality by group of causes (figure 1)." which seems questionable at best.
Obviously they are not considering counterfactuals here, but even not considering those it still seems like huge calls.
I'll have a closer look later, might well be way off the mark here - the thing did get published in the Lancet after all and I'll certainly never get anything published in there...
The U.S. State Department will reportedly use AI tools to trawl social media accounts, in order to detect pro-Hamas sentiment to be used as grounds for visa revocations (per Axios).
Regardless of your views on the matter, regardless of whether you trust the same government that at best had a 40% hit rate on ‘woke science’ to do this: They are clearly charging ahead on this stuff. The kind of thoughtful consideration of the risks that we’d like is clearly not happening here. So why would we expect it to happen when it comes to existential risks, or a capability race with a foreign power?
4
This is... not what the attached source says? Scott estimates 40% woke, 20% borderline, and 40% non-woke. 'at best' means an upper bound, which would be 60% in this case if you accept this methodology.
But even beyond that, I think Scott's grading is very harsh. He says most grants that he considered to be false positives contained stuff like this (in the context of a grant about Energy Harvesting Systems):
But... this is clearly bad! The grant is basically saying it's mainly for engineering research, but they're also going to siphon off some of the money to do some sex-discriminatory ideological propaganda in kindergartens. The is absolutely woke,[1] and it totally makes sense why the administration would want to stop this grant. If the scientists want to just do the actual legitimate scientific research, which seems like most of the application, they should resubmit with just that and take out the last bit.
Some people defend the scientists here by saying that this sort of language was strongly encouraged by previous administrations, which is true and relevant to the degree of culpability you assign to the scientists, but not to whether or not you think the grants have been correctly flagged.
His borderline categorisation seems similarly harsh. In my view, this is a clear example of woke racism:
Scott says these sorts of cases makes up 90% of his false positives. So I think we should adjust his numbers to produce a better estimate:
* 40% woke according to scott
* +20% borderline woke
* +90%*40% incorrectly labeled as false positives
= 96% hit rate.
1. ^
If you doubt this, imagine how a typical leftist grant reviewer would evaluate a grant that said some of the money was going to support computer science for white men.
It's not clearly bad. It's badness depends on what the training is like, and what your views are around a complicated background set of topics involving gender and feminism, none of which have clear and obvious answers. It is clearly woke in a descriptive non-pejorative sense, but that's not the same thing as clearly bad.
EDIT: For example, here is one very obvious way of justifying some sort of "get girls into science" spending that is totally compatible with centre-right meritocratic classical liberalism and isn't in any sense obviously discriminatory against boys. Suppose girls who are in fact capable of growing up to do science and engineering just systematically underestimate their capacity to do those things. Then "propaganda" aimed at increasing the confidence of those girls specifically is a totally sane and reasonable response. It might not in fact be the correct response: maybe there is no way to change things, maybe the money is better spent elsewhere etc. But it's not mad and its not discriminatory in any obvious sense, unless anything targeted only at any demographic subgroup is automatically discriminatory, which at best only a defensible position not an obvious one. I don't know if smart girls are in fact underconfident in this way, but it wouldn't particularly susprirse me.
It's not clearly bad. It's badness depends on what the training is like, and what your views are around a complicated background set of topics involving gender and feminism, none of which have clear and obvious answers.
The topic here is whether the administration is good at using AI to identify things it dislikes. Whether or not you personally approve of using scientific grants to fund ideological propaganda is, as the OP notes, besides the point. Their use of AI thus far is, according to Scott's data, a success by their lights, and I don't see any much evidence to support huw's claim that their are being 'unthoughtful' or overconfident. They may disagree with huw on goals, but given those goals, they seem to be doing a reasonable job of promoting them.
0
I agree with the very narrow point that flagging grants that mention some minor woke spending while mostly being about something elde is not a sign of the AI generating false positives when asked to search for wokeness. Indeed, I already said in my first comment that the flagged material was indeed "woke" in some sense.
5
As a scientist who writes NSF grants, I think the stuff that you're labeling woke here makes up a very small percentage of the total money that actually gets spent in grants like these. Labeling that grant as woke because it puts like 2% of its total funds towards a K-12 outreach program seems like a mistake to me. (And in an armchair-philosophy way, yes the scientists could in principle just resubmit the grants without the last part -- but in practice nothing works like this. Much more likely is that labeling these as "woke" leads people, like the current administration, to try to drastically reduce the overall funding that goes to NSF, with a strong net negative effect on basic science research.)
3
"Labeling that grant as woke because it puts like 2% of its total funds towards a K-12 outreach program seems like a mistake to me."
It's a mistake if it's meant to show the grant is bad, and I suspect that Larks has political views that I would very strongly disagree with, but I think it does successfully make the narrow point that the data about NSF grants does not show that an AI designed to identify pro-woke or pro-Hamas language will be bad at doing so.
0
FWIW the point that I was trying to make (however badly) was that the government clearly behaved in a way that had little regard for accuracy, and I don’t see incentives for them to behave any differently here
2
Yeah, I agree with the more general point.
-2
It seems pretty appropriate and analogous to me - the administration wants to ensure 100% of science grants go to science, not 98%, and similarly they want to ensure that 0% of foreign students support Hamas, not 2%. Scott's data suggests that have done a reasonably good job with the former at identifying 2%-woke grants, and likewise if they identify someone who spends 2% of their time supporting Hamas they would consider this a win.
4
I don't think the issue here is actually about whether all science grants should go only to actual scientific work. Suppose that a small amount of the grant had been spent on getting children interested in science in a completely non-woke way that had nothing to do with race or gender. I highly doubt that either the administration or you regard that as automatically and obviously horrendously inappropriate. The objection is to stuff targeted at women and minorities in particular, not to a non-zero amount of science spending being used to get kids interested in science. Describing it as just being about spending science grants only on science is just a disingenuous way of making the admin's position sound more commonsense and apolitical than it actually is.
An idea that's been percolating in my head recently, probably thanks to the EA Community Choice, is more experiments in democratic altruism. One of the stronger leftist critiques of charity revolves around the massive concentration of power in a handful of donors. In particular, we leave it up to donors to determine if they're actually doing good with their money, but people are horribly bad at self-perception and very few people would be good at admitting that their past donations were harmful (or merely morally suboptimal).
It seems clear to me that Dustin & Cari are particularly worried about this, and Open Philanthropy was designed as an institution to protect them from themselves. However, (1) Dustin & Cari still have a lot of control over which cause areas to pick, and sort of informally defer to community consensus on this (please correct me if I have the wrong read on that) and (2) although it was intended to, I doubt it can scale beyond Dustin & Cari in practice. If Open Phil was funding harmful projects, it's only relying on the diversity of its internal opinions to diffuse that; and those opinions are subject to a self-selection effect in applying for OP, and ...
6
I would have a lot less concern about more central control of funding within EA if there was more genuine interest within those funding circles for broad exploration and development of evidence from new ideas within the community. Currently, I think there are a handful of (very good) notions about areas that are the most promising (anthropogenic short-term existential or major risks like AI, nuclear weapons, pandemics/bioweapons, animal welfare, global health, and development) that guide the 'spotlight' under which major funders are looking. This spotlight is not just about these important areas—it’s also shaped by strong intuitions and priors about the value of prestige and the manner in which ideas are presented. While these methodologies have merit, they can create an environment where the kinds of thinking and approaches that align with these expectations are more likely to receive funding. This incentivizes pattern-matching to established norms rather than encouraging genuinely new ideas.
The idea of experimenting with a more democratic distribution of funding, as you suggest, raises an interesting question: would this approach help incentivize and enable more exploration within EA? On one hand, by decentralizing decision-making and involving the broader community in cause area selection, such a model could potentially diversify the types of projects that receive funding. This could help break the current pattern-matching incentives, allowing for a wider array of ideas to be explored and tested, particularly those that might not align with the established priorities of major funders.
However, there are significant challenges to consider. New and unconventional ideas often require deeper analysis and nuanced understanding, which may not be easily accessible to participants in a direct democratic process. The reality is that many people, even within the EA community, might not have the time or expertise to thoroughly evaluate novel ideas. As a result, they may
4
I like this also, I can imagine a system with a few mainstream assessors (or coalitions thereof) representing the major causes in the movement, and then small-scale ones who get access to better funding to develop smaller causes.
Although, I think both suffer from that same domination of the majority. Ultimately, if there aren’t people in the community willing to vote for the newer causes, they won’t get funded under either system.
4
IIRC, defining "the community" has been an issue in prior discussions. What Manifund did for CC seems reasonable given the sums involved and that it isn't clearly a recurrent thing. If you get into voters controlling thousands of Other People's Money on a recurring basis, you're likely to get people trying to meet the requirements for a vote just to get money flowing to their favorite causes (whether effective or not). If you tenure existing community members and make a high standard for newbies to get voting rights, you're creating a two-tier community at risk of stagnation.
Those problems may not be insurmountable, but they are at least moderately formidable in my view. Still, I like to see experiments!
2
What’s the problem with this?
I am not sure there is such thing as an ‘ineffective cause’. After all, EA is a question. If a naïve voter wanted to send their money to a narrow cause in a rich country (ex. US cancer research), then either they actually want something broader (lives saved, suffering reduced) and it’s up to us to campaign & educate them as to their ‘real’ preferences, or they truly desire something that narrow. In the latter case, I am not sure it is up to you and I to tell them that they’re wrong, even if we vehemently disagree.
But I can see how that problem would practically restrict who donates to the pool. Some constraints that might work without creating a two-tiered system:
* Minimum required donation to vote (maybe even requiring a % pledge with some track record)
* Selecting from a pre-defined list of causes (including broader causes such as ‘reduce suffering’ or ‘reduce human suffering’)
* Allowing donors to restrict their donation (ex. “I want to reduce suffering, but I am neutral within that”)
4
For the EA community: People who are already dead-set on US cancer research (or whatever) go through the motions of EAness to get their vote, diluting the truthseeking nature of the community. And expending mentorships, EAG slots, etc. on people who are going through the motions to get a vote is not a good use of limited resources.
For the donors: The problem is that the donors will always have the option of the existing system -- e.g., deferring to OpenPhil, or to SFF, or EA Funds, picking recipients themselves, etc. To get anywhere, you've got to persuade the donors that your method will produce better results than the status quo. I think there's a strong argument that the collective wisdom of the EA community could theoretically do that, and a less clear argument that it could practically do that. But the argument that a community of 50% current EAs and 50% join-for-predetermined-pet-causes-folks could do so seems a lot weaker to me!
Maybe it works better under particularly strong versions of "some form of EA were to be practiced on a national scale"? But you'd have to convert at least large swaths of donors to the democratic system first. Otherwise, EA-aligned donors see their monies siphoned off for opera etc. and get nothing in return for their preferred causes from non-participating donors. Across the history of philanthropy, I suspect the percentage of donors who would be happy to hand over control to the demos is . . . low. So I'm struggling to see a viable theory of change between "bootstrapping" and national adoption.
2
For the donors, I don’t have a lot to add, but I’ll re-state my point.
Donors today at least claim to be worried about having outsized power in a way that could cause them to get their causes or allocations ‘wrong’. As you note, the wisdom of the crowd could solve this. If these donors don’t want things going off course, they could pre-specify a list of subcauses they’re interested in funding, or even a broader area, or this should keep those benefits while mitigating most of the harms.
2
Again, I know my take on this is a bit unorthodox, but it’s important to remember that we’re not bridging is-ought. If someone truly believes that funding the opera is the highest moral good, and then they determine the most effective way to do so, they are practicing truthseeking. But if they truly believed that preventing suffering was the highest moral good and still voted to fund the opera, then they would (probably) not be truthseeking. I think this distinction is important—both ‘failure modes’ produce the same outcome by very different means!
Whether it is bad to expend those resources depends on how you define EA. If EA is truly cause-neutral, then there’s no problem. But I think you and I agree that Actually Existing EA does generally prefer ‘reduce suffering’ as a cause (and is cause-neutral within that), and in this sense it would be a shame if resources were spent on other things. Hence, a bootstrapped version of democratic altruism would probably restrict itself to this and let people choose within it.
2
That's valid -- I do not have a brain scanner so cannot reliably distinguish someone who merely went through the motions to vote for opera vs. someone who seriously considered the world's most pressing problems and honestly decided that opera was at the top of the list.
In theory, EA principles are cause-neutral, so one could apply them to the belief that the greatest good is introducing people to the Flying Spaghetti Monster and being touched by his noodle-y appendage. And I guess I want opera people to do opera as effectively as possible (my hesitation is based on lack of clarity on what that even means in the opera context)? I'm just not interested in funding either of those, nor am I interested in providing strong incentives for people with those goals to join the existing EA community in significant numbers. They are welcome to start the Effective Opera or Effective FSM movements, though!
2
Yeah this could be a huge problem. Very hard to gatekeep too, the options I can think of (forum activity etc.) seem awkward at best.
It seems like some of the biggest proponents of SB 1047 are Hollywood actors & writers (ex. Mark Ruffalo)—you might remember them from last year’s strike.
I think that the AI Safety movement has a big opportunity to partner with organised labour the way the animal welfare side of EA partnered with vegans. These are massive organisations with a lot of weight and mainstream power if we can find ways to work with them; it’s a big shortcut to building serious groundswell rather than going it alone.
See also Yanni’s work with voice actors in Australia—more of this!
6[anonymous]
We've invited the Voice Actors association to our next advocacy event :)
ChatGPT’s usage terms now forbid it from giving legal and medical advice:
So you cannot use our services for: provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional (https://openai.com/en-GB/policies/usage-policies/)
Some users are reporting that ChatGPT refuses to give certain kinds of medical advice. I can’t figure out if this also applies to API usage.
It sounds like the regulatory threats and negative press may be working, and it’ll be interesting to see if othe...
2
In general, I tend to disregard anything any tech company adds to their terms of service. People often use lines added to the terms of service to read the tea leaves about a company's grand strategy, but isn't it more likely these changes to the TOS get made by some low-level employee in the legal department without the knowledge of the C-suite or other top executives?
And, indeed, The Verge seems to agree with me here (emphasis added):
My first thought when reading the line you quoted in the TOS is that was already common sense, and OpenAI's TOS probably already forbid that, or, if it didn't, it was a slip-up and they were trying to clarify a position they had already long held. The Verge article seems to confirm that.
Also, I generally don't buy the story that specialized providers of LLMs can or will provide LLMs that do better in dispensing advice in specialized fields. LLMs don't specialize. As for the software layer on top the LLM, it's hard to think anyone will do significantly better than OpenAI, Google, or Anthropic. Even if they could do a bit better in some specialized area like law or medicine, would it be enough to really make a difference? Finally, if we're talking about anything that involves a significant ratio of manual human effort to LLM queries, that would seem to blow up the whole unit economics of LLMs — which are already challenging enough. (I think it’s very likely we’ll see the popping of an AI financial bubble within 3 years or so, since current valuations are based on expectations of huge growth, and the factors underlying growth, like data and compute scaling, seem to be running out of steam.)
2
In my experience, you get better advice anyway if you frame the question as though you are a professional. So instead of, "here is a picture of my rash, what do you think?", you say, "A patient has provided this picture of a rash, what is your diagnosis?".
2
If OpenAI are sincere in adding this to their ToS or there’s further regulatory pressure, the models will presumably get better at preventing this. I think it’s important.
Microsoft continue to pull back on their data centre plans, in a trend that’s been going on for the past few months, since before the tariff crash (Archive).
Frankly, the economics of this seem complex (the article mentions it’s cheaper to build data centres slowly, if you can), so I’m not super sure how to interpret this, beyond that this probably rules out the most aggressive timelines. I’m thinking about it like this:
- Sam Altman and other AI leaders are talking about AGI 2027, at which point every dollar spent on compute yields more than a dollar of revenue, with essentially no limits
- Their models are requiring exponentially more compute for training (ex. Grok 3, GPT-5) and inference (ex. o3), but producing… idk, models that don’t seem to be exponentially better?
- Regardless of the breakdown in relationship between Microsoft and OpenAI, OpenAI can’t lie about their short- and medium-term compute projections, because Microsoft have to fulfil that demand
- Even in the long term, Microsoft are on Stargate, so still have to be privy to OpenAI’s projections even if they’re not exclusively fulfilling them
- Until a few days ago, Microsoft’s investors were spectacularly rewarding them for g
Ex-DeepMind scientist David Silver has just raised a $5 billion valuation for his new startup, and pledged to donate 100% of the proceeds from his equity stake via Founders Pledge.
Are we prepared for the AI money to start hitting?
I think preparing for AI money is generally smart given Anthropic & OpenAI Foundation, though I don't expect Ineffable specifically to have liquidity for at least a couple years.
It's possible that there are some clever schemes that could allow David or others to start donating sooner (eg some liquidity at a raise, or borrowing against value of stock), but historically it's not until IPO (and sometimes much later) before founders donate significant amounts.
-5[comment deleted]
The IHME have published a new global indicator for depression treatment gaps—the ‘minimally adequate treatment rate’.
00317-1/asset/ef0b5d6a-8e80-46df-813f-1696fe5b9204/main.assets/gr3_lrg.jpg)
It’s defined using country-level treatment gap data, and then extrapolated to missing countries using Bayesian meta-regression (combined with other GBD data; there’s already a critique paper on this methodology FWIW).
2
It seems like this should be normalized by total population. If a country only had one depressed dude, and he was untreated, I would say this is a small gap, but the map as it is would suggest it was the largest possible gap. Conversely, if every single person in the country was depressed, and only 33% were treated, this map would suggest the gap was very small.
1
It's in Table 3. I do agree this should be visualised as well.
2 weeks out from the new GiveWell/GiveDirectly analysis, I was wondering how GHD charities are evaluating the impact of these results.
For Kaya Guides, this has got us thinking much more explicitly about what we’re comparing to. GiveWell and GiveDirectly have a lot more resources, so they can do things like go out to communities and measure second order and spillover effects.
On the one hand, this has got us thinking about other impacts we can incorporate into our analyses. Like GiveDirectly, we probably also have community spillover effects, we probably als...
Don't know if this is useful, but years ago HLI tried to estimate spillover effects from therapy in Happiness for the whole household: accounting for household spillovers when comparing the cost-effectiveness of psychotherapy to cash transfers, and already found that spillover effects were likely significantly higher for cash transfers compared to therapy.
In 2023 in Talking through depression: The cost-effectiveness of psychotherapy in LMICs, revised and expanded they estimated that the difference is even greater in favour of cash transfers. (after feedback like Why I don’t agree with HLI’s estimate of household spillovers from therapy and Assessment of Happier Lives Institute’s Cost-Effectiveness Analysis of StrongMinds)
I wouldn't update too strongly on this single comparison, and I don't know if there are better analyses of spillover effects for different kinds of interventions, but it seems that there are reasons to believe that spillover effects from cash transfers are relatively greater than for other interventions.
7
I agree to your general analysis. Just some quick thoughts (if you have other ideas, I'd be excited to hear them!)
For us at ACTRA, communicating our benefits in terms of x times cash has been very helpful, because we do not have the typical "one outcome measure makes up for 80% of the effect" situation: Crime reduction leads to reduced interpersonal violence like homicides (DALYs), averts economic damage ($) and has wellbeing benefits (WELLBYs). Our current analysis suggests, that each of those three is about 1/3 of our total effect. So having something, where we cann sum these benefits up and then compare to other charities that might focus on only one of these is very helpful for us.
If we want to keep using cash estimates I see us either
1) incorporating best guesses for our spillover effects etc. in our CEA based on the scientific literature available, discounting them somewhat but not too heavily, to be comparable to the Give Directly estimate
2) Consciously deciding not to include spillover effects etc. and then comparing "apples to apples". So if there was a simple table saying "this is the direct Give Directly effect, this is the additional spillover effect and this is X, Y, and Z", then we can basically produce the same table for our charity with some fields saying n/a and that would make it comparable somehow...
OpenAI appoints Retired U.S. Army General Paul M. Nakasone to Board of Directors
I don't know anything about Nakasone in particular, but it should be of interest (and concern)—especially after Situational Awareness—that OpenAI is moving itself closer to the U.S. military-industrial complex. The article itself specifically mentions Nakasone's cybersecurity experience as a benefit of having him on the board, and that he will be placed on OpenAI's board's Safety and Security Committee. None of this seems good for avoiding an arms race.
2
OpenAI taking security more seriously seems good, and also I expect is good for reducing race dynamics (the less that US adversaries are able to hack US labs, the less tight I expect a race to be).
I liked Bob Jacob’s essay Is Effective Altruism neocolonial?.
Aid dependency is a really interesting problem, where charities can become victims of their own success. I think we should be very thoughtful about counterfactual government funding—even when, due to natural government inefficiencies, it might be less cost-effective.
One place I think EAs can do a lot of good is in charity entrepreneurship. There are often good emerging ideas that need a strong evidence base before governments will adopt them, but a shortage of ambitious people willing to take the...
The Trump administration has indefinitely paused NIH grant review meetings, effectively halting US-government-funded biomedical research.
There are good criticisms of the NIH, but we are kidding ourselves if we believe that this is to do with anything but vindictiveness over COVID-19, or at best, a loss of public trust in health institutions from a minority of the US public. But this action will not rectify that. Instead of one public health institution with valid flaws that a minority of the public distrust, we have none now. Clinical trials have been paus...
OpenAI have their first military partner in Anduril. Make no mistake—although these are defensive applications today, this is a clear softening, as their previous ToS banned all military applications. Ominous.
Microsoft have backed out of their OpenAI board observer seat, and Apple will refuse a rumoured seat, both in response to antitrust threats from US regulators, per Reuters.
I don’t know how to parse this—I think it’s likely that the US regulators don’t care much about safety in this decision, and nor do I think it meaningfully changes Microsoft’s power over the firm. Apple’s rumoured seat was interesting, but unlikely to have any bearing either.
Lina Khan (head of the FTC) said she had P(doom)=15%, though I haven't seen much evidence it has guided her actions, and she suggested this made her an optimist, suggesting maybe she hadn't really thought about it.
Greg Brockman is taking extended leave, and co-founder John Schulman has left OpenAI for Anthropic, per The Information.
For whatever good the board coup did, it’s interesting to observe that it largely concentrated Sam Altman’s power within the company, as almost anyone else who could challenge or even share it is gone.
From SE Geyges: Is METR a meaningful check on Anthropic?.
Part of the reason we are in this position is because nobody outside the EA community cares about AI safety enough to fund it or work in it. I’m very sympathetic to this; some of these close connections are inevitable.
However, it can also be true that these connections are
completelyunacceptable (edit), and that Anthropic should be trying harder than they are to find or create auditors that are genuinely independent. How do we do that?I think METRs checks are meaningful and maybe the best we have at the moment. They are also like you say compromised and the conflict of interests are immense with huge personal overlap between the labs and safety orgs, and funding streams too.
Geyges seems largely correct, but if we can't convince governments to regulate properly its better METR is in there doing it. After all METR exposed more about the hugging face hack than Open AI did on its own.
I don't think a framing of "these connections are completely unacceptable" is helpful given these problems. ... (read more)