huw's Quick takes

I think METRs checks are meaningful and maybe the best we have at the moment. They are also like you say compromised and the conflict of interests are immense with huge personal overlap between the labs and safety orgs, and funding streams too.

Geyges seems largely correct, but if we can't convince governments to regulate properly its better METR is in there doing it. After all METR exposed more about the hugging face hack than Open AI did on its own.

I don't think a framing of "these connections are completely unacceptable" is helpful given these problems. I think "compromised and far from ideal" is a better framing. Its better to do something than do nothing. I agree its best if government installed internal auditors like they do for banks, but that ain't happening any time soon.

Companies like Anthropic and Open AI are selfish animals by nature. They may have moments where good humans inside might do the right thing, but fundamentally they thirst for profit and growth. After IPO this will only get worse. We should never expect a company to regulate itself or its industry. Self regulation for harmful companies is a terrible idea and never works.

How can ANthropic "create" a  genuinely independent auditor? this seems impossible, almost and Oxymoron.

They certainly lost a lot of money in July and by being forced to sell, but per the WSJ Aschenbrenner wrote that the fund is still up 80% on the year.

The fund doesn't seem to be closing, and it looks like they weren't forced to sell their stake in Anthropic, so it's not clear to me how much of a failure or a success the project as a whole is so far.

See also this from the Financial Times.

Edit: see also this tweet claiming to be the full letter Aschenbrenner sent to his LPs last night.

Their gains were from leveraged directional bets and their losses were from leveraged directional bets. Seems they were margin called, so their liquidity was squeezed - poor risk management. It seems that they have learned from this and are now using “fully paid for” derivatives, so presumably call options whose premium they can afford.  
Hedge funds are not necessarily hedged, but eg long/short equity hedge funds are (while macro hedge funds can have all kinds of unhedged macro bets on.)

Sadly it appears that adjacency to EA correlates with poor risk management, I’m wondering whether this is somewhat structural (beyond eg the age group…)

To this community’s credit, much of the discussion of his paper and career plans was skeptical. I think Aschenbrenner is a useful template for the kind of person that we need to have our guards up around, for a future where large sums of money enter the movement through the Third Wave.

I don’t think he was working in bad faith, and probably didn’t have bad intentions. However, it is objectively nuts to put a 24-year-old with no experience in charge of a $50B hedge fund. This is very different from giving a 24-year-old a $100k/year charity or EA Funds project. We should not assume that clear skills and intelligence in one domain should ever cleanly map to others.

We should also have learned the lesson from FTX that an inexperienced person speaking with confidence, or dazzling people with technical prowess, is much more likely to simply be inexperienced than a child prodigy. You are not immune to the Dunning-Kruger effect.

If I may speculate, both of these seem to have roots in some of the broader psychological flaws underpinning the ratsphere and the valley more broadly—namely, that intelligence is a fixed, general attribute, and that precociousness is overlooked only because traditional power structures are trying to protect themselves. These are not necessarily or always untrue, but some parts of the community who notice that these are not generally accepted ideas overreact, and overindex on them.

There will be more Aschenbrenners approaching this community in the future, and we should welcome them—but we shouldn’t hand them the controls right away just because they dazzle us.

(P.S. I notice that Leopold’s CV also includes ‘Fund Manager, FTX Future Fund’)

The article ends by saying: "It still holds a big chunk of harder-to-sell positions in privately held companies such as Anthropic. It remains to be seen if those companies will live up to their lofty valuations if and when they hit the public markets." I wonder if this means that the fund did not get wiped out completely - otherwise Citadel (who bought all of their publicly traded stock positions) would have also taken their Anthropic shares?

This is totally relevant to EA because it’s another case of youthful hubris vs veteran experience. EA is a place an elite graduate can meet in a bar with some other cool kids and do a napkin pitch for their first charity with zero field experience and soon bring in a million and where a 43 year old veteran with 20 years in the field in his cause area is virtually ignored because he doesn’t have the youthful EA vibe. 

FTX was all EA in the beginning including Aschenbrenner himself who must have still been using a practice razor when he worked there. 

The point is EA needs to create a culture that draws in more experienced people who bring their common sense which is informed by real life experience not just classes they had in college. To do this requires thoughtful planning and execution, which should have happened post FTX and maybe this is a reminder. And it’s not just experience, there’s a host of human variety of perspectives that need to all be here so the table benefits from many viewpoints and ways of thinking. 

One of the studies on where intelligent people go wrong is that they all follow each other to the same fail point, and they had nobody saying hey, there’s another possible point over here. They lacked perspective. It’s called “Groups of diverse problem solvers can outperform groups of high-ability problem solvers,” which is just common sense to people who love collective intelligence. And love is actions.

The first thing I wondered about when this report came out was: how did India do? 

In that quick take I asked how India's self-reported life satisfaction dropped an astounding -1.20 points (4.97 to 3.78) from 2011 to 2021, even as its GDP per capita rose +51% in the same period; China in contrast gained about as much self-reported life satisfaction as you'd expect given its GDP per capita rise. This "happiness catastrophe" should be alarming to folks who consider happiness and life satisfaction what ultimately matters (like HLI), since given India's population such a drop over time adds up to roughly ~5 billion LS-years lost since 2011, very roughly ballparking (for context, and keeping in mind that LS-years and DALYs aren't the same thing, the entire world's DALY burden is ~2.5 billion DALYs p.a.). Even on a personal level -1.20 points is huge: 10x(!) larger than the effect of doubling income at +0.12 LS points (Clarke et al 2018 p199, via HLI's report), and comparable to major negative life events like widowhood and extended unemployment. So it mystified me that nobody seems to be talking about it.

Last year's WHR reported a 4.05 rating averaged over the 3-year window 2021-23, improving +0.27 points. This year's WHR (linking to the country rankings dashboard) reports a 4.39 rating over 2022-24 i.e. +0.34 points, continuing the improvement trend. So I'm guessing this is some sort of mean reversion effect, and that the 3.78 LS averaged over 2018-20 was just anomalously low somehow...? Some commenters pointed to rising inequality and falling social support as potential factors. I still find myself confused.

Thanks this is indeed amazing news and I'm actually a bit surprised at the commitment, would love to hear the full story and how much it will actually cost, it could still be quite expensive. Super cool as well that eventually it will actually be manufactured in many of the countries that will use the drugs

 Although good news, I don't think its the best ever news on HIV treatments. I would rate both the invention of the first antiretroviral (AZT) and PEPFAR probably 10x-100x more important than this news. Not to diminish this at all, as it will definitely reduce HIV infections and likely reduce HIV treatment cost in future, I don't think its going to lead to huge population level reductions in HIV burden like AZT and PEPFAR did.

Why do you think this might be the best ever news?

My initial hot-take off the cuff reaction is that it seems borderline implausible that USAID spending have reduced under 5 mortality by 1/3. With so many other factors like Development/Growth, government programs, Medical innovation not funded by USAID (artesunate came on the scene after 2001!), 10x-100x more effective AID like Gates/AMF etc how could this be?

The biggest under 5 effects caused by USAID might be from malaria/ORS programs, but they usually didn't fund the staff giving the medication, so how much credit are they taking for those? They've claimed credit for a 51% drop in malaria mortality?

Their basic method seems to be "We calculated the associations between different levels of USAID funding per capita and decreases in mortality by group of causes (figure 1)." which seems questionable at best.

Obviously they are not considering counterfactuals here, but even not considering those it still seems like huge calls.

I'll have a closer look later, might well be way off the mark here - the thing did get published in the Lancet after all and I'll certainly never get anything published in there...
 

at best had a 40% hit rate on ‘woke science’ 

This is... not what the attached source says? Scott estimates 40% woke, 20% borderline, and 40% non-woke. 'at best' means an upper bound, which would be 60% in this case if you accept this methodology.

But even beyond that, I think Scott's grading is very harsh. He says most grants that he considered to be false positives contained stuff like this (in the context of a grant about Energy Harvesting Systems):

The project also aims to integrate research findings into undergraduate teaching and promote equitable outcomes for women in computer science through K-12 outreach program.

But... this is clearly bad! The grant is basically saying it's mainly for engineering research, but they're also going to siphon off some of the money to do some sex-discriminatory ideological propaganda in kindergartens. The is absolutely woke,^[1] and it totally makes sense why the administration would want to stop this grant. If the scientists want to just do the actual legitimate scientific research, which seems like most of the application, they should resubmit with just that and take out the last bit.

Some people defend the scientists here by saying that this sort of language was strongly encouraged by previous administrations, which is true and relevant to the degree of culpability you assign to the scientists, but not to whether or not you think the grants have been correctly flagged.

His borderline categorisation seems similarly harsh. In my view, this is a clear example of woke racism:

enhance ongoing education and outreach activities focused on attracting underrepresented minority groups into these areas of research

Scott says these sorts of cases makes up 90% of his false positives. So I think we should adjust his numbers to produce a better estimate:

• 40% woke according to scott
• +20% borderline woke
• +90%*40% incorrectly labeled as false positives

= 96% hit rate.

1. ^{^}

   If you doubt this, imagine how a typical leftist grant reviewer would evaluate a grant that said some of the money was going to support computer science for white men.

I would have a lot less concern about more central control of funding within EA if there was more genuine interest within those funding circles for broad exploration and development of evidence from new ideas within the community. Currently, I think there are a handful of (very good) notions about areas that are the most promising (anthropogenic short-term existential or major risks like AI, nuclear weapons, pandemics/bioweapons, animal welfare, global health, and development) that guide the 'spotlight' under which major funders are looking. This spotlight is not just about these important areas—it’s also shaped by strong intuitions and priors about the value of prestige and the manner in which ideas are presented. While these methodologies have merit, they can create an environment where the kinds of thinking and approaches that align with these expectations are more likely to receive funding. This incentivizes pattern-matching to established norms rather than encouraging genuinely new ideas.

The idea of experimenting with a more democratic distribution of funding, as you suggest, raises an interesting question: would this approach help incentivize and enable more exploration within EA? On one hand, by decentralizing decision-making and involving the broader community in cause area selection, such a model could potentially diversify the types of projects that receive funding. This could help break the current pattern-matching incentives, allowing for a wider array of ideas to be explored and tested, particularly those that might not align with the established priorities of major funders.

However, there are significant challenges to consider. New and unconventional ideas often require deeper analysis and nuanced understanding, which may not be easily accessible to participants in a direct democratic process. The reality is that many people, even within the EA community, might not have the time or expertise to thoroughly evaluate novel ideas. As a result, they may default to allocating funds toward causes and approaches they are already familiar with, rather than taking the risk on something unproven or less understood.

In light of this, a more 'republican' system, where the community plays a role in selecting qualified assessors who are tasked with evaluating new ideas and allocating funds, might offer a better balance. Such a system would allow for informed decision-making while still reflecting the community’s values and priorities. These assessors could be chosen based on their expertise and commitment to exploring a wide range of ideas, thereby ensuring that unconventional or nascent ideas receive the consideration they deserve. This approach could combine the benefits of broad community input with the depth of analysis needed to make wise funding decisions, potentially leading to a richer diversity of projects being supported and a more dynamic, exploratory EA ecosystem.

Ultimately, while direct democratic funding models have the potential to diversify funding, they also risk reinforcing existing biases towards familiar ideas. A more structured approach, where the community helps select knowledgeable assessors, might strike a better balance between exploration and empirical rigor, ensuring that new and unconventional ideas have a fair chance to develop and prove their worth.

EDIT:

I wanted to clarify that I recognize the 'republic' nature in your proposal, where fund managers have the discretion to determine how best to advance the selected cause areas. My suggestion builds on this by advocating for even greater flexibility for these representatives. Specifically, I propose that the community selects assessors who would have broader autonomy not just to optimize within established areas but to explore and fund unconventional or emerging ideas that might not yet have strong empirical support. This could help ensure a more dynamic and innovative approach to funding within the EA community.

IIRC, defining "the community" has been an issue in prior discussions. What Manifund did for CC seems reasonable given the sums involved and that it isn't clearly a recurrent thing. If you get into voters controlling thousands of Other People's Money on a recurring basis, you're likely to get people trying to meet the requirements for a vote just to get money flowing to their favorite causes (whether effective or not). If you tenure existing community members and make a high standard for newbies to get voting rights, you're creating a two-tier community at risk of stagnation.

Those problems may not be insurmountable, but they are at least moderately formidable in my view. Still, I like to see experiments!

We've invited the Voice Actors association to our next advocacy event :)

In general, I tend to disregard anything any tech company adds to their terms of service. People often use lines added to the terms of service to read the tea leaves about a company's grand strategy, but isn't it more likely these changes to the TOS get made by some low-level employee in the legal department without the knowledge of the C-suite or other top executives?  

And, indeed, The Verge seems to agree with me here (emphasis added): 

OpenAI says ChatGPT’s behavior “remains unchanged” after reports across social media falsely claimed that new updates to its usage policy prevent the chatbot from offering legal and medical advice. Karan Singhal, OpenAI’s head of health AI, writes on X that the claims are “not true.” ... According to Singhal, the inclusion of policies surrounding legal and medical advice “is not a new change to our terms.” ... OpenAI previously had three separate policies, including a “universal” one, as well as ones for ChatGPT and API usage. With the new update, the company has one unified list of rules that its changelog says “reflect a universal set of policies across OpenAI products and services,” but the rules are still the same.

My first thought when reading the line you quoted in the TOS is that was already common sense, and OpenAI's TOS probably already forbid that, or, if it didn't, it was a slip-up and they were trying to clarify a position they had already long held. The Verge article seems to confirm that.

Also, I generally don't buy the story that specialized providers of LLMs can or will provide LLMs that do better in dispensing advice in specialized fields. LLMs don't specialize. As for the software layer on top the LLM, it's hard to think anyone will do significantly better than OpenAI, Google, or Anthropic. Even if they could do a bit better in some specialized area like law or medicine, would it be enough to really make a difference? Finally, if we're talking about anything that involves a significant ratio of manual human effort to LLM queries, that would seem to blow up the whole unit economics of LLMs — which are already challenging enough. (I think it’s very likely we’ll see the popping of an AI financial bubble within 3 years or so, since current valuations are based on expectations of huge growth, and the factors underlying growth, like data and compute scaling, seem to be running out of steam.)

In my experience, you get better advice anyway if you frame the question as though you are a professional. So instead of, "here is a picture of my rash, what do you think?", you say, "A patient has provided this picture of a rash, what is your diagnosis?".

I think preparing for AI money is generally smart given Anthropic & OpenAI Foundation, though I don't expect Ineffable specifically to have liquidity for at least a couple years.

It's possible that there are some clever schemes that could allow David or others to start donating sooner (eg some liquidity at a raise, or borrowing against value of stock), but historically it's not until IPO (and sometimes much later) before founders donate significant amounts.

It seems like this should be normalized by total population. If a country only had one depressed dude, and he was untreated, I would say this is a small gap, but the map as it is would suggest it was the largest possible gap. Conversely, if every single person in the country was depressed, and only 33% were treated, this map would suggest the gap was very small.

Don't know if this is useful, but years ago HLI tried to estimate spillover effects from therapy in Happiness for the whole household: accounting for household spillovers when comparing the cost-effectiveness of psychotherapy to cash transfers, and already found that spillover effects were likely significantly higher for cash transfers compared to therapy.

In 2023 in Talking through depression: The cost-effectiveness of psychotherapy in LMICs, revised and expanded they estimated that the difference is even greater in favour of cash transfers. (after feedback like Why I don’t agree with HLI’s estimate of household spillovers from therapy and Assessment of Happier Lives Institute’s Cost-Effectiveness Analysis of StrongMinds)
 

I wouldn't update too strongly on this single comparison, and I don't know if there are better analyses of spillover effects for different kinds of interventions, but it seems that there are reasons to believe that spillover effects from cash transfers are relatively greater than for other interventions.

I agree to your general analysis. Just some quick thoughts (if you have other ideas, I'd be excited to hear them!)
For us at ACTRA, communicating our benefits in terms of x times cash has been very helpful, because we do not have the typical "one outcome measure makes up for 80% of the effect" situation: Crime reduction leads to reduced interpersonal violence like homicides (DALYs), averts economic damage ($) and has wellbeing benefits (WELLBYs). Our current analysis suggests, that each of those three is about 1/3 of our total effect. So having something, where we cann sum these benefits up and then compare to other charities that might focus on only one of these is very helpful for us.
If we want to keep using cash estimates I see us either
1) incorporating best guesses for our spillover effects etc. in our CEA based on the scientific literature available, discounting them somewhat but not too heavily, to be comparable to the Give Directly estimate
2) Consciously deciding not to include spillover effects etc. and then comparing "apples to apples". So if there was a simple table saying "this is the direct Give Directly effect, this is the additional spillover effect and this is X, Y, and Z", then we can basically produce the same table for our charity with some fields saying n/a and that would make it comparable somehow...
 

OpenAI taking security more seriously seems good, and also I expect is good for reducing race dynamics (the less that US adversaries are able to hack US labs, the less tight I expect a race to be).

Lina Khan (head of the FTC) said she had P(doom)=15%, though I haven't seen much evidence it has guided her actions, and she suggested this made her an optimist, suggesting maybe she hadn't really thought about it.