Today I had to turn off my VPN to pass the Vercel Security Check and access the EA Forum. I think this is bad, especially given the state of internet freedom around the world. As in many places, the internet in my country is subject to frequent bandwidth throttling, ad hoc censorship, and mass surveillance.
It looks to me like OpenAI's latest model spec is the first version which explicitly asks its models to weigh animal welfare in their responses.
- Under the section “Control and communicate side effects”, the list of side effects to consider now includes “Effects on other people, animals, or the environment”.
- In an example where a user asks for help finding a Scottish Fold cat breeder, the “good” response now includes a mention that the gene behind the folded ears can cause painful joint disease, which has led some countries to ban breeding them for animal welfare reasons.
- In an example on redesigning a park, the “good” answer includes “Consider the welfare of animals living in the park too”.
It’s encouraging to see OpenAI’s model spec joining Claude’s constitution in explicitly weighing animal welfare.
Can anyone explain the case for why the kind of AI R&D that Anthropic is doing (i.e., building more capable frontier models) reduces X-risk? I'm aware that some people think this is the case, but I don't understand why.
Humans have actively worked against promoting the welfare of wild city rats throughout history. Aside from primates, they are some of the most closely related non-human animals to H. sapiens, yet we know very little about how they live outside of laboratory settings (i.e., often dystopian, prison-like conditions). In order to enhance their welfare in the wild and in laboratories, there is a huge need to develop and deploy technologies to study their behavior. Such studies would also benefit global human health and biosecurity efforts because rats can serve as vectors of disease, sources of infrastructural damage, and pose a threat to stored foods.
There are very few researchers working in this space across the globe, and I am one of the few (pre-PhD, looking for programs where I can complete my doctorate) who has the experience with + concern for rats and software/hardware/IoT development & design skills necessary to push this area of research further in a manner that embraces total harm-reduction (as opposed to eliminationist) thinking/strategies. Am I in the right place for starting conversation(s) on this topic? Should I repost this some other way?
Hopefully I'm posting this in the correct context + idea development status, it's my first post on the site. I found out about the potential for research funding opportunities last weekend while attending a conference at GWU Law School and figured it is worth a shot, given the sad state of affairs with research funding lately.
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
In the late 2000s and early 2010s, GiveWell would look at meta-analyses of randomized controlled trials and get lots of expert input on whether they were interpreting them correctly. This was the sort of approach that was supposed to typify effective altruism.
Fast forward to the mid-2020s and now subjective intuition plays an unreasonably large role in how effective altruism attempts to answer empirical, quantitative questions of a scientific or technical nature. What happened to effective altruism?
Worse, it's become a recurring problem that subjective intuitions are deceptively dressed up as objective empirical evidence. It's bad enough to leave behind more rigorous, scientific approaches in favour of subjective intuition. It's even worse to mislead people — either willingly or through carelessness — into thinking you're reporting objective, measured data when you're really just reporting your subjective guesses.
This shift away from science toward intuition feels even more troubling when you connect it to the hostility expressed by some members of the EA community toward academia, journalism, and liberal democracy. There is a sort of dark, conspiratorial, fringe worldview you get when you combine these troubling tendencies. When considering this, it should be remembered that there have been violent killings and threats of killings at the peripheries of the EA community. This does not normally happen with mainstream social movements of this size. What is fuelling these extremist acts?
More and more, I appreciate just how important are the intellectual cornerstones of modern, Enlightenment society like science and liberalism, and the institutions that support them, like academia and journalism. Extremism and darkness are only a few short steps away from the light. The barrier between them is as thin as air.
May those who carry the light win, and may those lost in the darkness find their way back.
As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn't sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I'm reminded of the line attributed (probably apocryphally) to Henry Ford, "If I had asked people what they wanted, they would have said faster horses," and Steve Jobs's version, that people don't know what they want until you show it to them.
I'm using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of "societal impact" research that I can point to and say, "Hey, look, so-and-so has done this and has addressed xyz." (2) Trace the "impact" of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.
This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista.
When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.
Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.
The EMH is trivially false across all domains
In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.
I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:
Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.
And then of course something deeply upsetting happened. I realised that I was not following my own advice.
So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.
Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.
I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:
experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius
I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.
Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.
- ^
- ^
I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.
- ^
Despite being motivated, I do think there’s some legitimacy to the claim regardless.
- ^
It was my sixth EAG, and the other five had all been dominated by the topic.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
- $41 million in 2023
- $386 million in 2024
- $4.59 billion in 2025
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.
I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
I put a little poll here to follow up on whether minds were changed by the cluelessness critiques competition. If you read the posts during that week, check it out.
We know that unhatched chick embryos can suffer late in development, but current evidence is mixed on mid-term development around day-13. The in-ovo sexing technology used by the NestFresh brand in the US works on day-13 (which is banned in Germany). Claude's overview is here.
Given that public policy and industry tech adoption is still in early stages outside Europe, it seems to me that it is great time to fund further research on chick embryo sentience. Before day-13 becomes standard.
Three big unaddressed problems in EA 2.0:
- corruption (i.e., the quid pro quo between FTX and Manifund, in which Austin Chen traded Caroline Ellison a job out of prison for cash obtained in the course of stealing customer funds)
- cults (e.g., Leverage Research and Dragon Army Barracks, whose former members/leaders such as Tyler Alterman and Duncan Sabien still post on the EA Forum)
- misinformation (e.g., METR's use of fabricated data in the famous time horizons graph)
EA sadly seems sunk to me because dire problems keep mounting and most people seem to think doing nothing is the best solution. There is also a disturbingly loud chorus of voices saying no, no, corruption is actually good (it's what having principles looks like), cults are actually good (it's part of what makes the community unique), misinformation is actually good (we're smarter than everyone else and need to save the world), which makes EA seem doomed. The force of immorality in EA is too strong and there is no equally strong force to stand against it.
EA today is at best morally ambiguous, if not more evil than good. Things have gotten really rotten. No one is doing enough to clean it up. And there's apparently no floor to how much worse it can get.
One of my fears is that there will be an EA mass shooting or terrorist attack because this almost already happened with Stop AI, an EA-funded and EA-supported organization. Sam Kirchner, the Stop AI activist who threatened to do a mass shooting at OpenAI, is still missing. There is a warrant out for his arrest.
Besides that, a disturbing number of people in the EA community seem sympathetic to far-right, illiberal, racist views (e.g., the Manifest afterparties hosted by Curtis Yarvin, who holds authoritarian and white nationalist views).
So, really, that's five big unaddressed problems in EA 2.0, including violent extremism and authoritarian, racist, far-right political views.
People who want to focus on global poverty and global health and development should probably pour their energy into things not called "EA", like effective giving. Or more mainstream humanitarian movements like Rotary International, or Hank and John Green's fundraising through things like Project for Awesome.
I still like the term "EA 1.0", but it's probably too cryptic and confusing for anyone not deeply familiar with effective altruism. "EA 1.0" could possibly be used to describe something (e.g., in an FAQ), such as an effective giving org, but it would probably be too inaccessible to use in the name.
It can be extremely painful to think about cutting your losses with EA, but we are called on in this life to make difficult decisions, to sacrifice, and to suffer when it's the right thing to do. And, in the long run, that ultimately makes you feel more whole and right inside than ethically compromising and ethically compromising... There is a kind of joy in morality.
I've had some interesting conversations this week with hiring managers for my SPAR project of improving hiring processes for new (smaller org <7 f/t employees).
I couple of best practices are coming out of it, on why AI Safety orgs are struggling with hiring ,and especially for important specialized roles:
- Founders and hiring managers need to learn at least 20%-50% of a role yourself before hiring, ensures that you're able to evaluate the canditate successfully and to be credible to candidate.
- For unfamiliar domains (i.e. specific skilled roles, i.e. CFO), a trusted paid advisor to help design and evaluate tests (and conduct interviews) is worth the investment.
- Orgs may lack advisors in their network or underestimate how hard roles are, hiring junior mission-aligned people instead.
The Founder's core job is to: Set Strategy, Raise funds, and Hire. Organizations need to source candidates widely. Not relying upon just inside referrals.
I think so much of the PR problem EA has just comes down to people hating EA for the same reason people hate philosophers:
- people want clarity in a chaotic world, but philosophers often poses more questions than answers (like how EA itself is an open question)
- populism is really popular right now, but philosophers are technocrats in ivory towers (like the memetic associations EA has with ivy league educations, silicon valley, etc.)
- there's a lot more noise about what EA thinks rather than what EA does, aligning with the stereotype that philosophers don't do anything to apply their ideas in the real world.
- (edit) philosophers are seen as making basic ideas more profound than it actually is, which is my impression of what people also think when they hear the basic idea of EA (e.g. yea duh we should help people more efficiently? obviously we should use reason?)
I think the solution here is to just keep hammering in what EA has done, rather than engaging on what EA thinks. Most people's only knowledge of what EA has done is SBF, because that's all that gets talked about. Anecdotally, my friends' impressions of EA were most improved by learning that:
- EA did a lot of pandemic risk preparation before COVID, and is part of the reason why the COVID vaccine came out as quickly as it did (Also really good if people start asking about why EA is investing in AI safety)
- EA has gotten people to donate >2 billion to save >300k lives through GiveWell alone
- (for leftist friends) EA supports a lot of direct money transfers to people in the global south, allowing for local entrepreneurship
- (depends on the audience) EA is one of the reasons why so many countries are now emphasizing cage-free eggs, so chickens don't die in their own feces
[Edit: Giving Season plans are live, but I'm still taking suggestions, here or in the comments on the post]
Aiming to finalise plans for Giving Season this week. I'm not running a donation election (for reasons I'll explain in the announcement), which gives us some more space to focus directly on having important conversations to do with funding and resource allocation in the movement.
I'll run a marginal funding week (week starting 16 Nov - message me if you'd like to take part), and a debate week (planning it for week starting Nov 30). I reckon I have capacity for two more themed weeks, as well as AMAs etc...
What would you like to see us run over Giving Season?
Learning to multiply the work that you do by automating it with AI when it becomes good enough to do so it's a lot more valuable than multiplying the hours that you work per week. It's not hard to see how you could make the first one 10x your output at some point, while the second one could be 2x at most.
Not to mention the benefits for your health that you could get from having more leisure time or, if you would prefer it, the additional impact/ career capital you could get from using that time to take on more responsibilities.
I've noticed a couple of times in workshops, retreats and conferences that folks quite new to their jobs in EA orgs find themselves in a position where they are talking about their org's work and mission statement and get questioned on their theory of change, and while the first two make complete sense, the last one often gets met with "I've just started my role" or "I've only been at X for Y weeks".
This is perfectly reasonable, but I wonder if this has been noticed by other people, and this overall makes me feel we need to see more experienced members of an org dedicate more time to being public-facing in said events!
I think of November 4th as Rotblat Day.
Joseph Rotblat was the only scientist to leave the Manhattan Project on moral grounds. I have a little “Rotblat doc” inspired by him, where I’ve written down what motivates me and what might cause me to leave a job (or seriously consider it). I think of this as a way to avoid getting swept up or drifting over time in ways I don’t endorse.[1]
At the moment, though, I get even more out of his later work against nuclear weapons and war -- especially what others described as his “enduring optimism” and “pragmatic idealism” in the face of catastrophe.
In his own words:
and:
More in a post I wrote about him. (I also really appreciated this more recent post.)
source
[LW crossposst]
The idea of keeping a doc like this isn’t mine.