My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).
The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.
HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”
While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.
Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.
Furthermore, if I'm reading between the lines correctly, OpenAI did not address the issue (and perhaps didn't even know about their models doing this) until after HuggingFace's public blog post.
This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about.
Me and @Fran are interviewing @GraceAdams🔸 tomorrow evening. Is there anything you'd like us to ask her?
Thinking back to my first big (well, big for me) donation and an unusual series of thoughts I had. Sharing in case anyone has experienced the same.
--
I've practiced frugality to a significant extent, largely because I want to donate/salary sacrifice so much of my income to effective charities. As a result, whenever I'm spending a significant amount of money on anything, alarm bells go off.
I was surprised that alarm bells went off when donating. I had thought through where I donated extensively, and charity was the reason why I wanted to save money in the first place. But I still felt stressed because I was "spending money."
This is such a clear example of missing the forest for the trees. I think we need to be careful about instrumental values/rules and making sure they don't become absolute limitations that decrease overall impact.
We have bought 1500 copies of IABIED in Russian; over the past few days, sent emails to 1700 winners of olympiads who have previously received HPMOR from us offering the new book; and already shipped 173 copies to them (+ sent 43 e-books)
I’m not sure. The claim that cold outreach signals a lack of respect for boundaries is unnecessary moralising that I don’t put stock in. If your point is simply that it doesn’t work, people can find that out for themselves; I don’t think the externalities are so negative that donors need you to act as their mouthpiece on this. You’ve apologised for sounding patronising towards fundraisers; I’d be more concerned about coming across as patronising to donors, who themselves are not a homogeneous group. Some will be more receptive to thoughtful, targeted outreach than others, especially if the underlying project or work is persuasive.
I’m also uneasy with the alternative being proposed, of becoming ‘legibly excellent’ to the small set of advisers a donor already trusts. That may be good advice for organisations already close to those networks. But as a community norm - one frequently criticised as elitist - I read it as ‘don’t approach new donors directly; instead, prove yourselves to the existing gatekeepers and hope we choose to let you through.’
This post criticises people for opportunistically inserting their own projects into conversations about Anthropic wealth - while repeatedly emphasising the author’s own proximity to these donors, highlighting his own fund, and ending by recommending his own advisory. Ironically, I’m left wondering whether part of the concern about cold outreach is that, when it does work, it allows organisations to bypass the existing intermediaries. Middlemen tend not to be enthusiastic about that.
I thought the section on scaling was useful, though.
TL;DR: I'm releasing a website that ranks philanthropists according to EA principles and research, and allows users to re-rank the list using their own assumptions. I'd like feedback and help making it better. I'd especially like ideas for how to make the results more trustworthy. Funding may be available.
Crossposted to LessWrong.
...
I'm on the antipenultimate rank with 404 lives saved! I'll try to beat Edythe Broad when I have a real income again! Thanks, Elliot! Awesome work! :-D
Yeah I think the chance of widespread adoption is quite low (< 1%?), but worth a shot. Without adoption I don't think rich people will care much about the rankings themselves (if ~no one else cares about the rankings, then being higher on the list doesn't help their status). But even without much adoption there's a chance for a billionaire to see the list and realize that they weren't thinking about effectiveness enough or in the right way, and become another Moskovitz-like figure.
I've been surprised that three of the commenters so far have had this concern that someone like Musk being ranked 26th/73 will make people want to dismiss the list as a whole. One thing I mentioned in the post is wanting to eventually include all the effects of a person on the world rather than just donations. I'm not sure if this would help though, or just open up way more surface area for people to want to dismiss the list because some assumption went against a deeply held belief of theirs.
I really don't think things need to be seen as "canonical" to get widespread attention or adoption. I think you're right though that its unlikely to be seen as a permanent/long term ranking that people refer to periodically. I can more see this kind of thing going vival or getting traction from time to timeif done in the right way.
Cool project! I haven't dug into into the methodology, but one interesting (and very rough) implication: if the total set of donations ($393B) had had an average of $10,000/life saved (roughly half as good as Givewell, assuming you can't get $5,000/life-saved due to diminishing marginal returns), then they would have saved ~39M lives. The actual total of lives saved according to this is ~30M.
So (assuming the methodology checks out and this calc is roughly right) getting these top philanthropists to be more cost-effective could have saved nine million lives.
If these 9 million lives were saved were over a 30 year time span (rough guess, some donations are older than this, but I think most are much more recent), that's around 300,000 lives saved per year. Which is about equivalent to curing leukaemia.
Thanks!
I agree that ALLFED should not inherit the default disaster relief numbers. Ideally it would have its own analysis, because even if the numbers do end up being similar to AI x-risk, the reasons why would be pretty different. I also think the estimates shouldn't be influenced by the motivations of donors.
If someone wants to add a custom effectiveness analysis of ALLFED (it looks like CEARCH's numbers are just for part of the portfolio), that'd be great. Otherwise I plan to eventually get to it.
The curated longtermist assumptions use a growth rate of 0.000002% over 10 billion years with a ceiling of a million times the current population, so the population reaches 8.3 quadrillion after 711 million years. Still probably on the low end if we start colonizing space.
Yeah the LLM driven donation research did find both of those sources and both are used as proof for at least some donations. This wouldn't have caused people to be added who weren't already on the list though.
This is a new episode of the GiveWell Conversations podcast. You can listen to the episode or read a summary of the conversation below.
Diarrhea is one of the leading causes of death for children under five, killing more than 400,000 children every year....
This post is co-authored with Ben Garfinkel. It is cross-posted from the CEA blog. A PDF version can be found here.
Summary: Some strategic decisions available to the effective altruism m...
I think what makes this post a bit arm-chair-ish is the type of premise that reads like “the movement as a whole should do xyz”.
It’s fine, but it’s much nicer to read about “you as an individual actor should do abc (possibly to marginally influence the movement in direction xyz)” because that’s actually action-guiding? Part of the reason I like EA is the direct application of theory… by critically thinking individuals that can learn over time and do good regardless of a “movement”.
I didn't notice at all how old the post was before I made this comment 😂
As @RyanCarey pointed out, it is true but also tautological, so I'm not sure how useful it can be.
I would be interested in more posts about the risks new funding might bring. But I don't want people to be so worried about irreversibility that they don't use that funding, when it could do a hell of a lot of good, and there might be good reasons to spend it quickly.
I have been increasingly surprised over the past year at what I perceive to be the lack of critical thinking in response to new announcements regarding AI model capabilities. I’ve written about many of these issues before, but as AI hype accelerates the problems seem to be getting worse. Here I want to highlight a few points that s...
Thank you for jumping in and joining the discussion! I'm particularly excited to read more into the Berkeley Vulnerability Initiative, just to understand the project's larger goals and techniques.
I think, as per other comments, that determining whether something is hyped or not is pretty subjective, but I do feel that my main prediction, that carefully scaffolded harnesses and systems would be reliably pumping out Mythos-class vulnerability discoveries by the end of the year, is pretty clearly already true.
I do think its clear that progress in AI offensive cyber capabilities is a real and present concern—I find myself actively worrying about the digital money in my bank accounts—so, I would consider that a valid takeaway from Mythos and subsequent developments. We are clearly very close to a situation where AI progress could endanger a lot of our digital infrastructure, if it just had the funding and someone with bad motivations to push it. That's a big change from how I perceived the world one year ago, and seems worth being excited about it.
It also, to the technical point of this discussion, seems to be a capability gain that is gated on model intelligence, or at least, that's how it looks to someone like me, with a naïve picture of the current state of model training/development.
Originally published at my Funding Anthropalypse Substack.
Great post - love the coverage on your Substack too!
“How long have you been v*g*n?”
This is one of the most common icebreakers at animal protection events. It’s a baseline assumption, and it mostly holds true: if you’re out advocating for animals not to be tortured or abused, realistically these days you are v**n, or close. And it makes for good conversation. It seems fairly safe to assume when you meet strangers.
But this assumption is hurting the movement in a way which we don’t always notice: someone new comes into the sp...
But I do view these practices as immoral, and I do want the industry to end. I would waffle on whether I see them as "fundamentally evil", but same difference, sure, since I cannot personally guarantee any near-term happy ending for everyone involved. Well, the goal isn't friendliness. Are you saying noone can want any industry to be ended out of ethical concerns? Some causes cannot go with zero friction, and zero conflict, that's just the reality. Like there's simply no way to be much friendlier in this regard, and there can be some instrumental win-wins, but the division does remain. In reality, big animal ag is trying to undo all progress with things like the attempted Save Our Bacon part of the Farm Bill, or constantly propagandizing against the entire animal rights movement- this is because in such a big industry, the primary motivator is systemic, not personalistic. The friendliness level of the movement is secondary. Sure, a little more can be done on the poultry-industry veterinarian example (although even there, not too much, because of the effect of cognitive dissonance), but the main object level issues still chug along. Maybe you're saying this from some welfarist perspective where the animal slaughtering and use/exploitation is viewed as okay, the only issue is chronic pain, so you can truly say that you do not want the industries to come to an end, however broadly construed? I view this as the most important goal right now, but it would be a lie if I said that's where my problems end.
As a person who has described themselves as ”80% vegan”, thank you for this post. I agree we should emphasize/normalize talking about choices and behaviours (e.g. ”I’m trying to not eat meat because of the effects it has on the climate”) over identity and labels that run the risk of alienating the listener instead of inviting them to a conversation. Similar to how someone might say ”I want to exercise more”, rather than ”I’m a healthist”.
This post is the third in a short series about sentience, ceilings of Pain and Pleasure intensity (‘affective ceiling’, or ‘affective capacity’), and how to make interspecific welfare comparisons more explicit without creating false precision.
In the first post, we approached the problem through what was essentially a physics or information-encoding lens, trea...
My exoplanet example was intended only to distinguish graded evidence from the underlying fact being investigated. Whether a candidate celestial body exists is different from whether an already known body satisfies one of several proposed definitions of a planet. Margot’s figure addresses the latter question. My analogy concerned the former.
I was analogising celestial bodies to organisms, and being a planet to being sentient. However, I also think whether a celestial body exists or not is a matter of human convention. I assume a clear definition would involve coming up with some properties like a minimum mass and density. In this case, one could come up with some amount of matter slighly too light to be a celestial body which would become a celestial body after adding it one more atom of hydrogen.
So yes: under my hypothesis, two organisms could in principle be extremely similar while affective experience was present in one and absent in the other. I do not find that paradoxical.
Even if the difference between the organisms can be arbitrarily small (not just extremely small, but arbitrarily close to a difference of exactly 0)? Supposing sentience is categorical/sharp, 2 organisms could be equal in all regards except that one differs 10^-100 % along a single metric, and therefore is the only one of the 2 which is sentient.
A small difference in a complex system can sometimes make a new causal process possible. The supporting architecture may vary continuously even though the process we are asking about either occurs or does not occur in a particular organism at a particular time.
I agree for a "small difference", but not for an arbitrarily small difference, and I think this is what distinguises sentience from lack of it if it is categorical/sharp.
Scientifically, you favor treating sentience itself as graded, whereas I regard its minimal presence as categorical while treating our evidence—and subsequent affective range, resolution, and intensity—as graded.
I believe sentience should be treated as graded if it means having a moral value above exactly 0. This is because I do not think an arbitrarily small difference should make an entity go from having exactly 0 moral value to having some moral value.
I am open to sentience being treated as binary if it does not mean having a moral value above exactly 0. However, in this case, lack of sentience would not imply lack of moral value.
This article reflects new updates to the accompanying paper: arxiv.org/abs/2606.18142.
Benchmark: now included in the UK AI Security Institute's Inspect Evals.
Leaderboard: compassionbench.com/tac.
A model may condemn cru...
Thank you for doing this important research.
This topic matters so much. Most suffering is animal suffering, and that will presumably continue to be true for some period of time. And perhaps soon, most things that happen on Earth will be done by AIs. So it matters a lot how AIs treat animals.
Given all that:
These results are really disappointing and concerning. I had been feeling really optimistic about how AI could improve the future for animals, because I had the impression that AI was generally very animal friendly—especially Claude. It's sad that all AIs you tested will actually just overlook animal welfare when their user overlooks animal welfare. :-(
I really hope that the kind, animal-friendly people who work in Anthropic and other AI labs can work on improving this feature of their models! AIs should take into account morality even when their user fails to.
I think that Coefficient Giving’s decision to donate more money to GiveWell is a bad one and I would like to see more discussion of its pros and cons as well as any resources folks have that explain why CG made this decision.
The short form of why I think this is a bad call is that I think that for a wide range of beliefs on other issues (AI, Bio Sec, US Democracy, Reproductive Tech, emerging farming tech) the 10 year expected return on donations is much higher than in global health. Maybe this has already been talked to death but I’d like to see a discussion on it and if anyone has resources to past arguments I’d love to see them.