My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).
The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.
HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”
While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.
Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.
Furthermore, if I'm reading between the lines correctly, OpenAI did not address the issue (and perhaps didn't even know about their models doing this) until after HuggingFace's public blog post.
This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about.
Me and @Fran are interviewing @GraceAdams🔸 tomorrow evening. Is there anything you'd like us to ask her?
Thinking back to my first big (well, big for me) donation and an unusual series of thoughts I had. Sharing in case anyone has experienced the same.
--
I've practiced frugality to a significant extent, largely because I want to donate/salary sacrifice so much of my income to effective charities. As a result, whenever I'm spending a significant amount of money on anything, alarm bells go off.
I was surprised that alarm bells went off when donating. I had thought through where I donated extensively, and charity was the reason why I wanted to save money in the first place. But I still felt stressed because I was "spending money."
This is such a clear example of missing the forest for the trees. I think we need to be careful about instrumental values/rules and making sure they don't become absolute limitations that decrease overall impact.
We have bought 1500 copies of IABIED in Russian; over the past few days, sent emails to 1700 winners of olympiads who have previously received HPMOR from us offering the new book; and already shipped 173 copies to them (+ sent 43 e-books)
This post was partly inspired by, and shares some themes with, this Joe Carlsmith post. My post (unsurprisingly) expresses fewer concepts with less clarity and resonance, but is hopefully of some value regardless.
Content warning: description of animal death.
I live in a small, one-bedroom flat in central London. Sometime in the summer of 2023, I started noticing...
This is a linkpost for Subjective Probabilities should be Sharp by Adam Elga, which was originally published in Philosophers' Imprint in May 2010. Here is an errata for it. Below is a summary...
I object to relying on infinities (not arbitrarily large finities) to guide decisions because they do not explain more empirical evidence than arbitrarily large finities.
But do they deserve to be privileged at the exclusion of the infinite EV distributions if and when the latter are at least as consistent with the evidence? Why? Shouldn't you use some principle of indifference or symmetry here?
If there's no finite upper bound on what the value the specific probability distribution can take, how can you be 100% confident it is not a probabilistic mixture with a distribution with infinite or undefined EV (but finite for every actual value), like 0.0001% probability to it being drawn from something like a St. Petersburg lottery?
In my cost-effectiveness estimate of corporate campaigns, I wrote a list of all the ways in which my estimate could be misleading. I thought it could be useful to have a more broadly-applicable version of that list for cost-effectiveness est...
Hi Saulius. I think this is such a great post. I just reread it because it was shared in the last EA Forum newsletter.
🧭 What: Participant-driven Working & Community event for people ambitious to reduce and prevent suffering for all sentient beings, near and long-term.
📅 When: 27-31 August 2026 (arrival possible from Thursday 10 am, official program runs from 3 pm to Sunday evening).
📍Where: Berlin, Germany, ...
FYI: To facilitate travel planing I added the following details:
Arrival day (Thursday): Registration opens at 10 am. Informal games and community-building activities will run throughout the day until the official opening session at 3 pm.
Sunday: The impact-focused program will run until dinner (6 pm). After dinner, we will have the closing session and continue with more community focused activities into the evening.
Departure day (Monday): We'll share breakfast and clean up together Monday morning. Check-out is by 10 am.
Attendees are warmly encouraged to spend the day (or more) in Berlin together after the summit. It's a great opportunity to keep connections and conversations going before heading home. We'll be using Slack well in advance of the event to coordinate logistics, travel, and anything else that comes up.
EA spends a great deal of effort asking which causes, interventions, and careers have the greatest impact. We spend much less time systematically asking which general-purpose skills make us better at answering those questions—and better at acting on the answers.
Reasoning under uncertainty, evaluating evidence, learning efficiently, communicating clearly, coordinating people, and revising beliefs affect performance across almost every role and c...
This is an update on my previous posts in which I detailed my plan on how I am going to donate the vast majority of my money to charity when I grow up. I want advice on whether my plan for the future is good, or what I should do differently. I am currently 15, so I have been trying to make decisions that will a) help me maximize my future income by getting into a good college and learning skills so I can donate more to charity and b) help me maximize my lifespan so I can live as long as possi...
Tl;dr: A few years ago, if you wanted to found an animal advocacy organisation, there was basically one incubator. Now there are at least nine programs, plus several adjacent accelerators, and the hard part has shifted from finding support to choosing the right kind. This post maps the main options across focus, format, funding, and fit, so you can spend your energy on your idea rather than on decoding the landscape.
AI disclosure: I asked Claude to research the curr...
Hello, I'm new here but it seems to me a little confusing in the current way the post is framed and I think this is because AI seems to bucket things that are a bit more nuanced. The title in itself is misleading because food fellowships by SMA is not an incubator so why is it in there. I dont think a human would write this into the post, same for the RP founder support and it even says Myrias is a venture studio not an incubator. So the title says a map of incubators but its a map of some parts of the talent system and not others with a few incubators and a few other things?
I’m not sure. The claim that cold outreach signals a lack of respect for boundaries is unnecessary moralising that I don’t put stock in. If your point is simply that it doesn’t work, people can find that out for themselves; I don’t think the externalities are so negative that donors need you to act as their mouthpiece on this. You’ve apologised for sounding patronising towards fundraisers; I’d be more concerned about coming across as patronising to donors, who themselves are not a homogeneous group. Some will be more receptive to thoughtful, targeted outreach than others, especially if the underlying project or work is persuasive.
I’m also uneasy with the alternative being proposed, of becoming ‘legibly excellent’ to the small set of advisers a donor already trusts. That may be good advice for organisations already close to those networks. But as a community norm - one frequently criticised as elitist - I read it as ‘don’t approach new donors directly; instead, prove yourselves to the existing gatekeepers and hope we choose to let you through.’
This post criticises people for opportunistically inserting their own projects into conversations about Anthropic wealth - while repeatedly emphasising the author’s own proximity to these donors, highlighting his own fund, and ending by recommending his own advisory. Ironically, I’m left wondering whether part of the concern about cold outreach is that, when it does work, it allows organisations to bypass the existing intermediaries. Middlemen tend not to be enthusiastic about that.
I thought the section on scaling was useful, though.