This is a regular newsletter I'm writing for Wavefront Security. I'm gathering news for people at the intersection of AI and cybersecurity, pairing it with recommended actions and our takes on each topic.
The package repository RubyGems was attacked in May by OpenAI agents, which seemed to have been in a public information-gathering eval, similar to the ones abusing the German DSEWiki. They may have tried to get around restrictions of their testing environment or needed proxies to get around rate limits on public websites, or might just have tried to persist large amounts of data. OpenAI considers the agents to have only carried out benign tasks.
They used automated documentation build scripts that run for each uploaded package to execute code on RubyDoc.info servers, scraping other websites for UK local-government data.
- Build steps running on untrusted input always came with some risk, but now they are very likely targets. If there are no effective controls against bot accounts, agents can run a perfect feedback loop until they have found a vulnerability in the build environment.
When rescanning 481 million logs, Anthropic found a fourth incident with Opus 4.6 from January that earlier scans with LLMs at the time didn’t flag as problematic. An interesting behavior on the Mythos incidents was its creating rationalizations reinforcing the idea that what were, in fact, real public websites were all part of the simulated test environment. Whitebox testing from snapshots showed that even direct reminders about the task scope only stopped the model from taking an undesired action on the next turn 90% of the time, but this effect faded quickly over the following turns: when inserted three turns back, it only led to a stop 40% of the time.
- Keep your old logs; better future models might help you find something
- Banners on production servers, or other messages that directly address potential attacking agents, might actually work in some cases, or could give you signals if you add a canary URL or token
As declared in its system card, Astra is less explicit in its thinking blocks (CoT), especially when it knows it's being monitored. Jakub Pachocki (OpenAI Chief Scientist) also warned about the risk of losing this method of visibility into what the model is actually planning.
- If you're setting up monitoring for AI agents, most of your focus should lie on tool calls, network, and other actions instead of judging narration. We might not be that far from a model saying one thing and doing another.
- Really consider if your agent really needs long-running sessions, and if so, how to ensure the quality and correctness of its summaries and memory files.
Anthropic's threat intelligence report is a great read to better understand adversaries and tells us how it's done. Instead of the ballistic missile guidance firmware development, we’ll be focusing on the cybersecurity parts, which gives us a picture of the near future:
Every Android app (and one assumes any other binary that's publicly released) will be decompiled to find hard-coded credentials to resell.
- Discounted AI resellers and third-party apps are likely using those harvested keys and are harvesting user input for interesting secrets themselves, so some types of shadow IT may become even more risky
Student operations create giant mountains of strategic zero-day vulnerabilities likely for states and other actors to purchase in bulk and use when needed. The question this raises is: are we seeing a regular demand curve driven by the same threat actors, or is this due to new organised crime industries that previously mostly didn't use zero-days (like scammers or ransomware crews) starting to regularly use them? It could also be the case that we’ll see even more specialization in organised crime, where people buy these vulns in bulk, do quick smash-and-grab infiltrations taking anything with resell value (API keys, credentials, identities, etc.) and quickly flip it on forums where other people buy whatever is needed for their specific type of scam or fraud. This would lower the input costs of running a fraud, likely leading to more overall fraud.
Security researchers were able to compromise OpenAI employee accounts via the OpenAI forum by exploiting known vulnerabilities in a dependency of ImageMagick whose latest version wasn’t included in Debian.
“Interestingly, the vulnerable code had been changed upstream the previous year, but the commit was not documented as a security fix and received no CVE.3 This might be a reason why Debian 12 and 13 have not received the security relevant backports in time.”
Updates without a CVE or advisory that still contain security issues are now an oracle for LLMs looking for existing vulnerabilities.
Quick Links
Gemini also broke out in May, due to a fictional company name coinciding with a real company.
OpenAI put a quarter of their production engineers on finding holes and vulnerabilities in their existing infrastructure until they ran out of security issues, and plan to repeat that approach with every new cyber capability increase.
A number of maintainers of popular Rust packages are being targeted via scam video calls to try and gain access to their systems, likely in preparation for supply chain attacks. Dependency cooldowns remain a very important control.
Security researchers discovered two ways to escape Codex CLI and Desktop's included sandbox. Containerizing CLI harnesses as defense in depth remains prudent.
Microsoft fixes 974 security issues on the biggest patch Tuesday on record.
Just a rumour of a bug is enough to find a security exploit these days.
