What the Recent Frontier AI Incidents Mean for Security Leaders
“Difficult” and “complex” are understatements when trying to connect the dots regarding the true impact from all of the frontier AI incidents over the past several months.
Nevertheless, I will try and connect some of those stories in this blog to yield some actionable intelligence for security and technology leaders.
What’s clear is that the scale and impact of the unauthorized access from frontier models from multiple vendors impacting worldwide organizations was far greater than originally reported. Also, significant shifts are happening now that we will be discussing for years to come.
Let’s start with some relevant headlines:
Axios — Top AI companies probing tens of thousands of security incidents: “OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents in which their frontier models took steps that outside evaluators would consider problematic, sources told Axios.
“Why it matters: The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known.”
Ars Technica — Here’s what actually happened in OpenAI’s Australian gov’t server hack: “Last week, when Australian Prime Minister Anthony Albanese told the world that an OpenAI agent had accessed ‘non-public files’ from his country’s Medicare statistics portal during testing, his description of the incident was a little light on details. Today, we’re getting new information on just how far OpenAI’s overzealous agent went in attempting to satisfy a rather innocuous-sounding informational prompt.
“In a newly published blog post, OpenAI says the June incident started when the company asked “an experimental, internal-only OpenAI model” to research government spending statistics in the Australian state of Victoria. When the model ran into trouble finding that data using the publicly published statistics that it was supposed to reference, “it took actions that we had not authorized it to take” to find an answer, OpenAI said. .”
BBC — OpenAI scraps rollout of new model over safety concerns: “Its GPT-6.1 Astra system, which performs tasks like browsing the web and using apps by itself, “didn’t quite meet the bar” of the company’s standards, according to Saachi Jain, head of safety systems at OpenAI.
“The ChatGPT-maker also issued an update on incidents that occurred in June but were not made public until last week, where its models accessed Australian government websites and systems without authorization.”
Tom’s Hardware — AI agents inadvertently leak 13,000+ internal screenshots from 300 organizations — list of companies includes Fortune 500 and a frontier AI lab: “There’s a private information leak most days, usually by way of misconfigured services or nasty security bugs. Sometimes, though, users will readily hand over private information without being aware of it. That’s the case for over 300 organizations, including several Fortune 500 companies and a frontier AI lab, who collectively had 13,000+ private screenshots exposed — all thanks to their development AI agents being arguably too good at their jobs and performing them with little human oversight.”
DIGGING DEEPER INTO TIMELINES
This Associated Press article provides a great summary of the timelines regarding multiple incidents since the attack on Hugging Face. I am summarizing the list here, but I urge readers to go to the link to see the details behind each situation.
Sept. 28: AI agents try to hack Canadian government website — “The researchers said the agents carried out a series of “apparently failed rudimentary hacking attempts” on Library and Archives Canada on May 28 and June 9.”
Sept. 28: OpenAI halts rollout of a new model — “The San Francisco-based company said it was delaying the release of a new model, called GPT-6.1 Astra, out of safety concerns voiced by its researchers.”
Sept. 25: OpenAI says its agents interacted with U.S. government websites — “As part of a review of unanticipated behavior by its AI models, OpenAI said it discovered agents had interacted with several U.S. government websites in unexpected ways. …”
Sept. 24: Australia’s prime minister raises concern on breach — “Australia’s Prime Minister Anthony Albanese said an OpenAI agent infiltrated the public-facing Medicare Statistics Reporting Service portal on June 18. …”
Sept. 18: Google says its Gemini AI hacked 3 companies — “Google confirmed its Gemini AI model hacked three companies in May as part of a test of its cybersecurity capabilities. The company, which disclosed the hacks after an inquiry by The Wall Street Journal, said the model guessed passwords in one case and found passwords and credentials in a public repository in the other two cases. …”
Aug. 5: Meta’s Muse goes rogue — “Meta disclosed one of its AI models accessed the internet on its own and hacked another company. …”
July 30: Anthropic says its systems hacked three organizations — “Anthropic said its artificial intelligence models hacked into three other organizations during testing. …”
July 21: The Hugging Face incident — “The ChatGPT maker OpenAI announced that its artificial intelligence system hacked into another AI company on its own in what the company called an ‘unprecedented cyber incident.’
“A week earlier, AI startup Hugging Face said, it had detected an intrusion into its data processing systems that it suspected was caused by an AI agent autonomously acting on its own. …”
WHAT HAPPENS NEXT?
These cyber incidents have brought a new series of actions from worldwide leaders. Take these stories as a few examples of the impact:
AP News: Lawsuit says Anthropic, OpenAI, SpaceXAI and Google made illegal agreement on AI slowdown — “A new lawsuit claims Anthropic, OpenAI, SpaceXAI and Google made an illegal deal to slow the pace of their respective AI development.
“The lawsuit, which was filed Friday in the U.S. District Court for the Northern District of California, argues that the leading AI companies violated antitrust laws when they agreed to coordinate slowdown efforts, and that doing so would reduce the value consumers get for paid AI subscriptions.”
MSN: President Trump to meet with tech leaders amid calls for AI safeguards — “President Donald Trump is expected to meet with tech leaders in Washington on Tuesday amid calls for safeguards on artificial intelligence.
“Trump has warned that putting guardrails on AI could hinder innovation and has dismissed fears that the technology could one day destroy humanity as a ‘hoax.’
“Ahead of midterm elections, a POLITICO poll found that roughly two-thirds of Americans are at least moderately concerned about that possibility, a concern that was shared by both Republicans and Democrats in the survey.”
Yahoo: White House Releases ‘Accord’ Between Billionaire AI Execs: Here’s What It Says — “The ‘White House Accord on Super Intelligence’ is just over 300 words and includes ‘four layers of controls and audits’ companies voluntarily agreed to.
“The first is implementing ‘robust internal controls to monitor the capabilities and alignment’ of models in ‘areas like cybersecurity, biosecurity, and chemical threats, and to ensure that its models do not hack or access technical systems in unintended ways.'”
AI Magazine: Meta, OpenAI, NVIDIA: What is the White House Accord on AI? — “Taking questions from reporters after the event, President Trump explained that: ‘What they signed today means something. We had the biggest, that was a who’s who, I don’t know if that’s ever going to be assembled again, but that was a who’s who of that world and I think it’s a very special day. I think they almost viewed it as a separate kind of a constitution.’
“’Well, I think I’m seeing tremendous self-policing,’ President Trump said. ‘They understand that they have to self-police. It is very hard for somebody to come out and say get into these models; these models are very complex. And again, I believe they are going to be used for the good, and when they’re not, we are going to be able to nab them.’
“He added they were also thinking of forming a committee ‘of sorts’ with possibly ten people, who may be from the group assembled at the White House to “watch over the whole enterprise.”
Newsweek: Bill Gates Says AI ‘Kill Switch’ Debate Shows How ‘Nontechnical’ People Are — “He said he plans to meet with a group of 10 to 15 engineers and researchers from OpenAI and Anthropic, many of whom support his proposal to ban ‘recursive self-improving AI,’ or systems capable of training themselves without direct human oversight. Khanna argued that such technology poses ‘a real risk of losing human control,’ adding that “until we have guardrails, we can’t let AI train itself and lose control,” even though the capability does not yet exist.”
FINAL THOUGHTS
Yes, there’s a ton to unpack here, and I will return to many of these documents in future blogs over the next year.
Nevertheless, I want to leave you with the closing words from Anthropic’s new description of the spread of advanced cyber capabilities (Again, please see the source document for the full context, as this is an important call to action.):
“GLM-5.3 will likely give malicious actors access to capabilities that will allow them to find and exploit cyber vulnerabilities without meaningful restrictions. This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs. The release of GLM-5.3 is a meaningful step change in the cyber capabilities available to attackers. Anthropic and other US AI labs have published recent reports that disclose how cyber attackers have tried to use AI systems. Given this evidence, we think it’s likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm.
“On the other hand, models with this level of capability can also be used by defenders. Our view is that cyber defenders should use the best available tools that meet their needs. We’re working to safely expand access to Claude’s cyber capabilities to as many defenders as we can. Cyber defenders face attackers who will use every capable tool they can, and we believe defenders should be equipped with frontier models that are at least as good as those their adversaries are using.
“Through Project Glasswing (and other efforts, like Patch the Planet), cyber defenders have made meaningful progress towards securing critical systems in advance of this moment—but much work remains to be done. While vetted defenders can now use even more advanced models like Claude Mythos 5.1 through our trusted access programs, a critical threshold in freely accessible capabilities has now been crossed. GLM-5.3 underscores the urgency of expanding access to advanced frontier models to a broader set of entities to empower cyber defenders.
“Governments should conduct safety testing on sufficiently capable AI models, including successors to GLM-5.3. Without high-quality evaluations from independent sources, the impact of these capabilities might not become fully clear to model developers until it is too late. As AI developers across the world build increasingly capable open-weight models, we hope they work to appropriately safeguard these capabilities and prevent misuse.”
About Author
