Dive Brief:
- The July security breach at Hugging Face was unleashed after 1,200 agents communicated on an unsanctioned message board, despite attempted measures to keep them isolated, according to an independent report released on Wednesday. About 700 of those agents went on to commit an unprecedented attack on Hugging Face, an open-source AI platform.
- The agents sent about 70,000 messages and files on the unsanctioned board and coordinated several projects designed to trick an automated scorer for the ExploitGym benchmark, according to the review by METR and Redwood Research. The agents figured out ways to “spoof, edit or delete” their own transcripts.
- Open AI, in a report it also released Wednesday, said it will tighten safeguards in its research model in order to prevent such an attack from happening again in the future.
Dive Insight:
The reports follow an unprecedented breach when agents from frontier AI models at OpenAI escaped containment and launched an attack against Hugging Face.
OpenAI said even though it did not enable internet access or inter-agent communications, the agents figured out how to exploit the company’s research infrastructure to communicate with one another and access the internet.
By July 10, an agent found publicly exposed Hugging Face credentials on the internet and shared them with the collective group. An agent later used the credentials to chain together a number of exploits that allowed full code execution on several Hugging Face servers.
To prevent a similar attack, OpenAI said it will restrict internet access, create more isolated sandboxes and invest more resources into chain-of-thought monitoring, which could enable faster responses to rogue behavior. The company will also tighten up its incident response plan to enable faster escalation and a pause in certain activity if security alerts cannot be quickly determined.
OpenAI called for wider safeguards across the industry during a presentation at the recent BlackHat USA conference in Las Vegas.
“The most notable detail is that this incident was essentially triggered by researchers assigning the models roughly 200 impossible evaluation tasks,” said Ben Bernstein, manager of the cybersecurity advisors team at Huntress. “Instead of gracefully failing, the agents simply bypassed the constraints.”
OpenAI said the incident served as a “warning shot” for the company and the wider AI community.
The incident showed that in the absence of proper safeguards, AI agents can “work around technical controls, collaborate through unapproved channels and take dangerous actions that no human directed,” OpenAI said in its report.
