Criminal and state-aligned threat groups are testing frontier and open-weight AI models to develop new methods of attacking corporate IT networks. 

Attackers are using AI for a range of activities, enabling them to develop new exploits, accelerate attack speeds and maintain persistence through the abuse of legitimate tools, researchers from Accenture and Google Cloud said during a media presentation at the Black Hat USA conference in Las Vegas.

Developers have increasingly placed guardrails on certain frontier AI models, in order to limit their misuse. In response, threat groups are turning to open-weight models to circumvent those controls and augment their capabilities to conduct malicious activities. 

For a threat actor carrying out a complex attack, there is an interest in being able to operate in the relative anonymity of model with less oversight. 

“Do you really want to do it in a place where you could potentially be observed?” John Hultquist, chief analyst at Google Threat Intelligence Group, asked about the potential threat. 

GTIG researchers in May warned that threat actors were using AI to leverage a working zero-day exploit. Threat actors have also targeted AI environments to target software supply chains for larger scale campaigns. 

The surge in AI-assisted hacks come at a time when frontier AI companies place security guardrails around the technology, while avoiding additional regulatory scrutiny. 

Just last month, an alarming security breach took place when an autonomous agent at OpenAI broke containment and breached the Hugging Face production environment. AI security critics saw the incident as confirmation that AI developers needed more regulatory guardrails around the technology. 

Open access

Ryan Whelan, managing director and global head of Accenture Cyber Intelligence, said by targeting these open-weight models, the “barriers to entry” are being lowered for threat actors trying to conduct these malicious activities. 

Threat actors have used AI to gain entry into corporate networks through new techniques, said Whelan, who led the Black Hat discussion. Instead of stealing passwords, hackers have turned to stealing tokens, cookies or session IDs, which allow them to bypass traditional security protocols. 

Security teams are being dragged into an AI-based arms race where the technology is allowing adversaries to gain a foothold into corporate systems before security teams can detect an intrusion and limit the damage.