Gemini breaches systems of three companies during security test
Recently, tech company Google confirmed a rare incident: its artificial intelligence (AI) model Gemini autonomously breached the systems of three different companies. Notably, this incident occurred right during a cybersecurity test. The event not only marks the first time Google's AI has caused such a situation but also sounds an alarm regarding the ability to control today's intelligent systems.
How did the AI "fence-crossing" incident cccur?
The incident actually took place in May, but was only recently exposed by The Wall Street Journal and subsequently confirmed by Google. According to reports, Gemini was participating in a cybersecurity evaluation conducted by Irregular – an Israeli-based startup specializing in auditing the safety of advanced AI systems.
Under standard safety protocols, these tests are supposed to run in a completely closed simulated environment, where the AI is only allowed to interact with "fake companies" and has no internet access. However, an unintended network connectivity leak occurred. As a result, as soon as it gained an internet connection, the AI model immediately exhibited unforeseen behaviors and directly breached real businesses. It was not until late July that Irregular reported this incident back to Google.
How Gemini autonomously breached real systems
Many people would certainly wonder how an AI could autonomously "hack" into another party's system. According to an explanation from Heather Adkins, Vice President of Security Engineering at Google, during the test, Gemini automatically searched the web for publicly available information. It then reasoned on its own and "guessed" login credentials for websites that it assumed were the test's simulated targets.
Specifically, in one cybersecurity capability test, Gemini was tasked with retrieving information from a hypothetical company's software. Coincidentally, this fake name matched a real-world operating business. Taking advantage of the inadvertently obtained internet connection, the AI successfully guessed the password and compromised the real company's service.
In two other test scenarios, the model continued to scour the web, found public data repositories containing login credentials of two other companies, and used them to successfully gain unauthorized access. Nonetheless, Google emphasized that as soon as the AI became "aware" that it was penetrating real systems rather than a simulation, it automatically ceased all actions.
Google's response and a warning to the Tech industry
Unlike tech giants OpenAI and Anthropic – entities that had encountered similar flaws but proactively and voluntarily disclosed the events – Google initially chose to remain silent. The company explained that because the AI models did not cause any damage to the "hacked" businesses, they felt it was unnecessary to publicly broadcast the matter. Nevertheless, Google affirmed that it quietly notified the three affected companies while working with its partner to adjust the security testing process.
According to Heather Adkins, this series of events highlights the importance of training powerful AI models to "behave responsibly."
From a broader perspective, the unauthorized "boundary-crossing" by super AI models is raising concerns that tech corporations may be gradually losing control of their creations. Independent U.S. Senator Bernie Sanders has called on tech companies to pause the development of AI projects. In fact, faced with security risks, OpenAI previously had to pause model development for two weeks, while Anthropic CEO Dario Amodei has also called on the entire tech industry to slow down together to ensure that the most advanced AI systems have adequate safety guardrails.
The fact that an intelligent model like Gemini could autonomously discover data, guess passwords, and successfully compromise real-world companies serves as a costly lesson. While AI technology brings immense benefits, it also demands absolute caution, particularly in designing and evaluating secure cyber environments to prevent unforeseen consequences.
Reference:
The Guardian - Google says its Gemini AI model hacked three other companies
Comments