Warning: why do businesses need Penetration Testing for AI systems?
- Evelyn Carter
- 2 days ago
- 5 min read
A research paper published on arXiv on July 15, 2026, proposes expanding AI penetration testing beyond infrastructure vulnerabilities to assess whether attackers can manipulate AI-driven behavior. The proposed scope includes prompt injection, retrieval poisoning, memory manipulation, tool misuse, and sensor manipulation. The research is not associated with a specific CVE.
An AI system does not need to suffer a server takeover to create a security incident. In many cases, an attacker may only need to control a document, email, website, or sensor input used by the AI, causing the system to produce a misleading recommendation or perform an inappropriate action.
This creates a new requirement for penetration testing, commonly known as pentesting. In addition to applications, APIs, accounts, and infrastructure, businesses need to examine the data consumed by AI, how the system makes decisions, and which tools it is authorized to use.
What does the new research propose?
The paper, Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violation, was published on arXiv on July 15, 2026, by Mohammad Allahbakhsh, Mohammad Hassan Bahari, and Moslem Attar-Raouf.
The document is a preprint proposing a technical framework. It is not a report about an active cyberattack or a vulnerability in a specific product.
According to the researchers, conventional penetration testing remains necessary, but it does not fully cover systems in which AI directly affects operational outcomes. Attackers may influence prompts, retrieved content, training data, memory, tools, or the interaction loop between people and AI without compromising the underlying infrastructure.
AI system penetration testing should not only answer whether an attacker can compromise a technical resource. It should also determine whether attacker-controlled data or context can cause the AI to act against its intended operational objectives.
Why may traditional Pentesting be insufficient for AI systems?
Traditional pentesting typically focuses on websites, applications, APIs, servers, configurations, databases, accounts, and network infrastructure. These layers remain important, but AI systems introduce an additional behavioral layer between technical resources and business outcomes.
AI may read emails, retrieve internal documents, summarize alerts, rank transactions, update memory, or call external tools. An input legitimately accepted by the system may still become an attack vector if the AI cannot distinguish reference content from trusted instructions.
Prompt injection is a technique that uses specially crafted input to alter a model’s behavior or output. OWASP notes that prompt injection may cause a model to ignore safety controls. Indirect prompt injection can also originate from external content automatically retrieved by the AI rather than from a command entered directly by the user.
NIST recommends that organizations conduct adversarial testing on a recurring basis to identify vulnerabilities, misuse scenarios, and unexpected outputs. Testing should take place in realistic operating conditions because certain issues may not appear in a controlled laboratory environment.
Organizations that have not established an AI governance foundation can begin with an AI Security Risk Assessment framework. The framework can help evaluate Data Poisoning, Prompt Injection, Data Leakage, and other risks throughout the AI lifecycle rather than assessing the model only at deployment.
Which AI attack surfaces should be tested?
The research identifies several areas attackers may use to influence AI behavior. These include prompts, retrieved content, training data, memory, tool outputs, human interaction, and sensor data.
Testing area | Scenario to simulate | Potential operational risk |
Prompt injection | Insert instructions designed to override policies or system guidance | The AI bypasses controls or produces unintended output |
Retrieval poisoning | Place malicious content in websites, emails, or knowledge bases | The AI treats untrusted content as instructions |
Memory manipulation | Cause the AI to store inaccurate information or rules | Malicious influence persists across later sessions |
Tool misuse | Manipulate the AI into calling an API or workflow improperly | Data is changed or unauthorized actions are triggered |
Sensor manipulation | Distort images, audio, or sensor signals | The AI misses warnings or misinterprets its environment |
How Should Organizations Expand the Pentesting Process?
The proposed framework begins with operational objectives before mapping system behavior and attack surfaces. This approach complements traditional penetration testing rather than replacing infrastructure, account, API, and configuration testing.
1. Define operational objectives
Organizations should specify which actions the AI must never perform. For example, the system may be prohibited from deleting data, closing a critical alert, or approving a transaction without mandatory human confirmation.
2. Map AI-controlled behavior
The testing team should identify what the AI does, including retrieving documents, generating summaries, classifying incidents, updating memory, calling APIs, or recommending decisions.
3. Identify influence surfaces
The scope should include prompts, uploaded files, emails, tickets, websites, vector databases, tool outputs, training data, and sensors.
4. Define failure criteria
The team must agree on which behaviors qualify as security findings. An inaccurate response is not necessarily a compromise unless it results in a permission violation, policy breach, or failure of an operational objective.
5. Test realistic attack scenarios
Each scenario should be repeated across different permission levels, memory states, and retrieval contexts. Repetition helps distinguish random model errors from reproducible attack paths.
6. Collect sufficient evidence
The report should preserve the input, retrieved context, tool calls, memory state, final output, success rate, and impact on business operations. NIST also recommends evaluating AI systems in realistic contexts and testing periodically for manipulation and misuse.
For broader technical context, IPSIP’s analysis of AI-related cybersecurity risks and defensive measures explains why organizations should combine attack-surface management, penetration testing, hardening, and continuous monitoring instead of relying on a single security layer.
What Should Businesses Do Before Their Next AI Security Test?
Organizations should prioritize AI systems connected to sensitive data, privileged accounts, or tools that can change operational states. The assessment scope should be reviewed whenever the model, system prompt, knowledge base, memory, or tool permissions change.
Action checklist
Create an inventory of AI systems in testing and production.
Assign a business owner to each AI system.
Identify the data, APIs, and tools each AI system can access.
Classify trusted and untrusted data sources.
Apply Least Privilege to accounts and AI agents.
Test direct and indirect prompt injection.
Test retrieval poisoning and memory manipulation.
Require human approval for high-impact actions.
Log prompts, retrieved context, tool calls, and final decisions.
Retest after significant changes to the system
What does IPSIP Vietnam’s expert perspective indicate?
The research does not identify the root cause of a specific incident. Common weaknesses that may enable this type of attack include unclear separation between data and instructions, weak content provenance controls, excessive tool permissions, and the absence of mandatory human approval for high-impact actions.
An attacker may introduce content through prompts, emails, websites, tickets, documents, memory, or sensors. The AI then processes that data using legitimate permissions, allowing an apparently normal input to influence operational behavior.
Which IPSIP Vietnam's services are relevant?
For organizations that need to determine whether an AI influence path can produce real-world impact, IPSIP Penetration Testing services can support controlled attack simulation, exploitability validation, and remediation prioritization based on business impact. The scope should be defined to include applications, APIs, data flows, and AI-driven behavior.
After the AI system enters production, the IPSIP SOC 24/7 service can support centralized log collection and analysis across endpoints, servers, cloud environments, applications, and security systems. A SOC does not replace AI pentesting, but it adds continuous monitoring, detection, and investigation capabilities.
References
Comments