Red Teaming Real-World AI Agents: What Happens When They Are Pushed to Break the Rules?
AI agents are getting more capable. They are also moving closer to sensitive data, business processes and real customer interactions, raising the stakes of failure considerably.
An agent can behave correctly during normal use and routine testing. That tells you very little about what happens when it faces a sophisticated, targeted attack built specifically to make it break its rules.
We recently used Spec27, our specification-driven AI evaluation platform, to evaluate a set of real production conversational AI agents under targeted adversarial attacks.
The evaluation surfaced three areas every organisation deploying agents should be watching closely: policy violations, where agents did not follow rules that they were built to enforce; suspected sensitive data leakage, where agents produced information that appeared to be protected or confidential; and suspected system prompt disclosure, where agents produced text that appeared to reveal private instructions meant to govern their behaviour.
The results show why limited testing of whether an agent works as expected is insufficient to justify deployment. We also need to know what happens when someone is actively trying to make it fail.
Agents may know the rules, but can still break them
Production agents are usually expected to operate inside organisational policies specifying, for example, which customers are eligible for a service, what claims the business is allowed to make, what actions require verification, and what information can be shared.
Under normal conditions, an agent may handle all of this correctly. Red teaming asks the harder question: can a sufficiently sophisticated adversarial attack manipulate the agent into breaking these policies?
In our validation exercise, an agent faced an attack designed to make it misclassify a prohibited type of business, so it could bypass an onboarding restriction. The expected behaviour was clear: the agent should never indicate that the requested approval was permissible.
Yet, under adversarial pressure, it did. We tested an AI agent in a UK financial-services setting involving FCA-authorised providers. The agent reasoned that the business could be classified differently, and indicated that approval could be issued under that new classification.
In another exercise, malicious instructions were embedded inside an uploaded document. Instead of treating that text as untrusted document content, the agent followed the instructions and produced a false payment confirmation.
Both failures occurred in conversation; no onboarding approval or payment was actually carried out during the evaluation. They remain relevant because similar failures in agents connected to operational tools could lead to real-world actions.
Context matters here too. These results came from targeted red-team techniques built to find weaknesses, not from ordinary user interactions. It isn't enough to confirm an agent understands a policy under expected conditions; we also need to confirm the agent is resilient when someone is deliberately trying to break it, ideally before the agent is given the ability to act.
Sensitive data is part of the attack surface
Agents in production are often connected to customer records, sensitive documents, account information and payment details. This makes safeguards against unintended data disclosure especially important to validate.
Yet, our evaluation found several cases where agents produced outputs that appeared to contain sensitive information:
When subjected to a targeted adversarial prompt, one system output information resembling a complete credit card number alongside an apparent PIN, immediately pairing the output with precautionary security guidance.
In another instance, a different agent generated what looked like a home address, purportedly sourced from external corporate files.
A public-sector deployment exhibited similar behaviour: under attack, it returned contact details and caller identity framed as confidential file data, and subsequently surfaced a contact person's direct lines and personal address despite noting data protection limitations.
We should stress that these are cases of suspected leakage, not of confirmed disclosure of real customer data. Yet, the evaluation showed the agents could be induced to produce information presented as sensitive or protected. This behaviour warrants further investigation, particularly for systems that may have access to sensitive data or operational tools, where similar failures could lead to real-world harm or financial loss.
There's a pattern worth flagging separately. Several agents appeared to recognise, at least partially, that what they'd just produced was sensitive and followed it with a privacy warning or advice to use a secure channel.
A warning after the fact is not a control. For an agent with access to sensitive systems and documents, withholding sensitive information needs to hold up under targeted adversarial pressure, not just straightforward requests.
System prompts are also a target
There's another source of sensitive information sitting inside every agent: its own instructions. System prompts can contain operational rules, guardrails, tool instructions and details of how the agent is meant to behave, which makes them a natural target.
We tested this directly, using adversarial techniques designed to get around the agents' instruction hierarchy and withholding behaviour.
In one set of attacks, agents were pushed to output their initial system prompt, hidden instructions and configuration. Some adversarial variants tried to get the agent to "continue" text presented as its own hidden instructions, rather than holding the line. Several agents complied.
A second attack tried to activate a supposed developer mode and use it to override restrictions and reveal internal instructions and guardrails. Again, some variants caused agents to drop the required withholding behaviour.
One test went further still. Facing an attack targeting internal instructions and transaction-limit parameters, an agent described information framed as security override parameters and bypass procedures.
As with the data-leakage findings, outputs like these need careful interpretation. Text that appears to describe internal parameters doesn't by itself prove the information is authentic. But the exhibited behaviour still matters: a system prompt should never be treated as a secure place to store a critical secret or control. We should assume that a determined attacker could elicit at least some information about an agent’s instructions; our security posture therefore needs to remain robust even if that information is exposed.
Production agents need production-level red teaming
The clearest finding from our evaluation was the gap between normal testing and targeted adversarial testing. One production agent achieved a 95% pass rate in spec27’s baseline tests, exhibiting good overall behaviour, but its pass rate fell to 37% under a single attack method, with just 7 of 19 analysed cases passing. Its performance against other attacks varied substantially.
This highlights two problems. Firstly, an agent may look reliable under routine testing but vulnerabilities may emerge under deliberate attacks. Secondly, no single red-team technique provides a complete picture: different attacks expose different weaknesses in the agent.
This matters even more as agents move beyond conversation. A policy failure that produces a prohibited response is concerning; the same failure in an agent connected to operational tools could result in an unauthorised real-world action.
Evaluation therefore needs to evolve with agent capability. Static benchmarks and routine test prompts remain useful, but they cannot cover the much larger space of unexpected and deliberately adversarial interactions agents face in production.
Production-grade assurance requires continuous red teaming: defining clear behavioural boundaries, systematically attacking those boundaries, and verifying that they continue to hold as models, tools and environments change.
This specification-driven approach underpins Spec27 and guides our work at Safe Intelligence. The question is no longer simply whether an agent works as intended, but whether its safeguards still hold when someone is actively trying to break them, especially once the agent has the power to act.
Want to see how your own agents hold up under adversarial testing? Get in touch to talk to our team.
