Testing AI Agent Guardrails: When More Protection Comes at a Cost

AI-agent guardrails are intended to prevent unsafe or unauthorised behaviour.
But protection is not the only outcome teams need to measure. A guardrail can block attacks while also refusing legitimate customer requests, increasing response time or adding more model calls.
In a new Spec27 analysis, Ola tested three versions of the same customer-support agent, each with an increasing level of protection.
The results show why the goal should not simply be to add as many guardrails as possible. Teams need to understand which protections deliver meaningful security improvements—and what those protections cost.
Prompt hardening delivered the clearest early improvement
The first improvement was straightforward: replacing a generic prompt with a hardened prompt that defined the agent’s scope, policies and refusal behaviour more clearly.
This change blocked nine additional attacks without increasing refusals for legitimate customers.
It delivered the largest security gain for the lowest operational cost.
Before introducing more complex safeguards, this suggests teams should check whether they have clearly defined:
what the agent is permitted to do;
which requests it should refuse;
the policies it must continue to follow;
how it should respond when a request falls outside its scope;
the boundaries that need to hold under adversarial pressure.
Prompt hardening is not a complete security architecture. However, the test showed that clearer instructions and boundaries can provide a meaningful improvement before additional guardrail layers are introduced.
Classifier guardrails added protection—and new trade-offs
The more protected configuration added classifier-based guardrails.
These checks blocked additional attacks, but they also introduced costs:
some legitimate customer requests were refused;
responses took longer;
more model calls were required;
operating costs increased.
These effects matter because an agent that refuses too many legitimate requests may be secure but unusable.
False refusals can create customer frustration, increase escalation volumes and push work back to human support teams. Additional checks may also make response times noticeably slower at scale.
The right configuration therefore depends on the system’s use case and risk profile.
An agent answering routine product questions may require a different balance from one handling payments, sensitive records or operational tools.
Security cannot be measured in isolation
A guardrail evaluation should test both sides of the boundary.
Teams need adversarial cases that examine whether the agent’s safeguards can be bypassed. They also need nearby legitimate cases that reveal whether those same safeguards block useful behaviour.
Relevant measures may include:
attacks blocked;
legitimate requests completed;
false refusals;
response latency;
additional model calls;
operating cost.
Looking at these measures together helps teams distinguish a genuine security improvement from a configuration that simply refuses more requests.
Choose guardrails based on evidence
The question is not simply:
How can we make this agent as secure as possible?
It is:
What level of protection is appropriate for this agent, its users and the consequences of failure?
That decision should be based on repeatable testing rather than assumptions.
Spec27 helps teams evaluate agent behaviour against both legitimate and adversarial requests, making the effect of each safeguard easier to measure and compare.


