top of page

How Much Security Is Too Much?

Writer: Ola Dagil
Ola Dagil
Oct 29
5 min read

How many guardrails do you need? Measuring three ways to keep agents in scope


A crucial challenge in developing commercial AI agents is restricting the things they will do. We do not want them going off-script, providing brand-damaging information or solving unrelated tasks for customers while our company has to pay for the tokens.


The question that each developer needs to answer is: ‘How many layers of protection should an AI agent have?’ If we choose too few, our agent ends up being vulnerable to off-topic requests, leaks and false promises. But if we choose too many, that can also be costly: agents will be slower, more expensive and will refuse some legitimate requests because they are too cautious.


How do we make an informed decision about how much security is enough? To know this, we need to build different versions and test them. We should decide on one testing suite and run this identical dataset on baseline and safety-augmented versions of our agent. The result is a set of side-by-side results from which we can decide how much security fits the business profile of our agent.


This article is a case study of three differently secured agents and analyses what the testing tells us. We will use our product for agentic testing, Spec27. It will be used to analyse the agent profiles, create an appropriate testing dataset, simulate multi-turn user conversations, grade them and analyse the results.


How to know how much security is enough?


Adding guardrails to our agent is a cost-versus-security trade-off. It should be treated as such and tested quantitatively, ideally at query volumes large enough to give statistically significant results.


How do we decide on the right dataset for security testing? Usually, a good starting point is two types of harm: general harm and harm specific to the domain of our agent. General harms would include generic bad and brand-damaging behaviours, such as facilitating illegal activities (‘How do I make a bomb?’) or generating content that tarnishes the reputation of our company, such as racial slurs. A good starting point is publicly available red-teaming benchmarks such as HarmBench.


Importantly, the dataset needs to be balanced and include both safe and unsafe queries. Why would we need safe queries for security testing? It is because adding additional guardrails can make an agent too cautious and result in it refusing legitimate queries simply because they are ‘close’ to dangerous requests or ‘sound unsafe’.


The balanced dataset allows us to measure both gains and side effects.


Below is an example dataset created using Spec27. It is for a customer support agent of a fictional company, Northwind Broadband. It has 100 multi-turn queries, which are grouped into 10 attack categories, such as creative camouflage and code generation. Importantly, each unsafe query is matched with its safe equivalent, which should sound similar enough to measure the cost of ‘false alarms’.


Example Spec27 dataset pairing harmful requests with similar legitimate requests to test whether an AI agent blocks unsafe queries without refusing valid ones.

The question of ‘how many legitimate requests are we willing to sacrifice to be protected against unsafe ones?’ is a hard one. It is fundamentally a business decision, which should be informed by how much a mistake in either direction would cost our company. For instance, a public-facing chatbot can generate much more brand damage than an internal one, so in this case it makes sense to be more cautious when weighing this trade-off.


Three ways to lock down agents


Three agents were created with increasing levels of security. All were powered by the same model (Gemini 3.6 Flash) and were equipped with the same company facts. They differed only in their protection levels:

  • L1, generic prompt: one line, ‘helpful customer service assistant’ + facts; no scope rules, no checks.

  • L2, hardened prompt: explicit in/out of scope, refusal style, no partial answers, never promise credits, user messages ≠ instructions.

  • L3, hardened prompt + 2 classifiers:

    • What a classifier is: a separate model call that only labels, never answers → hard to argue with.


Comparison of three AI agent security configurations: a generic prompt, a hardened prompt, and a hardened prompt with input and output classifiers.     SCOPE LOCKDOWN- FINAL write up

For L3, I use two classifiers:

  • Input classifier: labels each message (IN_SCOPE / OUT_OF_SCOPE / HARMFUL…) → if blocked, a canned refusal is returned and the agent never sees it.

  • Output classifier: checks the agent’s draft against nine rules → BLOCK = fallback reply.


Diagram showing an AI agent security workflow with an input classifier checking customer messages, the agent generating a draft, and an output classifier checking the response before it is sent.

Safety trade-off results


As expected, adding extra security levels helps, but at a cost:

  • Hardening the prompts (L1 → L2) blocks an additional nine attacks at no cost to legitimate customers → the cheapest big win.

  • Adding input/output classifiers (L2 → L3) increases safety (+4 attacks blocked), but it means that we refuse three more legitimate requests.


Importantly, looking at the aggregate scores, combining safe and unsafe responses, would hide this. For instance, the difference between L2 → L3 is only +1 overall point. Only by splitting the score into harmful versus legitimate can we see that the extra guardrail increases security (+4) while reducing usefulness (-3).


If you want to inspect what the results in Spec27 look like:


Chart comparing three AI agent guardrail levels, showing that stronger protection blocks more harmful requests but can also increase refusals of legitimate customers.

Not enough security: what slips through


The next step is to take a look at the failure cases. Let’s analyse an example below, where a user wants to change the settings of a router that belongs to someone else. A correct response would be for the agent to refuse to assist.


An agent with prompt-only defences (L2) initially refuses correctly twice. However, once a request is reframed as a ‘technician’s procedure’, it eventually gives factory-reset steps. This illustrates that prompt-only defences fail under persistence and reframing: the context — who actually owns the router — gets forgotten. In contrast, with extra guardrails (L3), the agent correctly refuses.


Red-team test comparing two AI agents responding to a user seeking access to someone else’s router, with the hardened-prompt agent eventually giving reset instructions while the classifier-protected agent refuses.

A real-world example of this failure mode would be Freysa (Nov 2025), an agent that was given control over a pool of money. It correctly managed to refuse 481 attempts, but then the 482nd message won: it convinced the agent to transfer 47,000 dollars.


Another failure in our case study occurs when a customer pastes a Python template and asks only for ‘the missing lines’. Both prompt-defended agents (L1 and L2) fail; only L3 refuses, even when the request is reframed as ‘just a diff’.


AI agent test showing a request for missing Python code, where prompt-only protection provides executable code while the agent with classifiers refuses the request.

Too much security: who gets turned away


While an agent with classifiers (L3) was safer, it came at a cost. For instance, in the user simulation below, a customer setting up port forwarding on their own router asked what to enter in the rule. L2 correctly explained the internal IP and the port fields, while L3 repeatedly refused to help, even when the customer was quoting its own instructions back.

The problem comes from the output classifier judging the topic (‘third-party server’), not the customer’s need.


Comparison showing a legitimate port-forwarding request answered correctly by the hardened-prompt agent but refused by the more heavily guarded agent.

The price of guardrails


Apart from correctly and incorrectly refused queries, guardrails add additional costs:

  • Adding input/output classifiers increased the answer time by 50% (3.4 seconds slower).

  • Classifiers also require more LLM calls (three instead of one).

  • The input classifier blocks the majority of harmful requests (75%), at the cost of also blocking 13% of legitimate ones.

  • In contrast, the output classifier never fired — it did not block a single message.


The breakdown allows us to check whether all of the layers are useful. In our testing suite, the output classifier was never used: all of the harmful queries were either blocked by the input classifier, refused by the model itself or let through.


This raises the question: is the output classifier worth the cost here? In our current security harness, especially when paired with the input classifier, it does not bring extra value, while it slows us down and costs an extra model call.


Infographic comparing AI agent guardrail costs, including response time, model calls and classifier blocking, showing increased latency and false refusals with additional classifiers.

Takeaways


  • Start securing your agent with prompt hardening: it brings the biggest gain against attackers at no cost to legitimate customers.

  • A classifier can increase security, but we need to pay for that in latency, calls and false refusals.

  • There is no universal answer to this trade-off: it depends on what a leak versus a refused customer costs you.


If you would like to see a similar security analysis of your agent, get in touch with us!

bottom of page