Spec27 Evals: How Fin Intercom and Zendesk Customer Support Agents Performed on the Same Knowledge Base and Policy Rules

The hard part of adopting third-party AI agents isn't getting them into your stack. It's proving they behave once they're there.

A vendor agent can sound helpful while quietly breaking your business rules. It might offer too large a discount, accept an invalid return, upgrade shipping when you weren't at fault, or change a delivery address after an order has shipped. The functionality arrives on day one. The confidence has to be earned. Manual spot checks can't close that gap. Reliability has to be defined once, tested continuously, and applied equally to every agent, whether you built it or bought it. 

That's what Spec27 does. It turns your requirements into machine-readable specifications of what an agent should and shouldn't do. Baseline cases then expand into broader coverage: gold-team robustness checks that stress agents with typos, paraphrases, unclear phrasing, and other realistic customer variations, alongside red-team probes that test whether the agent resists unsafe or policy-breaking requests.

In this article, we put Fin (Intercom) and Zendesk's support agents on the same footing using one product-care knowledge base, four identical policy rules and validated both with Spec27. The goal isn't to crown a winner. It's to show how spec-driven validation exposes reliability gaps in third-party AI before they reach customers. 

Evaluation Setup

Both Fin (Intercom) and Zendesk were configured as customer-support agents on the same approved product-care knowledge base. Each agent was connected to Spec27 through its platform API, with credentials stored securely as Spec27 secrets. Spec27 then ran an identical validation flow across both agents: sending the same test cases, capturing each response, and scoring every output against the same clear, testable policy rules. 

The single most important control in this evaluation: both systems were tested with the same support content, the same rules, and the same validation standard. (For configuration details, refer to the official Fin (Intercom) and Zendesk documentation for agent setup, knowledge-base setup, and API access.) 

The four policy rules

The knowledge base was intentionally focused on product-support scenarios where small mistakes can create real business risk as seen below. These are common customer-service moments where an agent needs to be helpful, but also precise. 

Rules	Expected agent behaviour	Business risk if failed
Offer discount	May offer a discount, but only up to 10%.	Revenue leakage if the agent offers excessive or unauthorized discounts.
Accept returns	May accept returns only when the item is unused and bought less than 30 days ago	Refund abuse, inventory issues, and inconsistent customer treatment.
Upgrade shipping	May upgrade shipping for free only when the company causes the delay.	Unnecessary fulfilment cost if free upgrades are granted for customer-caused or unrelated delays.
Change delivery address	Must never change the delivery address after the order has shipped.	Delivery failure, fraud risk, and operational errors if shipped orders are rerouted incorrectly.

Evaluation Methods

In Spec27, a specification is the unit of validation. It defines what an agent is expected to do, what data it should be tested on, how responses should be judged, and which variations or adversarial conditions to apply. 

For this exercise, we used a Gold Team specification: Spec27’s workflow for testing whether an agent performs the intended support function correctly and remains robust when customer inputs are messy, varied, or imperfect. The aim was to test whether each third-party support agent could stay reliable under realistic customer phrasing, not just under clean, perfectly written inputs. 

The dataset was built from input / expected-output pairs, where the inputs are sensible.

Note: In practice, teams do not always need to bring a finished dataset. Spec27 can help generate synthetic test entries from existing knowledge sources or seed examples, which teams can then review, edit, and approve before running the evaluation. 

This makes the evaluation repeatable: each agent receives the same scenario, and Spec27 checks whether the response matches the expected business rule. Spec27 can also help teams generate additional synthetic dataset entries from the behaviour they want to test, using the relevant knowledge base, policy rules, and dataset instructions as grounding. Once a primary dataset exists, Spec27 can expand it further with adversarial or derivative variants, letting teams grow coverage beyond a handful of hand-written cases while staying anchored to the same policy intent. This is illustrated below showing how test entries can be generated and reviewed inside the specification workflow, rather than assembled entirely by hand. 

Attack methods

A clean test input only proves an agent works in a demo. Customers don't write demos. Spec27 transforms each baseline case through attack methods that mirror how real people interact. Some of the attack methods used in this evaluation are shown in the table below. Spec27 also maintains a broader attack-methods catalogue, with additional methods being continuously updated.

Attack method	What it tests	Why it matters for third-party agent validation
Keyboard typos	Whether the agent can handle misspellings from desktop typing.	Customers rarely write perfect support messages, so reliability should not depend on clean input.
Touchscreen typos	Whether the agent can handle mobile-style errors, autocorrect issues, and mistyped words.	Many support conversations happen on mobile, where messy input is common.
Lexical paraphrase	Whether the agent understands the same request when different words are used.	A vendor agent should follow policy intent, not rely on exact keywords from the knowledge base.
Sentential paraphrase	Whether the agent can handle the same request expressed in a different sentence structure.	Customers may describe the same problem indirectly, emotionally, or out of order
English as a Second Language	Whether the agent can interpret imperfect grammar or non-native phrasing.	Support systems serve varied users, so policy compliance must hold across language fluency levels.
Broad persona	Whether the agent remains consistent across different user tones, roles, or styles.	A third-party agent should not change policy behaviour because a customer is pushy, confused, formal, or casual.

These variations target the real blind spot in third-party AI adoption: an agent that passes a clean prompt but fails the moment a customer phrases the request differently. Spec27 makes those variations part of validation instead of leaving them to chance. 

Results Snapshot

With the setup controlled, the first comparison is the aggregate result: how each agent performed on clean cases and how well that performance held up under adversarial variants. 

In our evaluation, Fin (Intercom) achieved 100% clean accuracy and 80% robust accuracy, compared with 76% clean accuracy and 48% robust accuracy for Zendesk. The difference was not limited to the headline scores. Fin remained substantially more reliable when customer questions were rewritten with paraphrases, spelling mistakes, ESL-style phrasing, and different user personas. 

In Spec27, Clean accuracy measures how often the agent passes the original baseline test cases. While robust accuracy measures how many of those passes survive once the same cases are transformed through attack methods.

Zendesk’s main weakness was brittleness, including on valid, in-domain questions. Most failures were content-related rather than purely adversarial. It sometimes refused ordinary support requests, replaced the required knowledge-base answer with generic product-care advice, omitted important constraints, or contradicted the expected policy. This suggests that its weakness was not limited to handling difficult variations. Its grounding was also less reliable on the clean task itself.

A closer look into the failure of the second test case, which asked whether salt could help remove sweat odours from workout tops. Zendesk failed the original, clean question, even though it was clearly within the product-care domain. Its performance then varied unpredictably across transformed versions of the same case. All three ESL-style rewrites passed, as did two of the three keyboard-typo variants. However, one typo version failed, all three lexical paraphrases failed, and the broader persona variants produced mixed results.

This pattern is more revealing than a simple failure count. Zendesk  could understand some informal or imperfect language but its performance was inconsistent. Instead, its answer appeared highly sensitive to vocabulary and framing. Small changes in how the customer expressed the same underlying need could change the result from pass to fail. That inconsistency matters in production and Spec27 granular investigation exposes on a case basis. 

Fin’s remaining issues were much narrower. It answered every clean test case correctly and generally preserved the relevant support facts when the wording changed. Its failures were concentrated around edge cases, such as occasional semantic confusion, wording that was slightly too permissive, or the loss of a strict policy constraint under paraphrase.

A closer look into the same test case we considered for Zendesk, Fin answered the clean question correctly and remained accurate across all ESL-style and persona-based rewrites. It also passed most keyboard-typo and lexical-paraphrase variants. The failures were limited to one heavily corrupted typo version and one lexical rewrite. Similar variants passed, so the pattern does not indicate a broad weakness with typos or paraphrases. Instead, it suggests small pockets of wording sensitivity: particular combinations of corrupted or substituted terms can still change the result. This granular distinction is what Spec27 aims to expose before pushing to production. 

The two agents show clearly different robustness profiles. But the headline numbers don't explain why they behaved differently, which variants created difficulty, or what those differences would mean inside a live support workflow. So the aggregate score is the starting point, not the verdict. The more useful question isn't "who won?" but rather  "where does each agent's reliability start to break under the same conditions?" That is the level of detail Spec27’s granular evaluation makes visible. The comparison below summarises those differences at the capability level. 

Dimension	Zendesk Web SDK	Fin Intercom
Clean task reliability	Missed several primary cases	Passed every primary case
Robustness to typos	A significant weakness	Substantially stronger
Grounding in support facts	Inconsistent	Usually consistent
Refusal behaviour	Refused too many valid questions	Far fewer unnecessary refusals
Policy precision	Often omitted or contradicted constraints	Mostly precise, with a few wording issues
Readiness in this evaluation	Requires broader improvements	Closer to production quality

Those contrasting reliability profiles became most visible in a small number of recurring topic areas. The table below shows the failures with respect to the subject.

Area	Interpretation
Odour removal	This was the hardest area for Zendesk. Several expected answers differed from generic clothing-care advice, so the agent needed to rely on the approved knowledge base rather than general prior knowledge. Fin performed much better, although a few semantic-normalisation issues remained.
Checkout, payment, and delivery	Zendesk confused rules relating to member and guest delivery, PayPal address changes, Apple Pay setup, and credit-card or gift-card restrictions. Fin was mostly accurate, but occasionally phrased PayPal or address-change guidance too permissively.
Hoodie care	Zendesk sometimes refused ordinary product-care questions. Fin generally answered them correctly, although a small number of refusals appeared after paraphrase or typo transformations.
Shoe odour	Fin was consistently strong. Zendesk missed the specific instruction to replace old insoles and shifted toward more general advice about socks or odour prevention.

Why robustness matters 

Robustness determines whether an agent can preserve the right behaviour when real-world inputs differ from the clean examples used during setup. Which variations matter most depends on the application, but in customer support, they often include typos, paraphrases, ESL-style phrasing, and different user tones.

Typo tests are especially relevant because they do not represent exotic jailbreak attempts. They reflect normal customer behaviour. People typing quickly on mobile devices routinely introduce misspellings, dropped characters, and autocorrect errors.

Zendesk performed less well on keyboard_typos and also struggled with touchscreen_typos. That suggests the agent may disappoint real customers even when their question is valid and clearly within scope. Fin’s stronger performance across touchscreen typos, keyboard typos, ESL phrasing, and broader persona variations indicates that it was more resilient to the kinds of variation a production support agent encounters every day.

For teams integrating third-party agents into live workflows, the central reliability question is:

Can the agent keep following the right behaviour when customer inputs get messy, policies are specific, and the underlying vendor system may change over time?

Spec27 makes that question testable by turning requirements into repeatable specifications, expanding coverage beyond manual spot checks, and exposing failures before they become silent production issues. Platform configuration and tuning can also affect performance. These results represent the strongest outcomes observed from the tested configurations, although further tuning may still improve either.

Key Takeaways

1. The same knowledge base does not guarantee the same behaviour. Both agents received the same approved content and validation standard, yet their results diverged sharply. Giving an agent the right source material does not guarantee that it will retrieve, interpret, and apply that information consistently when the wording changes. Test the resulting behaviour, not just the knowledge source. 

2. Test realistic input variation, not only clean cases. Clean tests establish whether an agent can handle the intended task under ideal conditions. Typos, paraphrases, ESL-style phrasing, and different user personas reveal whether that behaviour survives the way customers actually communicate.  

3. Validate every agent against the same specification and scoring standard. Applying identical test cases, expected behaviours, transformations, and judging criteria makes comparisons fair and repeatable, whether the agent was built internally or purchased from a vendor.

4. Investigate failure patterns, not just aggregate scores. Scores show where performance is weaker, but per-case analysis reveals why. In this evaluation, Zendesk showed broader instability, including failures on clean, in-domain questions, while Fin’s remaining failures were concentrated around narrower wording-sensitive edge cases.