Spec27 Blog

Four new speakers join AI Strategy & Practice in London, bringing experience across engineering, responsible AI, governance and business strategy.

Safe Intelligence is bringing AI practitioners and leaders together in London on 21 October, with a limited number of tickets available at 50% off.

A red-team score is a starting point. Here are the questions that help turn an AI agent test result into action.

What changes when an AI agent that passes its original tests is challenged adversarially? This Spec27 run tested 20 primary behaviours across four attack methods and 160 adversarial variants, showing why clean performance, robustness, and attack-method coverage need to be interpreted separately.

SPEC27 UPDATE · 15 SEPTEMBER 2026 An anonymised Spec27 scan shows how an AI agent that performed strongly on clean inputs became substantially less reliable under adversarial testing. A public-facing AI agent passed 90% of its clean tests. We then tested the same behaviours using a range of adversarial methods. Under the scan’s robust-accuracy measure—the percentage of tests for which none of the attacks caused a failure—the score fell to 20%. In other words, only one in five

The moment your agent goes public, you stop choosing your users. Some of those users will be clear and cooperative. Some will be vague, confidently wrong, in a hurry or persistently looking for an exception. A few will be deliberately adversarial. Your agentic system receives all of those interactions through the same interface. Most teams begin by evaluating whether an agent does what it is supposed to do. That is the right place to start, and it's where most evaluation stop

A useful comparison requires more than reviewing a few polished answers. Both agents need to face the same scenarios, business rules and scoring criteria. Putting two support agents through the same tests We evaluated Intercom Fin and Zendesk’s AI support agent using the same customer-service scenarios. The tests examined whether each agent could continue to follow the relevant business rules when customer requests became messy, varied or incomplete. This matters because a co

Public-facing AI agents need to be tested against more than expected questions and cooperative users. Happy-path testing is only the beginning A lot of agent testing starts with straightforward checks: Can the agent answer the question? Can it use the correct knowledge base? Can it complete the requested task? These checks are useful, but they do not reveal how the agent will behave under the unpredictable conditions created by real users. What else should teams test? Before

Agent readiness depends on more than producing a good response. Teams also need to understand how an agent behaves when inputs become messy, policies are challenged and external tools are involved. What does “ready for real users” mean? Ahead of AI Enterprise Conference NYC, we were thinking about one of the practical questions behind AI agent deployment: How do you know an agent is ready to put in front of real users? A successful answer to one carefully written prompt is no

Choosing a route into AI support is only the beginning. This article explains how robustness changes across AI-agent-first platforms, service platforms with AI, CRM/contact-centre ecosystems, and in-house builds -- and why every route needs repeatable testing across behaviour, policy, tools, workflows, safety, and change.

Intercom Fin and Zendesk were tested against the same knowledge base, policy rules and realistic customer-input variations. Here’s where their reliability held—and where it began to break.

Model upgrades are not simple component swaps. They can change how AI agents escalate, refuse, call tools, format outputs, and respond to edge cases. This post explains where drift hides and how teams can re-validate agents before switching models.

AI support agents are moving from novelty to infrastructure. This guide maps the third-party vendor market, including what these agents do, why adoption is accelerating, where they perform best, and what risks buyers should consider before deployment.

Sentiment classification looks like a simple task: take a financial statement and return a positive or negative label. It is used in research and production for financial-news monitoring, reporting workflows and downstream analysis. Clean accuracy, however, tells us only how a model behaves on the wording it was given. If the same financial relationship is expressed with different words, a different sentence structure or an added concessive clause, the classification should r

From Test Prompts to Validation Specs: Uplevelling AI Agent Testing

In a hurry? There's a preloaded example you can run in CI today, jump to how you can try this yourself.

On 11 June, we joined the Google DeepMind Startup Sessions at the Ministry of Sound in London – a morning of talks, demos, and live pitches bringing together AI startups and the Google DeepMind community. The lineup included a talk by Dr Raia Hadsell, a recap of Google’s annual developer conference, and a pitch showcase featuring around ten selected startups. We were glad to be one of them, presenting Spec27, our product for validating how AI agents behave before deployment a

At AI Engineer Europe 2026, Spec27 Principal Engineer Michal Cichra explored why engineering decisions need to be captured clearly for both human teams and AI systems. Watch the talk and learn how BDD, ADRs, and PRDs connect to specification-driven validation.

Third-party AI agents can help teams deploy faster, but they also create a validation challenge: the buyer owns the operational risk without full visibility into the internals. This guide explains how to evaluate vendor-built AI agents from the outside using specifications, realistic test cases, pass criteria, re-validation, and evidence.

How many agents do we have? And when do they start hiring their own agents? AI Agents are taking the enterprise by storm (at least on executive roadmaps!) and many leading AI tech companies are rolling out frameworks and platforms for AI Agents. Navigating through what works and what doesn't can be hard to keep track of. There's huge value in a lot of the deployments, but they also each present their own challenges. As the number of agents increases the interactions between t