top of page

Announcing Datafactory Beta: Build Agent Test Datasets from Benchmark Scenarios

Writer: Jovanca Garnadi
Jovanca Garnadi
3 hours ago
2 min read

Building reliable AI agents starts with deciding what to test: the tasks an agent should complete, the edge cases it should handle and the requests it should refuse or escalate.


Today, we’re launching Datafactory in beta. Datafactory lets you explore benchmark scenarios and generate structured test cases tailored to your organisation’s products and policies. We’re starting with retail e-commerce, with additional benchmarks planned.


Datafactory retail customer support benchmark showing the test generation workflow and a return scenario with six possible outcomes.
Explore the retail customer support benchmark and the possible outcomes for each scenario.

Turning scenarios into useful tests


Test cases can come from production conversations, support tickets, Q&A forums and product requirements. For new agents, they also need to cover behaviours that have little or no usage history. Translating those sources into realistic scenarios, supporting data and clear success criteria takes substantial manual work.


The preparation becomes more involved when agents take actions. An end-to-end test needs the right starting conditions: customer and order records, product data, access to the relevant tools and clear criteria for the resulting actions.


A return test, for example, needs an order, the items being returned, their delivery date and the applicable returns policy. Depending on those facts, success could mean processing the return or explaining why it cannot be accepted.


Benchmark scenarios provide a useful starting point. Teams can adapt them to their own business and extend them with requirements, policies and customer interactions specific to their agents.


What you can build with Datafactory


Datafactory scenarios for customer identification, delivery tracking, cancellations, order changes and returns, with associated company policies and outcomes.
Select scenarios relevant to your agent and inspect the associated policies and outcomes.

With Datafactory, you can:

  • Explore common scenarios represented in a benchmark and choose those relevant to your agent.

  • Generate structured test cases with user goals, initial messages, supporting data and expected outcomes.

  • Customise cases for your organisation using your product catalogue, business policies and agent use cases.


You select the scenarios, provide information about your organisation, and review the company profile and policies before generating your test set.


The resulting cases bring together:

  • A scenario and user goal.

  • An initial user message and instructions for the user simulator, defining what the simulated customer knows and is trying to achieve.

  • Supporting records, such as products and variants, customers and orders.

  • Relevant policies and conversation requirements, such as information the agent must collect before taking an action.

  • Expected outcomes and success criteria, including required changes to records where applicable.


This lets you build cases for different outcomes within a scenario, including requests that should be declined or escalated under your policies.


Datafactory exchange scenario linking user information, supporting records, policy conditions and expected outcomes, including authorisation and refusal cases.
Inspect how an exchange scenario connects user information, supporting records, conditions and expected outcomes.

Download your test set as JSON, or export the test cases to an Excel spreadsheet for review.


Try the beta


Datafactory launches with the Retail customer support benchmark, covering scenarios such as order tracking, cancellations, returns, exchanges and delivery problems. You can inspect example cases for SuperBox, a fictional retailer, before creating cases for your own organisation.


You can browse the benchmark without an account. Creating your own test set requires a Spec27 account.


Explore the retail benchmark, select Create test set, and customise the scenarios for your organisation. Review your policies, generate your cases and download the result.


bottom of page