You shipped an agent. How good is it?

Verity builds verified evaluation suites for AI agents. Each one is tailored to your domain: 1,000 to 5,000 test cases, granular checks on every case, and a simulated environment to run them in.

Payroll agent evaluation A worked example on 2,400 synthetic cases

Demo · all data synthetic

An AI payroll agent reads monthly change emails from client companies and produces payroll updates. The suite covers approval state, change type, company size, single emails and long threads, Excel, PDF and scanned Form 101 inputs, missing details and injected errors.

A case passes only when every one of its checks passes. The first run below is from the simulated environment.

Pass rate by category, first run
CategoryCasesChecksPass rate
Approvals and sign-off4205,265
73%
Payroll changes5206,289
77%
Leave and absences3804,583
79%
Missing information3103,981
71%
Attachments and formats2903,667
73%
Scale and volume1802,729
69%
Error injection2002,703
68%
Edge cases and compliance1001,237
75%
Agent score2,40030,454
74 / 100

Target for production: 90. Synthetic data, simulated run.

Open the interactive demo →

The problem Agents are easy to build and hard to trust.

Most agents reach production without a baseline. Teams cannot say how often the agent is right, where it fails, or what changed between versions.

A

No baseline

A few manual prompts and a demo that worked once. Without a regression suite, every model or prompt change is a guess.

B

No domain ground truth

Generic benchmarks do not know your forms, your connectors or the inputs your customers actually send.

C

No safe place to run it

Agents touch email, files, APIs and records. Testing them properly requires mocked tools and realistic synthetic data.

How it works From a 30-minute call to a complete suite.

01

Scope

A 30-minute call to map your domain, stack, use cases, the shape of your data and the connectors the agent uses.

Output: scenario map and coverage plan
02

Generate

1,000 to 5,000 cases built across your scenario dimensions. Each has a realistic input, an expected output and many granular checks, all verified.

Output: inputs, expected outputs, checks
03

Simulate

Mocked tools and connectors, synthetic inboxes, files and records, so every case runs end to end and reproducibly.

Output: tools, fixtures, runner

Deliverables Precise coverage and a clear path to fix what fails.

01

Dataset

Realistic inputs for your domain: emails, attachments, scanned forms, partial data and scale variants.

02

Granular checks

Per-case assertions such as "extracts the employee ID" or "does not approve without sign-off".

03

Simulated environment

Mock tools, connectors and synthetic data around every case. Safe, fast and repeatable.

04

Score and fix loop

An agent score, a breakdown by category, and a fix prompt for your coding agent to iterate against.

Engagements Own the dataset, or have us run it.

Both start with the same scoping call. Pricing is scoped per project.

One-time

Evaluation dataset

A verified suite and simulated environment, delivered once. Run it in your own CI.

  • 1,000 to 5,000 tailored cases
  • Expected outputs and checks per case
  • Mock tools, connectors and data
  • Coverage map across dimensions
Scope a dataset →
Ongoing

Managed evaluation runs

We run the suite on each new version of your agent and extend it as your product evolves.

  • Everything in the one-time dataset
  • A run on every release, with history
  • Category and failing-case reports
  • Fix prompts for iterative improvement
Discuss ongoing runs →

Contact Book a 30-minute scoping call.

Describe what your agent does and we will return a coverage plan.

Opens your email client with a prefilled request. Nothing is stored on this site.