You shipped an agent. How good is it?
Verity builds verified evaluation suites for AI agents. Each one is tailored to your domain: 1,000 to 5,000 test cases, granular checks on every case, and a simulated environment to run them in.
Payroll agent evaluation A worked example on 2,400 synthetic cases
Demo · all data synthetic
An AI payroll agent reads monthly change emails from client companies and produces payroll updates. The suite covers approval state, change type, company size, single emails and long threads, Excel, PDF and scanned Form 101 inputs, missing details and injected errors.
A case passes only when every one of its checks passes. The first run below is from the simulated environment.
| Category | Cases | Checks | Pass rate | |
|---|---|---|---|---|
| Approvals and sign-off | 420 | 5,265 | 73% | |
| Payroll changes | 520 | 6,289 | 77% | |
| Leave and absences | 380 | 4,583 | 79% | |
| Missing information | 310 | 3,981 | 71% | |
| Attachments and formats | 290 | 3,667 | 73% | |
| Scale and volume | 180 | 2,729 | 69% | |
| Error injection | 200 | 2,703 | 68% | |
| Edge cases and compliance | 100 | 1,237 | 75% | |
| Agent score | 2,400 | 30,454 | 74 / 100 |
Target for production: 90. Synthetic data, simulated run.
The problem Agents are easy to build and hard to trust.
Most agents reach production without a baseline. Teams cannot say how often the agent is right, where it fails, or what changed between versions.
No baseline
A few manual prompts and a demo that worked once. Without a regression suite, every model or prompt change is a guess.
No domain ground truth
Generic benchmarks do not know your forms, your connectors or the inputs your customers actually send.
No safe place to run it
Agents touch email, files, APIs and records. Testing them properly requires mocked tools and realistic synthetic data.
How it works From a 30-minute call to a complete suite.
Scope
A 30-minute call to map your domain, stack, use cases, the shape of your data and the connectors the agent uses.
Generate
1,000 to 5,000 cases built across your scenario dimensions. Each has a realistic input, an expected output and many granular checks, all verified.
Simulate
Mocked tools and connectors, synthetic inboxes, files and records, so every case runs end to end and reproducibly.
Deliverables Precise coverage and a clear path to fix what fails.
Dataset
Realistic inputs for your domain: emails, attachments, scanned forms, partial data and scale variants.
Granular checks
Per-case assertions such as "extracts the employee ID" or "does not approve without sign-off".
Simulated environment
Mock tools, connectors and synthetic data around every case. Safe, fast and repeatable.
Score and fix loop
An agent score, a breakdown by category, and a fix prompt for your coding agent to iterate against.
Engagements Own the dataset, or have us run it.
Both start with the same scoping call. Pricing is scoped per project.
Evaluation dataset
A verified suite and simulated environment, delivered once. Run it in your own CI.
- 1,000 to 5,000 tailored cases
- Expected outputs and checks per case
- Mock tools, connectors and data
- Coverage map across dimensions
Managed evaluation runs
We run the suite on each new version of your agent and extend it as your product evolves.
- Everything in the one-time dataset
- A run on every release, with history
- Category and failing-case reports
- Fix prompts for iterative improvement
Contact Book a 30-minute scoping call.
Describe what your agent does and we will return a coverage plan.