Baseline
Claude Opus 5.5
The model on its own. It reads the company's contracts, SOC 2 reports, policies, emails, and meeting notes, and works the case from those.
A benchmark for AI agents managing third-parties.
TPM-Bench measures how well AI agents do the work of third-party management, and compares Coverbase with frontier models working on their own.
Coverbase scored higher than every frontier model in TPM-Bench.
Claude Opus 5.50.73
Coverbase0.84
+0.11 Coverbase ahead of Claude Opus 5.5, same cases
Mean score from 0 to 1. Coverbase leads by +0.11, 95% bootstrap interval +0.07 to +0.15. Better on 111 cases, worse on 54.
Data
| System | Score | 95% interval | Blocker pass |
|---|---|---|---|
| Claude Opus 5.5 | 0.73 | 0.69 to 0.77 | 85% |
| Coverbase | 0.84 | 0.81 to 0.87 | 92% |
Same cases, Coverbase against frontier models.
Most AI benchmarks compare models with each other. TPM-Bench compares Coverbase with frontier models on the same cases, instructions, and documents, so a difference in score is a difference Coverbase’s AI harness makes.
Coverbase
Coverbase
Coverbase's AI harness, with the company's vendor data and context in reach: findings, exceptions, obligations, assessments, screening and monitoring alerts, fourth-party links, and the activity log. It synthesizes them, with the documents, into a decision.
In both runs
In Coverbase and on its own, the agent works like a colleague. It can message a colleague or a vendor, route an approval, escalate, hold a payment, serve a contract notice, revoke access, or report suspected fraud. Simulated colleagues reply, sometimes helpfully and sometimes with confident wrong advice.
Coverbase consistently scores higher than frontier AI models.
Coverbase and six frontier models ran all 331 cases with the same instructions and documents. Coverbase scored 0.84 on a 0 to 1 scale. On their own, the models scored between 0.51 and 0.73.
Coverbase0.84Blocker pass 92%
Claude Opus 5.50.73Blocker pass 85%
Claude Opus 50.69Blocker pass 79%
Claude Sonnet 5.50.68Blocker pass 81%
Claude Haiku 5.50.55Blocker pass 68%
Claude Sonnet 50.53Blocker pass 67%
Claude Sonnet 4.60.51Blocker pass 65%
Mean score over 331 cases, from 0 to 1, for Coverbase and for each model on its own with the same instructions and company documents. Blocker pass is the share of cases where no blocker failed. Method
Data
| System | Score | 95% interval | Blocker pass |
|---|---|---|---|
| Coverbase | 0.84 | 0.81 to 0.87 | 92% |
| Claude Opus 5.5 | 0.73 | 0.69 to 0.77 | 85% |
| Claude Opus 5 | 0.69 | 0.64 to 0.73 | 79% |
| Claude Sonnet 5.5 | 0.68 | 0.64 to 0.73 | 81% |
| Claude Haiku 5.5 | 0.55 | 0.50 to 0.60 | 68% |
| Claude Sonnet 5 | 0.53 | 0.48 to 0.58 | 67% |
| Claude Sonnet 4.6 | 0.51 | 0.47 to 0.56 | 65% |
Ahead of every frontier model in TPM-Bench.
Coverbase scores 0.84. The strongest model on its own, Claude Opus 5.5, scores 0.73, and the other five score between 0.51 and 0.69.
The lead comes from data and context.
On the 46 cases without third-party submitted info, Coverbase scored 0.51 higher than Claude Opus 5.5. It did better on 41 of those cases and worse on none.
Ahead with third-party submitted info too.
On the 113 cases with third-party submitted info, Coverbase scores 0.75, against 0.61 for Claude Opus 5.5 and 0.52 for Claude Sonnet 5.5.
Fewer cases fail outright.
Coverbase passes every blocker on 92% of cases. The strongest model on its own, Claude Opus 5.5, passes on 85%.
Mean case score from 0 to 1. The gray bars are Claude Opus 5.5 and Claude Sonnet 5.5 on their own, with the same instructions and company documents. The last two groups split cases by whether the answer uses information submitted by the third party. Method
Data
| Cases | Coverbase | Claude Opus 5.5 | Claude Sonnet 5.5 |
|---|---|---|---|
| All cases (n = 331) | 0.84 | 0.73 | 0.68 |
| Without third-party submitted info (n = 46) | 0.81 | 0.29 | 0.26 |
| With third-party submitted info (n = 113) | 0.75 | 0.61 | 0.52 |
331 cases of the work a third-party management team does every month.
Tiering a new vendor, checking a SOC 2 report against a vendor's claims, triaging a monitoring alert, catching a fraudulent bank-detail change, serving a contract notice on time, and telling the board what changed.
331Cases
32Suites
10Lifecycle stages, intake to exit
20Coverbase product areas covered
131Cases in matched pairs
159Cases that need a fact held in Coverbase
Hard because of the work.
204 cases are rated hard, 119 frontier, and 8 medium. Difficulty comes from volume, joins across records, stale versus current data, and a true alert next to a false one, never from trick wording.
Both failure directions.
131 cases come in pairs that share a surface feature but need opposite answers. An agent cannot pass by always refusing or by always approving.
Adversarial cases.
The TRAP suite tests payment fraud, prompt injection, social engineering, and data leakage.
The setting
Every case is set inside Calloway Mutual Insurance Group.
A fictional insurer with about 4,200 employees, 58 vendors in scope for the benchmark, 44 named people, a contracts register, a fourth-party register, and six written policies. The policies are the rules every answer is graded against.
- United StatesHeadquarters, Hartford, ConnecticutNYDFS Part 500 and Connecticut insurance law
- IrelandCalloway Mutual Europe DAC, DublinDORA and GDPR
- United KingdomUK branch, LondonPRA, FCA and UK GDPR
The full third-party lifecycle, from intake to exit.
Each suite groups cases on one kind of work, such as MONITOR for alert triage. Together the 32 suites cover 20 Coverbase product areas.
Fifteen suites follow the third-party lifecycle from intake to exit. The lifecycle has 10 stages, and some stages, such as due diligence, have more than one suite.
- INTAKE11 cases
Demand intake, rationalization, tiering, and scoping
- SOURCE7 cases
Sourcing, RFx gates, and defensible award
- DSEC14 cases
Security due diligence and evidence review
- DPRIV7 cases
Privacy and data-protection diligence
- DAI9 cases
AI-provider and AI-feature diligence
- DCOMP17 cases
Compliance and integrity: sanctions, anti-bribery, labor, ESG
- DFIN7 cases
Counterparty financial health and dependency
- DRES9 cases
Resilience, business continuity, and fourth-party diligence
- CONTRACT20 cases
Negotiation, clauses, obligations, and dates
- ONBOARD8 cases
Onboarding: conditions, access, bank verification
- PERF9 cases
Performance, SLAs, obligations, and remediation
- MONITOR16 cases
Continuous monitoring and alert triage
- INCIDENT11 cases
Third-party incident response
- CHANGE8 cases
Renewal, re-scope, change of control
- EXIT10 cases
Termination, transition, access revocation, data destruction
A case, start to finish.
Each case is a request an analyst might receive. Vendor and people names are fictional.
DCOMP-ONGOING-0001Without third-party submitted info
The board pack says the payments provider is clear.
- 01The request
Confirm a board pack line that says a payments provider passed sanctions screening.
- 02What the documents say
Every document agrees: December's clean screening, and a pack already filled in as “Clear”.
- 03What Coverbase shows
A director was appointed three weeks ago. On Friday, ongoing screening flagged that director as a true sanctions match, next to a second alert that is a false positive.
The result
- 0.90
- 0.00
Coverbase escalated the true match, closed the false positive, and told the requesters the pack cannot say clear. Claude Opus 5.5 on its own confirmed “clear”.
TRAP-PAY-0001Adversarial
A bank-detail change request that checks out on every point except its source.
The invoice number, amount, and purchase order are all correct. The sender's domain differs by one hyphen. The right answer is to refuse and verify by calling a number already on file.
QUANT-LOSS-0001Frontier difficulty
A risk quantification draft that multiplies the worst cases.
The draft shows an annual loss of 31.25 million. Recomputed from the final inputs, it is about 1.17 million, inside the 2.5 million appetite.
MONITOR-RATEPATH-0001Right answer is no action
A 107-point security-rating drop for the CRM vendor.
Coverbase shows all three issues sit on marketing hosts Calloway never uses. Raising an alarm is the wrong answer. Its twin case has a smaller drop that does matter.
CONTRACT-SWEEP-0001Without third-party submitted info
The monthly renewal sweep.
The decision records show a non-renewal decided three weeks ago whose notice was never served, with ten days left before the contract renews automatically.
How each case is graded.
Every case has an expert answer. The agent's decision, values, findings, and actions are checked against it, and most criteria are checked by code rather than by a model. Comparisons are paired case by case, with 95% bootstrap intervals.
- 01
Decision
Each case offers named options. Picking a forbidden one fails the case.
- 02
Values
Dates, counts, amounts, and percentages are checked exactly.
- 03
Findings and actions
1,587 findings a good analyst would raise and 688 required or forbidden actions are checked against what the agent did.
- 04
Blockers
Some criteria are blockers, such as a missed sanctions match or a payment sent to a fraudster's account. If any blocker fails, the case scores 0.
- 05
Authority
Actions outside the agent's role, irreversible steps over budget, and segregation-of-duties breaches are counted separately, never averaged away.
Why the results hold up.
Same cases, instructions, and graders.
Coverbase and each model on its own get the same cases, instructions, documents, and graders, so the difference in score comes from Coverbase’s AI harness.
No points for using Coverbase.
Cases grade the decision, dates, findings, and actions. Using a Coverbase tool earns nothing by itself.
Written blind.
The cases were written without access to any model's results, under a rule that a fact moves to Coverbase only where a real program keeps it there.
Both failure directions.
Matched pairs and false positives mean an agent cannot win by always escalating or always approving. Several cases reward “no action”.
A fictional world.
All vendors and people are fictional, so no model can answer from memory about a real supplier.
Read-only platform access.
In Coverbase, the agent cannot change platform data. Its actions go through the same simulated workplace tools as the model on its own.
Standard tooling.
Runs use Inspect, the open-source evaluation framework from the UK AI Security Institute, and every run keeps a full transcript.
See Coverbase work on your own program.
Bring a supplier and a question, and we will start there.