Evaluation Suites

Accuracy is a number.
We measure it every night.

A demo score tells you how a system did on someone else’s examples. We build a test suite from your own cases, labelled by your experts, and score every system against it every night and before every change. Your risk team gets evidence, not an opinion.

Book an evaluation review See a nightly run

01 / Premise

“91% accurate” means nothing.
Which 9% fails, and does it matter?

Vendors quote one headline number from a sample they chose. In production the errors are not spread evenly: they cluster in the cases that matter most, and nobody is measuring them by type or by severity.

The demo

The vendor quoted one number, measured on examples they chose.

In production

In production, the errors clustered in the cases that matter most.

Model risk review

Model risk still asks how you know it works, and a headline number is not an answer.

Model risk review — Document extraction

Validation year · 2026

Vendor demoApr

Go-liveJun

Model updateAug

ValidationNov

RenewalJan

Model risk requests · draft

Accuracy on our own cases, by case typeRequested

Error severity, not just error rateRequested

Evidence the model update was re-testedRequested

The agreed accuracy floor and who set itRequested

What happens when the score dropsRequested

5 of 5 open · validation due in November

02 / Evaluation review

In two weeks, we test every AI system you run or are buying against your own cases, and rank where the evidence is thinnest.

Test first

Low

Medium

High

Gap in evidence ↑

2 wks46810 wks

Effort to test →

Severity

Refusals

Drift

Citations

Edge cases

Change gate

Credit memo assistant

Credit

Client report drafter

Client service

Research notes

Research

Complaint drafter

Customer ops

Sanctions screener

Compliance

KYC file builder

Onboarding

Evidence gap, 0.1 to 1.2

0.11.2

Shortlisted

1.2

0.5

0.2

0.3

0.4

0.1

0.2

0.1

0.5

0.2

0.1

0.4

0.3

0.2

0.1

0.9

0.2

0.3

0.1

0.5

0.7

0.2

0.3

0.8

0.1

0.3

0.6

0.4

0.2

0.1

0.3

0.1

Memo assistant · severity

Report drafter · citations

Complaint drafter · edge cases

KYC builder · severity

Research notes · edge cases

Test first · your systemsDay 10

Credit memo assistant

Severity-weighted suite, 1,200 cases

High

6 wks

Client report drafter

Citation check on every figure

High

5 wks

Complaint drafter

Add 300 edge cases from history

Medium

4 wks

KYC file builder

Severity suite, from test 01

Medium

3 wks

Research notes

Add unsupported-claim cases

Medium

8 wks

Five suites, ranked by evidence gap

Suite 04 reuses suite 01

Days 1–6

A sample of your historic cases drawn for every system

→ case sample

Days 7–9

Each system scored on accuracy, severity, citations, refusals and edge cases

→ scored matrix

Day 10

A scored report and a suite plan for every system

→ suite plan

Illustrative estate of six systems. Real reviews score each one against your own cases, with every failure attached.

First step

Start with a two-week evaluation review.

Book an audit

The scored report is yours, whatever you decide

03 / Building the suite

From your history
to a nightly score.

We draw cases from your own history, your experts label the right answers, and the suite runs every night from then on. Every prompt, model or rule change is scored against it before release.

Suite log — Credit memo assistant · Q3

020406080100

Case sampling

Days 0–14

Expert labelling

Days 8–30

Severity scoring

Days 14–42

Citation checks

Days 36–64

Nightly runs and floor

Days 56–86

Evaluation and handover

Days 70–100

Day 0

Log

D03Baseline: 11 case types untested, one headline score of 91%

D14Sample closed: 1,200 cases drawn from 18 months

D30Labelling complete with four credit analysts

D42Severity weights agreed with the risk owner

D64Every figure checked for a source citation

D86Nightly runs live · floor set at 92%

D100Change gate live · suite handed to your team

Case types untested

Against the baseline signed on day 3

Cases labelled

Live

0 of 3

Suite checklist · Q3

Severity scoringPlanned

Citation checksPlanned

Nightly runs and floorPlanned

Every figure comes from the platform’s own logs, scored again every night.

Traceability

Click any score.
See what is behind it.

Every score in a nightly run can be opened: what was measured, what it was compared with, the cases behind it and who reviewed it. Your model risk team reads the same run we do.

Evaluation run — Credit memo assistant · night 22

Suite v2.7 · 1,240 cases22 Sep · 02:00

Auto-playing · click any score

Behind the score

Overall accuracy94.6%

Measured1,173 of 1,240 cases right, weighted by severityRun S-0922 · 02:00

Compared withFloor 92% · last night 94.4%Floor agreed 3 Jul

EvidenceEvery case, answer and expected answer stored with the runRun S-0922 · case list

ReviewedModel owner22 Sep 2026 · 08:30

Evidenced · 4 of 4Same run in the model risk file

Illustrative run for one system. Names, references and figures are examples.

04 / Across systems

Built once. Scored across every system.
Each system gets its first score faster than the last.

The scoring, severity weights and change gate built for the first system go into a shared library. The next system adds its own cases and gets its first nightly score in days rather than weeks.

Illustrative five-system estate. Timings depend on how many cases each system needs labelled.

Playbooks

Four systems we run,
written up from delivery.

What we measure first, what we build and how each one is checked. Read them before the audit, or bring them to it.

All nine playbooks →

PlaybookMandate GuardrailsInvestment guidelines turned into approved, versioned rules, tested before and after each trade, with every breach and override on record.Read the playbook →PlaybookClient Reporting FlowClient reports built from reconciled books, each figure tied out and each commentary claim checked against attribution before release.Read the playbook →PlaybookResearch IntakeOvernight house-format notes from filings, transcripts and broker research. Every figure and quotation carries a source.Read the playbook →PlaybookInvestor ReportingLetters, statements and DDQ answers drafted from your books, with every figure traced to source before release.Read the playbook →

05 / Value

Errors caught you can bridge,
line by line.

Each addition to the suite catches errors that used to reach a reviewer or a client. The model risk file shows which addition caught what.

Suite report — Credit memo assistant · month 6

Errors caught before a person sees them

% · axis starts at 92

94.2% →94.2%+0.0 pts

96

95

94

93

92

94.2%

+1.2 pts

+0.5 pts

+0.2 pts

+0.3 pts

96.4%

Headline suite

month 0

Severity weights

severe errors first

Edge cases

300 from history

Citation checks

every figure

Change gate

before every release

Caught

measured at month 6

+2.2 pts on 41,000 cases a quarter≈0fewer errors reaching reviewers a quarter

Illustrative bridge for one system. Each line is measured on the same 1,240-case suite.

Related work

Measured against the week
before we arrived.

Client names are withheld under mandate. Each engagement reports the baseline instrumented at the diagnostic and the figure at the third monthly review.

Financial Services · Evaluation suitesCredit memo preparation across a $6.4B bookEvery figure is extracted with a citation back to the page it came from. The suite scores citations nightly, and the audit trail satisfied the bank’s model risk function without amendment.Time per memo6 hrs → 40 minMemos prepared1,240Field-level citations100%Read the case study →Asset Management · Evaluation suitesOvernight research intake for an equity teamAnalysts started each day reading filings and broker notes. House-format notes are now ready overnight, and the suite holds back any statement without a source.Morning reading2 hrs → 25 minSources read each night300+Unsourced statements publishedNoneRead the case study →

All work →

Not built yet?

Start with a two-week workflows & processes audit.

We map what your teams do today, measure the baseline and rank what is worth building, before anything goes into production.

Book an audit

The process map is yours, whether we build or not

06 / Engagement

Three ways in.
Each one earns the next.

Engagement 012 weeks · fixed fee

The evaluation review

We test any AI system you run or are buying, ours or anyone’s, against a sample of your own cases. You get a scored report showing where it fails and how badly.

Covers

Any system, any vendor

Access

Read-only

Output

Scored report

Book an audit

Engagement 026 weeks · per system

The suite build
for production

We build a suite from your own history, labelled by your experts, weighted by severity, and agree the floor each system is held to.

Runs in

Your cloud

Cases

Your history

Floor

Agreed per system

Book a call

Engagement 03Monthly · per system

Nightly evaluation

Every system is scored every night and every change is scored before release. The suite is yours from day one, and your team can run it without us.

Report

Monthly, one page

Notice

One month

Exit

Suite handed over

Book a call

3264.ai

Know it works. Every night.

Book an evaluation review

Evaluation suites built from your own cases, run every night and before every change.

32 → 64

Accuracy is a number.We measure it every night.

“91% accurate” means nothing.Which 9% fails, and does it matter?

In two weeks, we test every AI system you run or are buying against your own cases, and rank where the evidence is thinnest.

Start with a two-week evaluation review.

From your historyto a nightly score.

Click any score.See what is behind it.

Built once. Scored across every system.Each system gets its first score faster than the last.

Four systems we run,written up from delivery.

Errors caught you can bridge,line by line.

Measured against the weekbefore we arrived.

Start with a two-week workflows & processes audit.

Three ways in.Each one earns the next.

Know it works. Every night.

Accuracy is a number.
We measure it every night.

“91% accurate” means nothing.
Which 9% fails, and does it matter?

From your history
to a nightly score.

Click any score.
See what is behind it.

Built once. Scored across every system.
Each system gets its first score faster than the last.

Four systems we run,
written up from delivery.

Errors caught you can bridge,
line by line.

Measured against the week
before we arrived.

Three ways in.
Each one earns the next.