A procurement checklist for AI vendors, from the build side

AI vendor demos in financial services have started to look alike, and there is a simple reason: most of them are built on the same handful of models. The part that differs, and the part you will live with for years, is everything around the model.

The industry now has a name for that part. Thoughtworks summarises it as "Agent = Model + Harness": the harness is everything except the model¹. OpenAI described "harness engineering" in February 2026 as improving an agent's output by shaping its environment while the model stays fixed². Anthropic separates the agent harness, which lets a model act, from the evaluation harness, which tests it³. In a regulated firm, the harness is the tools the model may call, the checks its output must pass, the people who approve it, and the record of what it did.

When you buy an AI system, that is what you are buying. The checklist below is organised around it. For each area there is a question to ask, what a good answer sounds like, and a warning sign.

Ownership

Ask: When the contract ends, what do we keep? Who owns the prompts, the evaluation sets, the runbooks and the source code, and could our own team run the system without you?

A good answer: All of it is yours from the first day, in your own repositories, and the documentation is written for your engineers rather than for the vendor's support desk.

Warning sign: Prompts described as the vendor's proprietary IP, or evaluation results shared only as a summary slide.

Model versions and retirement

Ask: Is the model pinned to a specific snapshot? What happens when the provider retires it?

Providers retire models on published schedules. Anthropic commits to at least 60 days' notice⁴, and OpenAI to at least six months for generally available models⁵. Behaviour can also change without a retirement. Between March and June 2023, GPT-4's accuracy on one reasoning task fell from 84.0% to 51.1% under the same API name⁶. In 2025, infrastructure bugs rather than any model change affected up to 16% of one production model's requests at the worst hour⁷.

A good answer: The snapshot identifier is recorded on every run. A migration means the full regression suite, a replay of recent production inputs, a per-field comparison against the incumbent, and a shadow period of one full business cycle before cut-over.

Warning sign: "We always use the latest model."

Evidence that it works on your documents

Ask: What is the golden set, where did it come from, and how is it scored?

A good answer: It is built from your own historical cases, including the ones that went wrong. Two experts label each case independently, and their agreement is measured before the set is trusted. Anthropic suggests starting with 20–50 tasks drawn from real failures³, and Husain and Shankar advise reading real traces and understanding failures before writing the tests⁸. Scores are reported per field, not as one average, and consistency is measured across repeated runs. Where a model grades outputs, it has been checked against human labels, because judges have biases: in the MT-Bench study, GPT-4 agreed with human preferences 85% of the time but gave the same verdict when the two answers were swapped in only 65% of cases⁹.

Warning sign: A single accuracy figure, or benchmark scores quoted without the model and the date.

The line between model and code

Ask: What does the model decide, and what does code decide?

A good answer: The model locates, extracts, classifies and drafts. Code computes, compares against thresholds, allocates, reconciles and writes to your systems of record. The hand-off between them is a typed record with a pointer to its source, validated before any calculation uses it.

Warning sign: "The model calculates the covenant ratios."

Reconstruction

Ask: A year from now, can you show why the system produced a particular number, without re-running anything?

This matters because output is not reproducible on demand: in one 2025 test at temperature 0, a thousand runs of the same prompt produced 80 different completions¹⁰. Records obligations assume the original can be reconstructed. The advisers' books-and-records rule¹¹ and the broker-dealer preservation rule¹² both assume the original record exists, and a regenerated answer is not the original.

A good answer: Every run stores its inputs, prompt version, model identifier, parameters, raw output, validator results and reviewer actions, for a retention period that matches your obligations.

Warning sign: "We can always regenerate it."

Autonomy and security

Ask: What can the system do without a person? What happens if a document contains instructions?

A good answer: Tools are tiered into read, propose and execute. Execute actions, such as releasing money or sending anything outside the firm, run under a named approver's identity and never from the model. The component that reads untrusted documents has no write tools. The evaluation suite includes red-team cases with injected instructions, and the expected behaviour on each is documented.

Warning sign: "Our model is trained to ignore malicious instructions." Training helps. It is not a control.

Uncertain items

Ask: What happens to the cases the system is not sure about?

A good answer: Each item carries a validator result, a source-support check and a confidence band. Anything that fails goes to a named reviewer, whatever its confidence. Auto-accepted items are sampled on a stated plan. For example, finding zero errors in 299 sampled fields supports, at 95% confidence, an error rate below 1% for that population.

Warning sign: "It's never unsure."

Running cost

Ask: What does a run cost, and what stops a runaway?

A good answer: Cost per run and latency are tracked per step, ideally following the OpenTelemetry conventions for generative AI¹³. Overnight work uses batch pricing, and long, stable documents are cached. On Anthropic's API, batch processing is 50% off and cache reads cost about 10% of the base input price¹⁴. Every run has a token ceiling and every loop has a step cap.

Warning sign: Per-seat pricing with no view of inference cost underneath it.

Regulatory claims

Ask: What exactly are you claiming about compliance?

Listen carefully to the wording.

• "SR 11-7 compliant" is out of date. SR 11-7 and OCC 2011-12 were superseded on 17 April 2026 by SR 26-2 and OCC Bulletin 2026-13. The new guidance places generative and agentic AI outside its scope and is non-enforceable^15,16.
• "Hallucination-free" should end the meeting. Legal research tools marketed along those lines hallucinated 17–33% of the time in a preregistered 2024 study¹⁷.
• "EU AI Act compliant" deserves a follow-up question. After the Digital Omnibus, the Annex III high-risk obligations apply from 2 December 2027¹⁸, and whether a given system is in scope is a legal question, not a product feature.

A good answer: Specific statements of what the system does to support your obligations. That means prompt and output logs, model version tracking and human review, which FINRA's 2026 oversight report recommends¹⁹. For EU funds, it means information you can put into your DORA register of ICT third-party providers²⁰.

How the vendor is paid

Ask: What are we paying for?

A good answer: A defined capability, priced before build and accepted against a baseline measured before work started: cycle time, error rate, cost per file.

Warning sign: Open-ended time and materials, with success defined after the fact.

Red flags at a glance

• No access to the evaluation set, or no evaluation set.
• Accuracy quoted as one number, or without model and date.
• The model does arithmetic that ends up in a report.
• No stored run records; "we can regenerate it".
• The model can send, pay or post on its own.
• Compliance claims tied to rescinded guidance, or to certifications nobody issues.

What persists

Over the life of a system, the model underneath it will change several times. The harness is what persists: the checks, the approvals, the records and the tests that tell you whether the new model is as good as the old one. That is where your due diligence should go.

Related playbook · Private CreditCovenant WatchBorrower certificates recomputed from the credit agreement's own definitions, with headroom, add-back caps and cure rights tracked every test date.Read the playbook →