Short answer. AI vendor evaluation comes down to written answers on five things, all before you sign: what happens to your data and if it trains their models, how the tool performs on your own documents instead of a demo set, what it costs once usage grows, how you get your data out if you leave, and who carries the liability when the tool is wrong. Ask for those answers in the contract. A vendor who can’t answer in writing is telling you something.
Facts verified 23 September 2026. Not legal advice.
Every AI demo you will ever see was run on data the vendor chose, and that is where most AI vendor evaluation goes wrong.
That is fair. Nobody demos on a mess. The trouble starts when a Calgary operations lead watches a flawless forty-minute walkthrough, signs a three-year order form that afternoon, and discovers in month two that the tool falls over on the scanned field tickets and half-filled PDFs that make up most of the company’s real paperwork.
The contract, meanwhile, is where the risk actually lives. In February 2024, in Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for wrong refund advice its website chatbot gave a customer, and rejected its argument that the chatbot was a separate legal entity responsible for its own actions. The customer never dealt with the software company behind the bot. They dealt with Air Canada. I wrote up what the Air Canada chatbot ruling means for your business separately, and the short version for buyers is this: the vendor’s mistakes arrive with your name on them.
The federal government’s cyber agency has reached the same place from the security side. The Canadian Centre for Cyber Security’s May 2026 primer on AI security says contracts with AI vendors should include explicit data usage, privacy, audit and liability clauses, including a ban on using your data to train models without permission. That is your checklist’s spine. The questions below put it in plain language.
Where AI vendor evaluation should start
AI vendor evaluation starts with your own work, before any vendor call. Write down the process you want to change, the documents it touches, who handles them and what good looks like. Without that page, every vendor answer sounds reasonable and you have nothing to test it against.
If you haven’t mapped the process yet, do that first. The processes to audit before buying AI tools covers how, and a short AI workflow audit will tell you if the thing you are shopping for is even the right fix.
Then send the questions in writing, ahead of the second meeting. Written answers are slower to get and far harder to walk back.
What should you ask an AI vendor about your data?
Ask where your data is stored and processed, if it trains any model, how long it is kept, and which subcontractors see it. Tie each answer to a contract clause or named policy page. Under Alberta and federal privacy law you stay accountable for personal information you hand a vendor.
1. Is our data used to train or improve any model, yours or anyone else’s? The major business plans now say no by default. OpenAI’s enterprise privacy page, for example, states that it does not train its models on business data by default. You want that commitment in your contract, along with what happens if the default ever changes.
2. Where is our data stored, and where is it processed? These are two different answers, and vendors often give you only the first. Storage in Canada with processing in the United States is common. How much that matters depends on your data and your clients, which I cover in does it matter where your AI keeps your data.
3. How long do you keep prompts, files and outputs, and can we set that ourselves?
4. Which subprocessors touch our data, and will you tell us before adding one? A subprocessor is any other company the vendor passes your data to: a cloud host, a model provider, a support desk. Many AI tools are a thin layer over someone else’s model, so the list tells you who you are really trusting.
How do you know the tool works on your data?
Make the vendor prove it on your documents, in a short paid or free pilot with success measures you set before it starts. A tool that performs on a curated demo set tells you very little about how it handles your scanned forms, your abbreviations and your exceptions.
5. Will you run a pilot on a sample of our real documents, with results we score? Pick twenty to fifty real examples, including the ugly ones. Decide in advance what counts as a pass. If you need a starting point for the scoring, how to measure an AI rollout has one.
6. Which model or models sit underneath the product, and what happens when they change? Model providers retire and replace models on their own schedule. A tool that worked in March can behave differently in September. Ask how you will be told, and if you can hold a version while you retest.
7. Can we talk to two customers our size, in our sector, who have used it for more than six months? Six months matters. Everyone likes a tool in week three.
Watch for claims that the product is “AI-powered” without any detail on what the AI does. Regulators have started treating that as a problem in its own right: in March 2024 the US Securities and Exchange Commission brought two enforcement actions against investment advisers over false claims about their use of AI. Canadian law firms have warned that similar exposure exists here under securities and competition law.
What does the AI tool cost once people actually use it?
Ask for the price at your expected usage in year two. The entry price is the least useful number you’ll get. Many AI tools add usage charges for tokens, credits or actions to a per-seat fee, so the bill grows with adoption. Get renewal caps and overage rates in writing.
8. What is the all-in cost at double our starting usage? Seats are the visible part. Usage is where the surprises sit. Anthropic’s Enterprise plan, for instance, lists a seat price plus usage billed at API rates. Microsoft sells Copilot Studio capacity in credit packs. Neither is hidden, but neither is on the first slide either. What an AI tool really costs after the licence walks through the full list.
9. What is the renewal price cap, and how long is the minimum term?
10. If we leave, how do we get our data out, in what format, and how fast do you delete it? Ask them to show you an export. A button in a settings screen you can see today beats a promise in a support article.
Who is liable when the AI gets it wrong?
By default, you are. Customers, regulators and courts look at the business that deployed the tool. Your contract decides how much of that risk the vendor shares, through indemnities, liability caps, security commitments and breach notice. Read those clauses before the feature list, and have counsel read them too.
11. What do you indemnify us for, and what conditions come with it? An indemnity is a promise to cover your costs if a specific claim is made against you. Some vendors offer one for copyright claims over AI output. Microsoft’s Customer Copyright Commitment is the best known, and it comes with conditions: for configurable services such as Azure OpenAI and Copilot Studio, customers must put Microsoft’s required mitigations in place to stay covered. Conditions are normal. Not knowing them is the risk.
12. What is your liability cap, and does it cover data breaches? Many standard contracts cap liability at the fees paid in the last twelve months. On a small subscription, that figure won’t cover much.
13. Which security audits can you show us, and how fast will you tell us about a breach? A SOC 2 Type II report, meaning an independent auditor has tested the vendor’s security controls over a period of months, is a reasonable ask for any tool that will hold client data.
14. Can we switch off features, such as web browsing, memory or connections to our other systems, for the whole company? Admin controls decide what your team can do with the tool on a bad day. If the answer involves emailing support, that’s your answer. This matters more once the tool can act on its own, which is why AI agent permissions deserve their own conversation.
Which answers should end an AI vendor evaluation?
Walk away from vague answers on training data, refusals to pilot on your documents, no export path, and liability caps that make the contract meaningless. One weak answer can be negotiated. Three usually means the product or the company is not ready for your data, whatever the demo looked like.
| Question | A good answer sounds like | Red flag |
|---|---|---|
| Training on our data | No by default, in the contract, with notice before any change | “We anonymize it first” |
| Storage and processing location | Named regions for both, listed separately | “Our cloud is very secure” |
| Pilot on our documents | Yes, with your scoring, in two to four weeks | “Our demo data is representative” |
| Underlying model changes | Named models, advance notice, a retest window | “We always use the best model” |
| Cost at twice the usage | A written quote | “Most customers never hit the limits” |
| Leaving | Standard export formats, a deletion date in writing | Export on request, timing unclear |
| Liability and security | Current audit report, breach notice in days | “We’re working on SOC 2” with no date |
My position: in 2026, I would not sign a first AI contract longer than twelve months. The products, the prices and the underlying models are all changing faster than a three-year term can account for. Pay the higher monthly rate for a year if you must. The flexibility is worth more than the discount.
Some vendors will push back and say a longer term funds the onboarding. Fair enough. Then ask for an exit clause tied to the pilot measures you set in question five.
Paste this into your AI. Drop in the vendor’s written answers or their contract, then use this prompt to run a first-pass AI vendor evaluation before your lawyer’s clock starts.
You are reviewing an AI vendor for a [number]-person [industry] company in Alberta. Below are the vendor's written answers and/or contract terms. For each of these topics, quote the exact text that answers it, or write NOT ANSWERED: 1. Use of our data for model training 2. Where data is stored and where it is processed 3. Retention period and who controls it 4. Subprocessors and notice of changes 5. Underlying models and notice of model changes 6. Pricing at double our starting usage, renewal caps, minimum term 7. Data export format and deletion timeline on exit 8. Indemnities and their conditions 9. Liability cap, and does it cover data breaches 10. Security audits and breach notification timing Then list the three weakest answers and draft one follow-up question for each. Do not give legal conclusions. [Paste vendor answers or contract here]
Send the vendor your questions before the next meeting, and don’t book the signing until every one of the fourteen has an answer you can point to in writing.
Questions people ask
Ask if your data trains their models, where it is stored and processed, how long it is kept, which subprocessors see it, if they will pilot on your real documents, what it costs at double your starting usage, how you export your data if you leave, what they indemnify, and how quickly they report a breach. Get the answers in writing.
Most major business plans say they do not by default. OpenAI’s enterprise privacy page, for example, states it does not train on business data by default. Smaller vendors vary, and consumer plans often have different settings. Confirm the answer for the specific plan you are buying and get it written into the contract.
Usually the business that deployed it. In Moffatt v. Air Canada, decided in February 2024, a British Columbia tribunal held Air Canada responsible for its website chatbot’s wrong advice and rejected the argument that the chatbot was a separate legal entity. Your contract decides how much of that cost the vendor shares. This is not legal advice.
Data use and training restrictions, data location, retention, subprocessor notice, audit rights, breach notification, liability caps and indemnities. The Canadian Centre for Cyber Security’s May 2026 AI security primer specifically recommends explicit data usage, privacy, audit and liability clauses in AI vendor contracts, including a ban on unauthorized use of your data for model training.
Long enough to test the tool on a real sample of your work, which for most document or email tasks is two to four weeks. Set the success measures before it starts, include your messiest examples, and have the people who do the work score the results. A pilot on the vendor’s data does not count.
It depends on the job. Big platforms usually win on security paperwork and admin controls. Specialists often fit a narrow process better. Many specialist tools run on a big provider’s model underneath, so ask which one, and apply the same data, pilot, cost, exit and liability questions to both.
