Our team will be out of office on Friday, May 1, 2026. We’ll be back and ready to assist you starting Monday, May 4th.

How Do I Test Whether an AI Tool Actually Does What the Vendor Says?

Contents

Pick the single promise the vendor leads with, find the mechanism that is supposed to deliver it, and name what that mechanism cannot see. That is the whole test, and it takes about ten minutes before you hand over a credit card.


The method is sitting in this week's news, and almost everyone is reading it wrong

On Monday, November 16, 2026, OpenAI announced that four outside organizations, Apollo Research, METR, Transluce and Goodfire, would become "embedded assessors" with access to internal models, training processes and some employees, to "give people a clearer view of how labs are evaluating and managing risks." By Wednesday, researchers and former safety employees were publicly measuring the distance between that promise and the product.

Notice how. They did not argue about vision or intentions. They named three checkable things: the scope of the access, the whitelist controlling which employees the assessors could talk to, and the absence of internal Slack. Three items, each one either true or not.

That is the skill. Not the outrage, the method.

So here is the direct answer to the question in the headline. To test whether an AI tool does what a vendor says, take the one specific promise the marketing leads with, ask what mechanism has to work for that promise to come true, then ask what the mechanism cannot see. If the vendor cannot name the mechanism, the claim is not a claim, it is a mood. You need ten minutes and a willingness to be slightly unpopular on a sales call.

Candidly, I think this is the highest-return skill a non-technical owner can build in the next twelve months. Tool prices are falling and claims are getting louder at the same time. That combination is a trap.


Key Takeaways

Editor: render as a shaded callout box.

  • The researchers who questioned OpenAI's new assessor program named three checkable items, and you can copy that structure for any vendor claim.
  • Every AI claim has three testable parts: the promise, the mechanism meant to deliver it, and the gap that mechanism cannot see.
  • A benchmark number measures a test, not your business, and published research shows how far those two diverge.
  • Capterra's 2026 survey of 3,385 software buyers found only 34 percent had a smooth purchase and implementation.
  • The most useful question you can ask a vendor is what their tool does not measure.

The problem is not that vendors lie. It is that you are grading the wrong thing.

Most owners think the risk in buying AI software is getting scammed. It is not. Fraud is rare and usually obvious. The real risk is a claim that is technically accurate and completely useless to you.

Here is the thing. When a company says its model handles 96 percent of what a far more expensive competitor handles, that sentence contains a real measurement. What it does not tell you is which 96 percent, on what tasks, scored by whom, and whether the missing 4 percent is the part your business depends on. Microsoft shipped MAI Code 2 at Ignite on November 17 with a claim in that shape, about 96 percent of the capability at 5 percent of the price. A user posted it to r/ClaudeAI asking whether anyone would switch. Most replies were skeptical. As summarized in a November 18 recap of the subreddit, one commenter noted that humans share 98.8 percent of their DNA with chimpanzees.

That comparison is doing real intellectual work. A high percentage of overlap tells you nothing about whether the remaining difference matters, and in a coding tool the last few percent often decides whether you ship or spend Saturday fixing things. The pattern repeated that week on r/LocalLLaMA, where a developer led with 770 tokens per second and then unpacked it himself into 776 for structured output, 262 in chat, and 194 for prose. One number became three, and the three tell a different story.

[PLACEHOLDER, JONATHAN TO REPLACE WITH A REAL INCIDENT: a specific time you bought an AI tool on a vendor claim and it did not hold up. Include what you paid, how long before you noticed, and what it cost you. Suggested shape: "In [month year] I paid [amount] for [tool] because [the claim]. It took me [time] to notice that [the gap]."]

I am not telling you that to be relatable. I am telling you because the gap was visible before I paid and I did not look. The information was public and I did not ask the one question that would have surfaced it. That is the part that stings, and the part this post exists to prevent.

Reframe it this way. You are not trying to catch a liar. You are trying to find the edge of a true statement. Those are different jobs, and the second one is far easier.


The research says the gap between the score and the work is normal, not exceptional

Four pieces of evidence. Sit with the second one in particular.

One. Most AI benchmarks have never been validated as measuring what they claim. In November 2025, 29 expert reviewers led by Andrew M. Bean at the Oxford Internet Institute reviewed 445 large language model benchmarks from leading machine learning conferences. In their words, they "find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims." Plain English: the industry's own scorecards often fail to establish that they measure what their name says. See arxiv.org/abs/2511.04703.

Two. When a benchmark is broken, the error is big enough to flip a buying decision. A team including researchers from Berkeley, Stanford and Princeton audited agentic benchmarks, the ones that score AI on multi-step real work. Their published abstract states that flaws in task setup or reward design "can lead to under- or overestimation of agents' performance by up to 100% in relative terms." One widely used benchmark relies on insufficient test cases; another counted empty responses as successful. Read that again. A benchmark scored a blank answer as a win. See arxiv.org/abs/2507.02825.

Three. Buyer regret is now the normal outcome, and it has one dominant cause. Capterra's 2026 Software Buying Trends Report surveyed 3,385 software decision-makers across 11 countries. Only 34 percent had both a smooth purchase and a smooth implementation. Sixty-six percent hit unexpected disruption, regret, or both. And 89 percent of buyers who regretted a purchase had first run into an unexpected disruption during implementation. Regret is not random. It is downstream of a surprise, and a pre-purchase test exists to find surprises early.

Four. A named case, where the stakes stop being theoretical. On November 26, 2024, the Federal Trade Commission took action against Evolv Technologies over its AI-powered weapons screening systems, in use at the time in more than 800 schools across 40 states according to the complaint. The FTC alleged Evolv deceptively advertised that its Express scanners would detect all weapons and that its AI made them more accurate and more cost-effective than metal detectors, including cutting labor costs by 70 percent. The complaint alleged the scanners flagged laptops, binders and water bottles while failing in several instances to detect weapons, specifically a seven-inch knife brought into a school in October 2022 and used to stab a student.

That is what connects all four. The number described a controlled test; the buyer lived in an uncontrolled environment. Nobody had to lie. The score simply did not travel.

To be fair to OpenAI, because fairness is the point of this method: the company told NPR it expects "to deepen these relationships over time and share what we learn to help establish best practices." It also noted that no standards for embedded evaluations exist yet, that assessors can ask to speak with employees who are not whitelisted, and that there is a process for reporting things learned informally. Reasonable people disagree about whether that goes far enough. What is not in dispute is that the disagreement is checkable, because the claim was broken into parts.


The framework: Promise, Mechanism, Gap

Three questions, the same three the researchers asked. They work on a safety program, a coding model, or a 399 dollars a month subscription.

Promise. Write down in one sentence the specific thing the vendor says will happen. Not "saves time." Not "AI-powered." The actual claim, with a number or an outcome in it. If you cannot find a sentence that specific, that is your answer. Sam Altman's September 2026 pledge was that independent evaluators would get "employee-like access" to internal systems. That is a testable promise, which is why it became measurable in November.

Mechanism. Ask what has to be true for the promise to happen. The researchers did not ask whether OpenAI meant well. They asked who the assessors are allowed to talk to, and the answer was a whitelist. Apollo Research disclosed that when an employee who has not been whitelisted reaches out, the assessors must loop OpenAI in before any substantive conversation. Your version: on what data was that result produced, who scored it, how many examples, and does it run on your inputs or their curated sample? The best live version is one sentence. "Show me that result on something I bring."

Gap. Ask what the mechanism cannot see. Nobody takes this step, and it is where the money is. Sydney Von Arx of Nightingale said "the assessors have very narrow scopes to their access," but her strongest point was the omission: "Nobody has access to internal Slack… Imagine being an 'employee' and only being allowed to talk about company information with whitelisted coworkers and having no Slack." Her reason for caring is your reason too. Outside eyes "help you find your blind spots."

Three founders modeled this publicly this week. Daniel Kimber of Ballet published a number where most people publish an adjective: "AI agents like Claude are quick to set up, but they don't do the same thing twice. In our benchmark they got 78% of tasks right on every run."

Tracewell Gordon of TruLata did something rarer. Asked at launch whether his free scan checks Bing indexing, he answered "Good catch. The free scan doesn't check Bing indexing today," then repeated it in his own launch-day update: "Fair gap to call out." And RankSpot co-founder Daniil Poletaev corrected his own founding premise, writing that he "truly believed SEO should be 100% automated" and that this is exactly how the product launched six months earlier. RankSpot 2.0 relaunched November 18 built on human approval instead.

A vendor who names their own gap is handing you free diligence. Buy from those people.


Practical steps: the ten-minute test

Run this before any AI purchase over a hundred dollars a month. Set a timer.

1. Copy the loudest sentence from the sales page into a document. (1 minute.) The one with a number, a percentage or a promised outcome. If it has no number in it, write down what a number would look like and ask the vendor for it.

2. Write the one job you are actually hiring this tool for. (1 minute.) One sentence, your words, your workflow. Julia McCoy ran 100 how-to prompts through ChatGPT, Claude and Perplexity and published the results on November 18, because every carousel about getting cited in ChatGPT in fourteen days omitted the actual data. Her method was deliberately boring: fresh sessions, personalization off, logging whether each engine searched and what it credited. ChatGPT skipped searching entirely on 24 of the 73 questions it answered. Copy that posture.

3. Ask the mechanism question in writing. (2 minutes.) Email or chat, not a call, because you want it on the record. "What data was that number produced on, who scored it, and how many examples?" A good vendor answers in two sentences. A nervous one sends a case study.

4. Ask the gap question. (1 minute.) "What does your tool not measure or not handle well?" Gordon answered that in public unprompted. If your vendor cannot name one honest limitation, you are talking to marketing, not to the product.

5. Run five real inputs through the trial. (3 minutes.) Your messy work, not their samples. Capterra found poor trial experiences are a top reason successful buyers cut a vendor from the shortlist.

6. Write down the disruption you fear most, then ask about it. (1 minute.) Remember: 89 percent of regret followed an unexpected disruption. The damaging word is unexpected. Expected disruption is just a project plan.

7. Date the decision. (1 minute.) Record the claim you believed and the one result that would prove it wrong, then set a reminder for day 30. That turns a purchase into a measurement.

For steps 3 and 4, use this prompt, built on our Perfect Prompt Framework 2.0.

[The Job]
Help me stress-test one specific claim an AI software vendor is making before I buy.

[The Background]
My business: [WHAT YOU DO, IN ONE SENTENCE]
The tool and price: [TOOL NAME AND PRICE]
The exact claim from their sales page: [PASTE THE SENTENCE]
The one job I want it to do: [YOUR ACTUAL WORKFLOW]
What I am not technical about: [E.G. BENCHMARKS, INTEGRATIONS, DATA HANDLING]

[The Deliverable]
Three things, in plain English, no jargon:
1. The promise restated as something measurable.
2. The mechanism that would have to be true for the promise to hold, plus the three
   questions I should email the vendor to confirm it.
3. The likely gap, meaning what this kind of claim usually does not cover, plus the
   one question that exposes it.
Written from the perspective of a skeptical procurement advisor speaking to a
non-technical small-business owner.

[The Questions]
Ask me any questions you have.

Frequently Asked Questions

How long should I test an AI tool before buying it?
Long enough to run your real work through it, usually days rather than weeks. Capterra's 2026 report found successful adopters narrow to about three vendors and buy within three months, so speed is not the enemy. Vagueness is. A short test on real inputs beats a long test on sample data.

What if the vendor will not let me test on my own data?
Treat that as an answer, not an obstacle. Demos run on curated inputs by design, so refusing yours suggests the result may not survive contact with reality. Ask for a short paid pilot instead. If a vendor needs a multi-year contract before showing you anything on your own material, the risk is being transferred to you.

Are AI benchmark scores useless, then?
No. They are evidence about a test, not about your business. Use them to compare tools roughly, never to predict your outcome. One published audit found flaws in task setup or reward design can swing measured agent performance by up to 100 percent in relative terms, so treat any headline score as a starting question.

What is the one question I should ask on a sales call?
"What does your tool not measure or not handle well?" A confident vendor names a real limitation in one sentence, the way TruLata's founder publicly admitted his free scan does not check Bing indexing. A vendor who cannot name any limitation has either not tested the product honestly or is not planning to tell you.

I already bought a tool that is not delivering. What now?
Quantify it this week, before renewal. Write down the original claim, what you actually observed, and the monthly cost. Then check your contract for a cancellation or downgrade window, since misleading AI claims have drawn regulatory action and some vendors will negotiate rather than argue. Decide on the number, not on what you already spent.


The close

The gap between what an AI company promises and what it ships is now wide enough that its own former employees are measuring it in public. That is remarkable. It is also a gift, because they showed you the method on their way out the door.

Do not copy the outrage. Outrage is cheap and it protects nothing. Copy the discipline. Take the promise. Find the mechanism. Name the gap. Ten minutes, no technical background, and one uncomfortable question asked while a salesperson is being very nice to you.

Here is what I believe about the next twelve months. Model prices will keep falling and claims will keep getting louder, because when a product gets commoditized the marketing gets amplified. The owners who thrive will not be the ones with the best tool stack. They will be the ones who can take any sentence a vendor says and, in ten minutes, tell you where its edge is.

You already know how to do this. You do it with contractors, with suppliers, with hires. You stopped doing it the moment someone said the word AI, because you assumed you were not qualified to ask.

You are qualified. Ask the question.


Ready to stop guessing? Inside the White Beard Strategies membership we run this exact test on live tools every month, with real claims and real numbers, so you are never the one who finds out after the invoice clears. Grab the training replays if you want the frameworks today, or join the membership if you want a room full of entrepreneurs pressure-testing their AI decisions with you.


About the author

Jonathan Mast is the founder of White Beard Strategies, an AI coaching and mentorship company for entrepreneurs, and the creator of the Perfect Prompt Framework 2.0. He teaches non-technical owners how to use AI with discipline instead of hype, and speaks regularly on practical AI implementation for small business. He has an unfashionable habit relevant to this article: before he repeats a number, he reads the original source document.

About the Author