If your AI tool feels dumber than it did last month, you have no way to prove it, and that question (how do I know if my AI model got worse) is now the most expensive blind spot in your business. Here is the answer and the file that fixes it.
You Cannot Buy Predictability. You Have to Measure It.
The answer is short and you will not like it: you cannot know unless you measured it first. There is no dashboard. There is no notification. Your vendor is not going to email you. The only way to know whether your AI model got worse is to have written down, in advance, what "working" looked like, and to check it on a schedule.
That is the whole post. Everything after this is how to do it in about ninety minutes.
I am writing it today because in the last forty eight hours, four separate complaints landed across the AI forums I read, and not one was about capability. Nobody argued the models are weak. They argued the models are unknowable. People could not tell which model replaced which, whether a price change was a price change or a repackaging, or whether the thing they were talking to this week was the thing they built on last week.
That is a trust problem, not a technology problem. Trust problems do not get fixed by the next release.
Here is the thing. For three years the industry optimized for capability, and capability arrived. The frontier models are extraordinary, and they are converging. When four vendors sit within a few points of each other on every public benchmark, capability stops being a differentiator, and what you actually need from a business tool reasserts itself: will it do the same thing tomorrow that it did today?
Predictability is now worth more than capability. And you cannot buy it. It is not a feature, not on the pricing page, not in the model card. You have to measure it yourself, in your business, on your work.
Key Takeaways
- You will not be notified when a model's behavior changes, so the only reliable signal is a baseline you captured before it changed.
- A published study of five hosted models found accuracy swings of up to 15 percent across ten identical runs at temperature zero.
- Per-token price and per-job price are two different numbers, and only one of them appears on the pricing page.
- Ten real prompts from your own business, with the correct answers written down, beat every public benchmark for deciding what to run.
- A predictably weaker model is more useful to a business than an unpredictably stronger one.
The Problem Nobody Will Solve for You
I made this mistake myself, and it cost me a week.
We run a content process at White Beard Strategies that leans on AI for a specific drafting step. It worked. Then, over maybe ten days, the output started coming back subtly off. Not broken. Not wrong. Off. Longer than it should be. Hedgier. Adding qualifiers we had explicitly told it not to add.
My first assumption was that we had changed something, so we audited our prompts and context files and had people re-read instructions. We burned real hours chasing a change on our side that we could not find, because there was nothing on our side to find.
Here is the part that still bothers me: I never did determine what happened. Not because I could not investigate, but because I had nothing to compare against. No record of what the output looked like in September. No saved example I had already judged correct. I had a vague memory, and a memory is not evidence. I could not tell my team "this is 14 percent worse than it was," because I had no number. I had a vibe.
That is the actual problem, and most people reading this are in the same position. You are running your business on a tool that can change underneath you, you have no instrumentation, and the vendor has no obligation to tell you. The forums are full of people arguing about whether a model got worse, and none of them can prove it either.
This is hard, and I will not pretend otherwise. It requires boring work today to protect against a problem that has not happened yet, and the industry has spent three years training us to expect the next model solves everything.
So reframe it. You are not building a test suite. You are writing down what "correct" means for your business, which is a document you should have anyway. The regression test is the side effect.
The Evidence: Your Tool Is Already Varying and You Cannot See It
Five findings, because the case here is not intuition. It is documented.
One. The same model, same prompt, same settings, ten times, produced wildly different accuracy. Researchers from Penn State and Comcast AI Technologies published "Non-Determinism of 'Deterministic' LLM System Settings in Hosted Environments" at the eval4nlp workshop in December 2025. Five API-hosted models, eight tasks, ten runs each, temperature zero, top-p one, fixed seed: everything a user can control to force consistency. Result: accuracy differences of up to 15 percent across identical runs. On one task GPT-4o scored 88 percent on its best run and 44 percent on its worst. Same model, same question, same day. Run locally without hosting optimizations, the same model was deterministic. The instability lives in the serving layer. [Source: Atil et al., aclanthology.org/2025.eval4nlp-1.12.pdf]
Two. A model's weights can be altered so it behaves normally until a specific condition appears. On November 10, 2026, Microsoft Research published "Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents." Reprogramming fewer than ten rarely-firing neurons is enough to backdoor an agent built on that model, with no retraining. The researchers' phrasing is blunt: the model passes standard evals and behaves normally, until a trigger appears. Their demo trigger was a date change. Once the clock passes midnight into 2027, the agent emits a command that downloads and runs attacker code with whatever access you gave it. Microsoft's guidance: know where models come from, limit agent permissions, monitor what agents do, review consequential actions before execution. Translation: verify signatures and hashes before loading third-party checkpoints, and pin your versions. [Source: Microsoft Research blog, Nov 10, 2026]
Three. Per-token price and per-task price are not the same number. The Batch, issue 378, November 6, 2026, on Gemini 4 Carbon versus Claude Opus 5.5: once Carbon's introductory discount ends it will cost exactly what Opus 5.5 costs per token, yet on Artificial Analysis' Index it finishes tasks with 29 percent fewer output tokens and costs 23 percent less per task. The Batch's conclusion is the one that matters: as frontier models converge on price and score, token efficiency becomes an increasingly competitive feature. Identical sticker price. Twenty three percent difference in what you pay. [Source: deeplearning.ai/the-batch/issue-378]
Four. That gap runs the other direction too. Google shipped Gemini 4 Hydrogen on November 12, 2026, replacing Gemini 3.8 Flash, at $1.50 per million input tokens and $7.50 per million output, double Flash's introductory rate of $0.75 and $3.75. Twice the sticker price. But in Artificial Analysis testing reported by VentureBeat, Flash at high reasoning cost an estimated $7.77 per benchmark task while Hydrogen at high cost $1.87. The more expensive model was roughly four times cheaper to actually use. Same article, opposite lesson: Hydrogen at xhigh scored identically to high on Terminal-Bench 4.0, 54 percent both, while estimated cost per task rose from $1.87 to $3.16. You can pay 69 percent more for zero improvement, and the pricing page will not warn you. [Source: VentureBeat, Nov 12, 2026]
Five. A model you depend on can be switched off for reasons that have nothing to do with you. Anthropic's S-1 filing, covered in The Batch issue 378, discloses that in June 2026 the U.S. Department of Commerce imposed worldwide export restrictions on two of its model configurations. Anthropic switched both off for every customer to comply, and restored them when the restriction lifted. Nobody did anything wrong. The model was simply gone. [Source: Anthropic S-1 via deeplearning.ai/the-batch/issue-378]
One caveat, because I would rather be straight than impressive. The forum complaints I opened with reached me secondhand through daily recaps, not from Reddit directly, and I could not verify the engagement numbers. One carried a claim that a frontier model had been capability-capped during training rather than guardrailed afterward. I have seen no evidence for it. What matters is not whether it is true. It is that people believed it instantly. That is what a trust deficit looks like.
The Fix: Ten Prompts, One File, Once a Month
The framework is deliberately small, because the version you actually run beats the version you design and abandon.
Pull ten real prompts out of your business. Write down the answer you know is correct for each one. Save them in one file. Run that file once a month. Log the model name and version alongside every result.
That is the whole system. I call it a baseline file, and its entire value is that it exists before you need it.
Why ten and not a hundred. The failure mode of evaluation is not inaccuracy, it is abandonment. A hundred-prompt suite is a project. Ten prompts is one sitting. You will run ten next month. You will not run a hundred. And ten prompts out of your actual workflow tell you more than every public benchmark combined, because benchmarks measure capability on problems that are not yours.
That is where the forums landed too. A thread asking whether Artificial Analysis is biased against open weight models drew 267 comments, a figure that reached me through a daily recap. The advice everybody converged on was not "use a better leaderboard." It was: run task-specific evals on your own workload.
The file needs five columns, spreadsheet or plain text: the prompt exactly as you use it, the correct answer, the model name and full version, the date, and the result. Score the result as pass, fail, or changed.
"Changed" is its own category on purpose. Most drift is not failure. It is output getting longer, hedgier, or restructured in a way that breaks whatever you do with it next. That is the category that cost me a week, and the one a pass/fail test misses entirely.
Practical Steps: Build Your Baseline File This Week
1. Open last month's work and pull ten prompts you actually used. Not hypothetical prompts, not impressive ones. The boring weekly ones: the client email, the proposal section, the data cleanup, the summary. If a prompt is load-bearing it belongs in the file. If you would not notice it breaking, leave it out.
2. Write down the correct answer before you run anything. For deterministic tasks, write the actual answer. For judgment tasks, write the three things a correct answer must contain and the one thing it must never do. People skip this step, and skipping it makes the exercise worthless, because without a pre-written standard you will grade whatever comes back as fine.
3. Record the model name and full version number every time. "Claude" is not a version. "GPT" is not a version. If the interface will not show you one, note the date and the exact product name. Without this column your file tells you something changed but not what, and that is half an answer.
4. Use AI to assemble the file, then check its work. The prompt, in Perfect Prompt Framework 2.0 form:
[The Job]
Create a model regression baseline file from the ten prompts I am giving you.
This is for: [MY NAME], owner of [MY BUSINESS], who is not a developer.
It matters because: I need to detect when an AI model's behavior changes before it damages client work.[The Background]
Here is what you need to know: my ten real working prompts are attached, and for each one I noted what I consider a correct answer. My business is [WHAT MY BUSINESS DOES]. The outputs feed into [WHERE THE OUTPUT GOES NEXT], so format changes break me as badly as accuracy changes.
Do not use: generic benchmark questions, trick questions, or anything I did not supply. Do not rewrite my prompts to improve them. Preserve them exactly as I run them.[The Deliverable]
Return: one markdown table, a row per prompt, with these columns: Prompt ID, Prompt Text, Expected Answer or Required Elements, Disqualifying Elements, Model and Version, Date Run, Result.
Must include: a short plain-English scoring rule for each row telling me how to decide between pass, fail, and changed.
Optimize for: accuracy and reusability by a non-technical person.
Written from the perspective of a quality assurance analyst speaking to a small business owner.[The Questions]
Ask me any questions you have.
5. Run the file on the first business day of every month. Put it on the calendar now, before you forget why it mattered. Same day, same settings, same prompts. Under thirty minutes once the file exists.
6. Compare each run to the run before it, not to your memory. Memory is what failed me. Read what came back last month, then what came back now. If anything moved, write down what moved in one sentence. Three months of those sentences is a drift record, and a drift record is leverage in a vendor conversation.
7. Decide in advance what a failure costs you. For each prompt, write one line: what happens if this silently degrades for thirty days before I notice. Some will be annoying. One or two will be expensive. Those justify pinning versions, keeping a second vendor configured, or keeping a human in the loop.
Frequently Asked Questions
Is ten prompts really enough to catch a problem?
Ten prompts drawn from your real workload catch the failures that matter to you, which is the only kind worth catching. A larger suite catches more in theory and nothing in practice, because you will stop running it by month three. Start with ten. Add one whenever a new task becomes load-bearing.
Do I need to be technical or write code to do this?
No. A spreadsheet with five columns and a calendar reminder is the entire technical requirement. The steps about pinning versions or verifying file hashes only apply if you call APIs or load model files directly. If you do neither, skip them without worry.
How do I pin a model version if I am just using the chat app?
Usually you cannot, which is exactly why the logging column exists. Consumer chat interfaces route you to whatever the vendor decides, often without announcement. Record the product name and date on every run, so when behavior shifts you can say when it shifted and what you were using.
Should I switch to whichever model scores highest on the benchmarks?
Benchmarks measure general capability on problems that are not yours, and the public leaderboards disagree with each other often enough that technical communities argue about methodology constantly. Your ten prompts are a better decision tool than any index. Run candidates against your file and pick the one that holds steady.
Is a cheaper model per token actually cheaper for my business?
Not reliably. The Batch documented two models at identical per-token pricing where one cost 23 percent less per task because it used 29 percent fewer output tokens. Per-token price is on the pricing page. Per-job price is in your invoice. Track the second one.
Write It Down Before You Need It
I opened with a week I lost chasing a change I could not prove. Here is the part I actually regret.
It was not the week. It was that I had spent two years telling entrepreneurs to treat AI like a business system, and I had not instrumented my own. I had opinions about the model and no record of the model. When it mattered, I had nothing to point at.
You are going to be in that position. Not maybe. The research says your tool varies on identical inputs. Models get retired, repriced and replaced. A government agency switched one off for every customer in a week, and that is in an SEC filing. The question is not whether the ground moves under you. It is whether you have a mark on the wall when it does.
So go pull ten prompts. Write down what right looks like. Run it on the first of the month. You will spend ninety minutes and feel like you wasted them, right up until the day you do not.
The best model in the world is useless to you if you cannot tell what it did yesterday. A predictably weaker model beats an unpredictably stronger one in every business I have ever seen.
Measure it yourself. Nobody else is going to.
Want the rest of the operating system? This file is one piece of how we teach entrepreneurs to run AI as a business system instead of a slot machine. Inside the White Beard Strategies membership you get the live trainings where we build these things together, plus the replay library covering prompting, agents and evaluation. Join us at whitebeardstrategies.com.
Jonathan Mast is the founder of White Beard Strategies, where he teaches entrepreneurs to use AI practically rather than theatrically. He is the creator of the Perfect Prompt Framework, a speaker, and the host of a 500,000-plus member community of business owners learning AI together. He keeps his own ten-prompt baseline file in a plain spreadsheet, because the version you actually run beats the version you admire.
