Our team will be out of office on Friday, May 1, 2026. We’ll be back and ready to assist you starting Monday, May 4th.

Why is my AI bill so high when my prompts are short?

Contents

Because you are not paying for what the AI writes, you are paying for everything it re-reads on every single step, and new research puts output at roughly a ninth of the total; this article shows you which four things actually move that number.


The Hook

You tightened your prompts. You cut the fluff. You stopped being polite to the machine. And the bill did not move.

That is maddening, because you did the thing everybody told you to do, including me, and nothing happened. So you start to assume the pricing is a black box and you are not smart enough to understand it.

You are smart enough. You were aiming at the wrong target.

Here is the direct answer. Your AI cost is driven by two things: how much accumulated context the model re-reads on every step, and how many times it reaches for a tool. Not by how long your prompt is, and not by how long the answer is.

A team at the University of Washington just measured this across about 4,300 real agent sessions. Re-read context, what they call prefix tokens, was 59.5 percent of total cost. Newly appended text was 29.2 percent. The actual output, the words you read, was 11.2 percent.

Roughly one dollar in nine goes to generation. Eight go to the model reading back things it has already seen.

So when you shorten a prompt, you are negotiating over the smallest line item on the invoice. Candidly, that advice was the wrong lever, and I have given it myself. Shortening prompts is a clarity practice, and a good one. It is close to useless as a cost practice.

The levers that actually work are shorter sessions, fewer repeat lookups, instruction files that stay still, and testing before you raise an effort setting. None require a plan change or anything technical. They are process decisions, and you already own every one.

Key Takeaways

  • Re-read context, not generated text, is where AI money goes: University of Washington researchers measured 59.5 percent of agent cost in re-read prefix tokens versus 11.2 percent in output.
  • A median agent step reads back about 126,000 tokens of accumulated context while adding only 857 new ones.
  • Input cost grows faster than your session does, because every step replays the whole conversation, every tool log and every system instruction.
  • Raising an AI's effort setting mostly increases how many tool calls it makes, and Princeton researchers found more effort produced equal or worse accuracy in 21 of 36 tested combinations.
  • The four levers that actually reduce spend are shorter sessions, no repeat lookups, stable instruction files, and testing before raising effort. None require upgrading your plan.

The Problem

Let me describe where most business owners actually are with this, because the pricing pages do not.

You see one number at the end of the month. You have no per-task visibility. And the advice available to you is almost entirely about word count, because word count is the only part of this a non-technical user can see.

So you optimize the visible thing. You trim your prompt. You feel responsible. Nothing changes.

Here is the thing: the invisible part is where the money is, and it is structural rather than behavioral.

Researchers at HKUST, Alibaba and ByteDance put it plainly in their AgentSysBench paper. "Because each consecutive LLM call appends the entire historical conversation, tool execution logs, and system prompts to the context window, the input size scales quadratically with the turn count."

Quadratically. Turn twenty is not twice as expensive as turn ten, and nothing on your screen tells you it is happening.

The same team names a second cost nobody warns you about, the control-plane tax. "Beyond productive work that deciding the next action and producing outputs, tokens are spent on tool schemas, raw observations, and safety checks. This control-plane tax grows over multi-step sessions and fills context, raising latency and cost."

Translate that into your world. Every tool your assistant can reach for has a description riding along in the context, and every file it reads dumps raw output into the conversation. None of that is your prompt, none of it is the answer, and all of it is billed again on every later step.

Your work is measured in tasks. The meter runs on re-reading. So the better question is not how do I write shorter prompts. It is where is the re-reading happening, and which parts do I control.

The Evidence

Five findings, and the fourth one will change how you use the effort slider.

The cost split is lopsided and it has been measured. The TraceLab study out of the University of Washington, by Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy and Baris Kasikci, collected about 4,300 sessions, roughly 350,000 LLM steps and 430,000 tool calls, from 43 developers over about eight months across more than 20 model versions. Their conclusion, verbatim: "This inverts the usual intuition that generation is the expensive part: for coding agents the cost is overwhelmingly in re-reading context." Output is cheap in aggregate, they note, "simply because each step emits few tokens (median 214)."

The re-reading to writing ratio is extreme. From the same paper: "A median Claude step reads back 126k prefix tokens but appends only 857, while Codex reads 116k and appends 886, roughly two orders of magnitude more prefix tokens than append tokens." And that is with caching working well, at a 95.7 percent global hit rate. Cached reads run roughly a tenth of fresh input price in Anthropic's documentation, as low as 0.05 times base for Opus 5.5 and Sonnet 5.5, with OpenAI discounting cached input up to 95 percent. The discount is real. The volume still wins.

Repeat lookups are a large, invisible tax. The AgentSysBench team found that among 373,678 search queries from a production agentic-search application, 27 percent of unique queries recur and account for 67.3 percent of all search API calls. If your assistant checks the same price list four times in a session, you paid four times, plus the cost of carrying all four results forward.

More effort does not reliably mean better results. This is the one I want you to sit with. Princeton's Holistic Agent Leaderboard team, led from Princeton CITP by Sayash Kapoor, Benedikt Stroebl and colleagues, ran 21,730 rollouts across 9 models, 9 benchmarks and roughly $40,000 of compute. Verbatim: "in 21 of 36 model-agent-benchmark combinations, increased reasoning effort produces equal or lower accuracy." They also found two setups on one benchmark landing within two percentage points of each other on accuracy while differing nine times in cost, $171 versus $1,577. Separately, a team from UC Berkeley, ETH Zurich, UIUC and CMU studying 4,018 trajectories found o1 at high reasoning effort hit 29.1 percent issue resolution at $1,400, while the low-effort variant reached 21.0 percent at $400; generating two low-effort solutions for $800 and picking the less overthought one reached 27.3 percent, cutting cost 43 percent.

And the honest counterweight, because you deserve the argument against my framing. A separate study by Yichao Yuan, Ankita Nayak, Souvik Kundu and Nishil Talati found the opposite picture on time: "Across all benchmarks and models, execution is primarily LLM-dominated, with LLM inference accounting for 71-98% of total runtime," with tools at only 2 to 29 percent. The UW team measured tools at 59.8 percent of response time. Both can be true, because it depends on what counts as a tool. Web searches, sandboxes and page scrapes dominate the clock; lightweight in-process functions barely register. So I will say it plainly: the claim about where time goes is boundary-dependent and genuinely contested. The claim about where cost goes is not.

Andrew Ng made a related point in The Batch issue 374, published today. One of his own agents, paperreview.ai, makes around 5,000 to 10,000 web search calls on a typical day, far more than he does by hand. His argument lands on your bill: teams keep optimizing token throughput when for many applications the tool calls, not the tokens, are the bottleneck.

The Solution

Stop thinking about this as a price problem. Think about it as four dials on a machine you operate.

Dial one is session length. Every step replays everything before it. The UW data shows an average task takes 8.8 LLM calls and 10.8 tool calls, with session costs ranging from a $0.61 median to $178 at the 99th percentile. The expensive sessions are not the hard questions. They are the long ones.

Dial two is repeat lookups. The fix is not technical. Put the stable facts in writing once, in the instructions, instead of letting the assistant go find them again every time.

Dial three is instruction stability, and this is the most actionable finding in any of these papers. The AgentSysBench team: "Applications with static system prompts and append-only execution histories (e.g., Claude Code) achieve cache hit rates of up to 99%. In contrast, applications that frequently restructure, reorder, or dynamically fetch context at each turn (e.g., DeepResearch) see prefix reuse drop to 1% or lower."

Up to 99 percent versus 1 percent. Same work, different arrangement. Reshuffle your standing instructions and you break the cache, paying full freight on 126,000 tokens instead of a tenth of it.

Dial four is effort. Someone on r/ClaudeCode who built an effort-routing modification reported, in a post with 100 upvotes, that across about 80,000 of their own requests, cache reads were around 76 percent of cost and output only 8 percent, adding that effort "mostly changes how many tool calls a prompt makes, not how long the answers are." That matches the research, and it reframes the slider. Raising effort is not buying better prose. It is authorizing more trips to the well.

Candidly, this system works because it costs nothing to implement. You are changing when you start a new chat, what you write down once, and how you arrange a file.

Practical Steps

Six steps. None of them technical. You can do the first four this week.

1. Close the session at every task boundary. When a task is done, start a new conversation, even if the next one is related. Because input scales quadratically with turn count, a long session punishes you for continuity you are not using. If you would not bother re-explaining the last task to a new assistant, the model does not need to keep re-reading it either.

2. Build a standing facts block and write it once. Your pricing, service names, audience, constraints, approved terminology. Anything your assistant has had to ask about or look up more than twice belongs here. Remember the number: 27 percent of unique queries drove 67.3 percent of all search calls.

3. Freeze the arrangement of your instructions. Keep standing instructions at the top in a fixed order and append new material at the bottom. Do not reorder, do not reshuffle, do not swap which files load first. That one discipline is the difference between a cache hit rate near 99 percent and one near 1 percent.

4. Test at the lower effort setting before you raise it. Run the same task twice on real work, once at your normal setting and once a step lower. In a controlled r/ClaudeCode test with 185 upvotes, where answer keys were written before any model ran, Haiku at medium effort matched Sonnet's quality on the agentic task at 29 API requests and roughly $0.08 against Sonnet's 30 requests and roughly $1.68, while Haiku at high effort was correct but burned 385 requests and roughly $1.03 for no quality gain. That is practitioner evidence, not research, but it points where both papers point.

5. Count tool calls, not words, when something feels expensive. The UW data found tool calls longer than one minute were only 4 percent of all tool calls but accounted for 85 percent of total tool-call time. A handful of slow lookups does almost all the damage.

6. Mind the idle gaps if you are on Gemini. Of the three major providers, Google is the only one that also charges a time-based cache storage fee, which penalizes leaving a session open between turns. On Anthropic and OpenAI an idle gap mostly risks losing the cache. On Gemini it can cost you while you are at lunch.

Here is a prompt to run step two properly, built on the Perfect Prompt Framework 2.0.

[The Job] Build me a standing facts block that stops my AI assistant from looking up or asking about the same information repeatedly. This is for: [MY NAME AND ROLE], who runs [DESCRIBE THE BUSINESS]. It matters because repeat lookups are a large invisible share of what I pay, and each one also inflates the context re-read on every later step.

[The Background] The tasks I run most often are [LIST YOUR TOP THREE TO FIVE RECURRING AI TASKS]. The questions my assistant keeps asking or re-checking are [LIST WHAT IT RE-ASKS]. Facts that almost never change: [PRICING, SERVICE NAMES, AUDIENCE, APPROVED TERMS, CONSTRAINTS]. Facts that change monthly or faster: [LIST THEM]. Leave out anything that changes weekly, because stale facts in a standing block are worse than no block.

[The Deliverable] Return one standing facts block of 400 words or less, in flat declarative statements with no commentary, ordered most stable first and least stable last, plus a short list of what I deliberately left out and why. Written from the perspective of an operations architect optimizing for cache stability, for one non-technical business owner who will paste this into project instructions and not touch it again for 90 days.

[The Questions] Ask me any questions you have.

Frequently Asked Questions

Does shortening my prompt save money at all?

Barely. Output accounted for 11.2 percent of total cost in the University of Washington study, and your prompt is a small slice of the input. Shortening prompts improves clarity and reduces misfires, which is worth doing. It is not a cost strategy. Sessions and repeat lookups are.

What are prefix tokens and why do they cost so much?

Prefix tokens are the accumulated context, your instructions, the conversation so far and every tool result, that gets re-read on each new step. They are billed at roughly a tenth of fresh input price when cached, but the volume is enormous: about 126,000 re-read tokens per step against 857 new ones in the median case.

Will switching to a cheaper model fix my bill?

It helps. Claude Haiku 5.5 launched October 7, 2026 at $0.10 input and $0.50 output per million tokens for prompts up to 100K, which Anthropic puts at roughly 75 percent cheaper to run on average than Haiku 4.5. But a cheap model running a sprawling session still costs more than a mid-tier model running a tight one.

Why does a long conversation get expensive so quickly?

Because every step replays the entire history. As the HKUST and Alibaba researchers put it, input size scales quadratically with turn count, since each call appends the whole prior conversation, all tool logs and all system prompts. Turn thirty is not three times turn ten. It is considerably worse.

Should I turn reasoning effort all the way down to save money?

No. Test it per task type rather than setting it globally. Princeton found higher effort produced equal or lower accuracy in 21 of 36 combinations, but the other 15 improved. The point is that effort is a per-task decision you verify with your own work, not a default you set once and trust forever.

The Close

You trimmed your prompts and nothing happened. Now you know why: you were negotiating over eleven percent of the bill while the other eighty-nine sat invisible, in the part of the conversation nobody shows you.

I think the industry got this backwards, and I include my own past advice in that. We taught prompt brevity because prompts are the part you can see. The meter was running on re-reading the entire session, over and over, on every step.

So here is the most direct thing I can say to you. If your AI spend bothers you, stop editing your words and start managing your sessions. Finish a task, close the conversation, open a new one. Write your stable facts down once instead of letting the assistant fetch them nine times. Put your instructions in a fixed order and leave them alone. Test the lower effort setting on real work before you reach for the higher one.

Four habits. Zero dollars to implement. No plan change, no new tool, no technical skill.

Credit where it is due: the University of Washington TraceLab team, the AgentSysBench researchers, Princeton's HAL team, the overthinking researchers at Berkeley, ETH Zurich, UIUC and CMU, and Andrew Ng for pointing out that agents are computer users too.

If you want to work through this alongside people turning these same dials on their own businesses, come find us. That is most of what we do.

The AI is not charging you to think. It is charging you to remember.


About the author. Jonathan Mast is the founder of White Beard Strategies, where he teaches non-technical entrepreneurs how to use AI to amplify the skill and experience they already have. He runs a 500,000 member Facebook community and the AI Insiders membership, speaks on practical AI adoption, and builds his trainings around the Perfect Prompt Framework 2.0.

P.S. The single cheapest habit on that list is also the one you will resist most: closing a conversation that is going well. I understand the reluctance. The session that feels productive at turn forty is the one quietly writing the invoice.

About the Author