AI 트렌드5 min read한국어로 읽기

Cache Prices Just Dropped. What Happens to Your AI Unit Costs?

A 75% cut in cache-read pricing changes how small teams should assemble prompts and price AI features.

The most important line in the first week of September's model announcements wasn't a benchmark score. It was a line on a price sheet. On September 1, Anthropic shipped Claude Fable 5.1 along with Mythos 5.1, its trusted-access twin, and cut cache-read pricing by 75% while keeping the headline token price at the same level as Fable 5. Google DeepMind followed with Gemini 3.8 Flash and OpenAI shipped GPT-6 Astra days later, so attention went to capability. But if you personally pay the API bill every month, the quiet repricing of cached tokens is the bigger event.

What does cheaper cache actually change about your unit costs?

Short version: prompts that repeat identically are now far cheaper than prompts you rewrite on every call. Fresh input tokens still cost what they cost, but the reused prefix got dramatically cheaper, so two teams shipping the same feature can end up with very different bills depending on how they assembled the prompt. "Keep prompts short" used to be the right instinct. "Keep the long part fixed and reusable" is the better instinct now.

Imagine a support bot that attaches a 3,000-token instruction block — return policy, shipping rules, tone of voice — to every inquiry. At 500 inquiries a day, you were paying full price for those 3,000 tokens 500 times. Wrap the same block as a cached prefix and only the first call reads it fresh; the rest are billed at the cache-read rate. Same feature, thinner invoice, which is a rare kind of win.

How should you restructure your prompts?

The whole trick is to split the prompt into a static block and a variable block, and always put the static block first. Caching reuses the matching prefix, so a single customer name or timestamp near the top will invalidate the huge instruction block sitting behind it. This is an hour of work, not a rewrite.

  • Top: system rules — role, prohibited phrases, output schema, brand tone. Only things that rarely change.
  • Next: always-on reference material — catalog summaries, price tables, FAQs, policy docs that every request needs.
  • Bottom: variable input — the customer question, session history, today's date, order number. Only this part changes per call.
  • Stop editing the static block casually — change one comma and the cache starts over. Batch prompt edits into scheduled releases.
  • Check your traffic shape — one warm-up call before a busy window lets the rest ride the cache, while a batch job that runs twice a day will barely benefit.

If you use RAG, there's one more thing. Pasting retrieved chunks near the top of the prompt breaks the prefix on every single call. Moving fixed instructions up and retrieved passages down is a pure reordering change that can move your hit rate a lot.

What should you check before turning caching on?

Caching is a cost lever, not a quality guarantee. Restructuring a prompt means changing the order in which the model sees your input, and answer quality can shift in subtle ways. Build a small eval set of 20 to 30 real questions and compare before and after.

Watch your procurement path too. GPT-6 Astra opened first through a defender-focused program and ChatGPT Business and Pro, with API access rolling out afterward — meaning "the model launched" and "we can call it from our backend" are two different dates. So don't hard-wire your caching strategy to one vendor. Keep the static/variable split as a structural pattern in your code and it survives a model swap.

Pairing this with model routing multiplies the effect. Simple calls like classification, tagging, and summarization can drop to a lightweight tier such as Gemini 3.8 Flash, while long-instruction drafting and support flows stay on the stronger model where the cache pays off. Running every call in your product through one model is usually the most expensive choice available.

Finally, look at what you put in that static block. If customer PII or internal contract terms end up inside a heavily reused prefix, you're storing risk that outweighs the savings.

Three things to do this week

No grand refactor required. These three steps are enough to show up on next month's invoice.

  1. Pick your highest-volume feature, rewrite its prompt as an explicit static block plus a variable block, and move the static part to the front.
  2. Log cache-read tokens separately from fresh input tokens and check the hit rate daily. No numbers, no optimization.
  3. Run a 20-to-30-question eval set before and after the reordering, and only roll out fully if quality holds.

Teams that read the footnotes on the price sheet beat teams that chase launch posts. Capability rankings keep flipping, but a team with a clean cost structure keeps its margin no matter which model is on top.

FAQ

What exactly does prompt caching store?
It stores the repeated front portion of your prompt — system rules, policy documents, catalogs and other static blocks — in a reusable form. From the second call onward, that span is billed at a much lower cache-read rate than fresh input tokens. Only the matching prefix is reused, so block ordering decides whether you get the discount.
Will caching hurt my answer quality?
Caching itself doesn't change outputs, but reordering your prompt to make caching work can shift quality in subtle ways. Build a small eval set of 20 to 30 real questions and compare answers before and after the change before rolling it out.
Is this worth it for a low-volume product?
The savings come from repetition, so a batch job that runs a few times a day will see little benefit. Still, separating static and variable blocks is good structure regardless of cost, and it makes swapping models later much easier.

← Back to Insights