Prompt caching can reduce the cost and latency of repeated LLM requests, but only when the workflow produces stable prefixes, enough reuse, and acceptable cache-hit behavior. Turning the feature on is not the same as proving return on investment.
This guide gives engineering and FinOps teams a practical prompt caching workflow: find eligible traffic, shape prompts for reuse, instrument cache metrics, calculate net savings, and roll out without hiding quality or reliability regressions. It also includes a seven-day audit that turns provider usage fields into a go, fix, or stop decision.
Prompt caching ROI: the quick answer
Prompt caching is usually worth testing when a workflow repeatedly sends a large, byte-identical prefix within the provider's retention window. It is not automatically worthwhile just because a model advertises discounted cached tokens.
Use this three-part gate before changing production prompts:
| Gate | Pass condition | Stop condition |
|---|---|---|
| Reuse | The same prefix is used several times before expiry | Most prefixes are one-off or user-unique |
| Economics | Observed read savings exceed writes, storage, and operating cost | The cache is created more often than it is reused |
| Outcome | Cost per accepted task improves without a quality or reliability regression | Lower token cost causes more retries, rejected outputs, or unsafe fallback |
The core calculation is:
net_savings = uncached_baseline_cost
- observed_cached_workflow_cost
- incremental_engineering_and_operations_cost
roi_percent = net_savings
/ incremental_engineering_and_operations_cost
× 100
If implementation cost is shared across many cache identities, amortize it over the expected evaluation period instead of assigning the entire project cost to one entry.
What changed for prompt caching in 2026?
Prompt caching is no longer one uniform discount mechanism. Provider designs now differ enough that a generic "cached tokens are cheaper" spreadsheet can produce the wrong answer.
For example, OpenAI's current GPT-5.6 documentation describes automatic prefix matching, explicit prompt_cache_key and cache_control controls, and separate cache-read and cache-write usage. Cache writes for that model may carry a premium, so the break-even calculation must include the cost of creating or extending a cache—not only the discounted reads. OpenAI also exposes cached, uncached, and cache-write token details in usage fields for supported requests.
Anthropic uses explicit cache breakpoints and time-to-live choices. Gemini explicit context caching can add storage charges. DeepSeek documents automatic context caching with separate hit and miss rates. These designs can all produce savings, but they need different telemetry and formulas.
What is prompt caching?
Prompt caching lets an LLM provider reuse computation for prompt content it has recently processed. Instead of charging and processing every repeated input token at the normal rate, the provider can apply a lower cached-input rate or a separate cache-read price to the reusable portion.
The reusable content is usually a stable prompt prefix. Common examples include:
- A long system prompt and policy block.
- Tool definitions shared by every agent turn.
- A large document, repository map, or product catalog queried repeatedly.
- Few-shot examples reused across a classification or extraction job.
- A conversation history shared by several possible next actions.
Provider implementations differ. OpenAI documents automatic caching for qualifying prompt prefixes and exposes cached-token details in API usage; supported newer models can also expose cache-write details. Anthropic supports explicit cache breakpoints and multiple time-to-live options. Google Gemini supports explicit context caches with storage charges, while DeepSeek documents automatic disk-based context caching with separate cache-hit and cache-miss input rates. Always confirm current model support and pricing in the provider's official documentation before committing savings to a forecast.
The ROI mistake: measuring the discount instead of the workflow
A cached-token discount is not the same as net savings. The workflow may also create cache-write charges, storage charges, extra requests, operational complexity, or quality regressions when teams optimize prompt structure too aggressively.
Measure the unit that matters:
Net prompt caching ROI = avoided uncached input cost - cache write/storage cost - implementation and operating cost
For production decisions, connect that result to an accepted outcome:
Cost per accepted task = total request cost / validated successful tasks
This prevents a misleading result where token spend falls but retries, rejected outputs, or human review increase. It also aligns prompt caching with a broader AI API cost optimization program instead of treating caching as an isolated billing trick.
A six-step prompt caching workflow
1. Find workloads with real prefix reuse
Start with request traces, not intuition. Group traffic by workflow and estimate how many input tokens remain identical from the beginning of one request to the next.
Good candidates usually have four properties:
- Large repeated input: the reusable prefix is material relative to the dynamic suffix.
- Frequent reuse: several requests reference the same prefix within the provider's effective cache lifetime.
- Stable ordering: system instructions, tools, examples, and reference material appear in the same order.
- Low cardinality: the application reuses a manageable number of prompt variants rather than creating a unique prefix for every user.
Typical high-potential workflows include coding agents with stable tool schemas, support assistants grounded in a shared knowledge package, document Q&A sessions, batch extraction with repeated examples, and multi-turn research agents.
Poor candidates include one-off short prompts, highly personalized prefixes, requests that change tool definitions on every call, and low-volume jobs that rarely reuse a cache entry.
Build a baseline table for each workflow:
| Metric | Why it matters |
|---|---|
| Requests per day | Determines reuse volume |
| Average input tokens | Establishes total input cost |
| Reusable prefix tokens | Defines the cacheable surface |
| Prefix variants | Reveals fragmentation |
| Reuse interval | Tests whether entries remain useful |
| Accepted-task rate | Protects quality and business value |
| P50/P95 latency | Measures performance impact |
2. Put static content before dynamic content
Prompt caching usually depends on matching the prompt from its beginning. A small difference near the front can prevent reuse for everything that follows.
Use this order where the provider and SDK permit it:
1. Stable system instructions
2. Stable policy and safety rules
3. Stable tool definitions
4. Stable reference material or examples
5. Semi-stable conversation context
6. Dynamic user input and runtime values
Do not place timestamps, request IDs, user-specific labels, randomly ordered JSON, or frequently changing feature flags near the beginning of the prompt. Normalize tool schemas and serialize structured content deterministically.
This is not permission to combine unrelated data into an oversized prefix. Keep tenant boundaries, authorization rules, and data-retention requirements intact. A cheaper prompt is not worth a privacy or isolation failure.
3. Define a cache identity and invalidation policy
Your application needs an explicit way to reason about prompt versions, even when the provider manages the cache automatically.
A practical cache identity can include:
workflow + prompt_version + tool_schema_version + knowledge_version + model_family
Track this identity in telemetry. When instructions, tool contracts, or reference data change, increment the relevant version. That makes cost changes explainable and prevents teams from confusing expected invalidation with a provider outage.
Set a reuse window based on workload behavior and provider support. A short-lived interactive agent may benefit from minutes of reuse. A recurring research workflow may justify a longer explicit cache if storage cost remains lower than repeated input processing.
4. Instrument hits, misses, writes, and accepted outcomes
At minimum, log these fields for every attempt:
- Provider, model, and workflow.
- Prompt version and cache identity.
- Total input, cached/read, cache-write, and output tokens when exposed.
- Cache hit or inferred hit status.
- Input, cache, output, and total estimated cost.
- Latency, status, retry number, and fallback path.
- Validated success or accepted-task result.
Use provider-reported usage fields as the billing source of truth when available. If a provider does not return a clean cache-hit flag, infer carefully from cached-token counts or billing records and label the metric as inferred.
Prompt caching telemetry belongs in the same trace as retries and model fallback. Otherwise a retry storm can look like a successful cache optimization. The AI observability implementation checklist shows how to connect per-attempt cost with application outcomes.
5. Calculate savings and break-even volume
Use a model that matches the provider's charging structure.
For automatic caching with a discounted read rate:
gross_savings = cache_read_tokens × (uncached_input_rate - cached_input_rate)
net_savings = gross_savings - incremental_operating_cost
For explicit caching with write and storage charges:
net_savings = avoided_uncached_cost
- cache_write_cost
- cache_storage_cost
- incremental_operating_cost
For a provider or model that charges different token rates for cache writes and reads, calculate the lifecycle of one reusable prefix:
uncached_scenario = prefix_tokens × total_uses × uncached_rate
cached_scenario = prefix_tokens × cache_writes × write_rate
+ prefix_tokens × cache_reads × read_rate
+ storage_cost
prefix_net_savings = uncached_scenario - cached_scenario
Do not assume cache_writes = 1. A changed prefix, expired entry, routing change, or explicit refresh may create another write.
You can estimate the break-even number of reuses for one cached prefix:
break_even_reuses =
(cache_write_cost + storage_cost + implementation_cost_per_entry)
/ savings_per_cache_read
Round up to the next whole reuse. Then add a margin for misses, invalidations, and traffic variability.
Worked ROI example
Assume a workflow has:
- 40,000 requests per month.
- 18,000 input tokens per request.
- A 12,000-token stable prefix.
- A 70% effective cache-hit rate.
- An uncached input rate of $3 per million tokens.
- A cached input rate of $0.30 per million tokens.
- $350 per month in amortized engineering and monitoring cost.
Monthly cached tokens:
40,000 × 12,000 × 70% = 336,000,000 cached input tokens
Gross savings:
336 million × ($3.00 - $0.30) / 1 million = $907.20
Net monthly savings:
$907.20 - $350 = $557.20
If the workflow produces 32,000 accepted tasks, caching contributes about $0.017 in savings per accepted task. That may be meaningful at scale, but the result is far more modest than simply quoting a 90% cached-input discount.
The rates above are illustrative, not a current provider quote. Replace them with your contracted or published rates and include cache writes, storage, regional pricing, service tiers, and gateway charges where applicable.
Copyable prompt caching ROI worksheet
Build the worksheet at the workflow and prompt-version level. A blended account-level hit rate can hide one profitable cache and many wasteful ones.
| Input | Symbol | Example question |
|---|---|---|
| Eligible requests | R |
How many requests could reuse this prefix? |
| Prefix tokens | P |
How many leading tokens are stable? |
| Cache reads | H |
How many eligible requests actually read cached tokens? |
| Cache writes | W |
How many times was the prefix created or extended? |
| Uncached input rate | U |
What would these tokens cost without caching? |
| Cache-read rate | C |
What does the provider charge for a hit? |
| Cache-write rate | CW |
Is creation priced at standard input or a premium? |
| Storage cost | S |
Is retention billed by token-hour or another unit? |
| Operating cost | O |
What monitoring and maintenance cost is attributable to the workflow? |
| Accepted tasks | A |
How many outputs passed the production acceptance check? |
Use these formulas:
eligible_prefix_tokens = R × P
observed_cached_tokens = H × P
baseline_prefix_cost = eligible_prefix_tokens × U
observed_prefix_cost = (H × P × C)
+ (W × P × CW)
+ S
net_savings = baseline_prefix_cost - observed_prefix_cost - O
net_savings_per_accepted_task = net_savings / A
Use consistent rate units, such as dollars per token or dollars per million tokens. If only part of a prefix is reported as cached, replace H × P with the provider-reported cached-token total.
Break-even shortcut for one cache write
When there is one initial write, no separate storage fee, and every later use is a hit:
break_even_reads = (write_rate - uncached_rate)
/ (uncached_rate - read_rate)
Round up to the next whole read. If the write rate equals the normal uncached rate, the first successful reuse creates gross savings. If writes carry a premium, additional reads are required. Real production break-even will be higher after misses, invalidations, storage, and engineering cost.
6. Roll out with a controlled experiment
Run caching as an engineering change with a measurable control group.
- Select one high-reuse workflow.
- Freeze the evaluation set and acceptance criteria.
- Establish uncached cost, latency, and quality baselines.
- Restructure only the stable prefix.
- Send a small share of production traffic through the cached path.
- Compare hit rate, cost per accepted task, P95 latency, errors, and fallbacks.
- Expand only when savings remain positive after operating cost.
Run the control and treatment on equivalent traffic. Keep the model, service tier, maximum output, sampling settings, tool set, and fallback policy fixed where possible. Otherwise a model or routing change can be mistaken for a cache benefit.
Before broad rollout, perform three deliberate invalidation tests:
- Change the prompt version and confirm the old cache is not incorrectly attributed to the new workflow.
- Change or reorder a tool schema and verify the resulting miss is visible in telemetry.
- Trigger the approved fallback path and confirm cache loss and incremental cost are attributed to the fallback attempt.
Keep retry and fallback policies separate from cache logic. A failed request may be safe to retry, unsafe to replay after partial streaming, or better served by an equivalent model. Use a defined model fallback strategy rather than treating every cache miss or timeout as the same failure.
A seven-day prompt caching ROI audit
A short production audit is more reliable than a forecast built from an advertised cache discount. The goal is not to prove that caching works in general. It is to determine whether one specific workflow, prompt version, model route, and retention policy produces repeatable value.
Create one audit row per day and keep the uncached control running throughout the test. If weekday and weekend traffic differ, extend the test until both patterns appear. Do not compare a busy treatment day with a quiet historical baseline.
| Day | Action | Evidence to capture | Decision question |
|---|---|---|---|
| 0 | Lock the experiment | Workflow ID, prompt version, model, provider route, fallback policy, acceptance test | Can another engineer reproduce the setup? |
| 1 | Measure the control | Requests, input tokens, output tokens, cost, P50/P95 latency, accepted tasks | What does the workflow cost without caching? |
| 2 | Enable limited treatment | Cache writes, reads, misses, retention mode, error rate | Are usage fields complete and correctly parsed? |
| 3 | Diagnose locality | Prefix cardinality, cache identities, miss reasons, tool-schema versions | Are misses caused by low reuse or implementation fragmentation? |
| 4 | Test invalidation | Prompt-version change, tool change, retention expiry | Does telemetry distinguish intentional invalidation from unexplained misses? |
| 5 | Test reliability | Retry and approved fallback scenarios | How much cache locality is lost during failures? |
| 6 | Reconcile economics | Baseline cost, observed cost, write/storage cost, operating cost | Is net savings positive after every relevant charge? |
| 7 | Make the decision | Quality-adjusted savings, confidence range, owner, next review date | Should the team expand, fix, or stop? |
Use matched cohorts, not account-wide averages
Assign comparable requests to control and treatment using a stable rule such as a hash of workflow ID plus tenant ID. This reduces the chance that customer mix, prompt length, or task difficulty explains the result. Exclude neither failures nor retries from cost totals; they are part of the production economics.
At minimum, segment the audit by:
- Workflow and prompt version.
- Model and provider route.
- Cache retention mode or TTL.
- Tenant class when prompts differ materially.
- Success, retry, fallback, and rejected-output outcome.
An account-wide cache-hit rate is useful for monitoring but weak for investment decisions. One high-volume workflow can hide dozens of cache identities that are continuously written and rarely read.
Add confidence and variance checks
Do not treat one day of positive savings as a rollout signal. Calculate daily net savings and inspect the range, not only the total.
daily_net_savings = daily_uncached_baseline_cost
- daily_observed_cached_cost
- daily_operating_cost
quality_adjusted_savings = daily_net_savings
/ daily_accepted_tasks
Use the median daily quality-adjusted savings as the primary summary, then report the worst day and the share of days that remained positive. A workflow with strong average savings but repeated negative days may be sensitive to traffic shape, expiry timing, or fallback behavior.
If traffic is low, define a minimum observation count before the audit begins. A practical rule is to wait until each cohort has enough accepted tasks to include normal retries, misses, and at least one retention-expiry cycle. The exact count depends on workflow variance; avoid presenting a universal sample size as statistically valid for every application.
Go, fix, or stop rubric
| Decision | Required evidence | Next action |
|---|---|---|
| Go | Net savings is positive on most measured days; cost per accepted task improves; quality, errors, and P95 latency stay within the approved guardrails | Expand traffic gradually and schedule a 30-day review |
| Fix | Gross savings exists, but writes, prefix fragmentation, expiry, or fallback cache loss makes results unstable | Repair the identified cause and rerun the same audit |
| Stop | Net savings remains negative, accepted-task cost worsens, or the workflow cannot meet quality, security, or reliability guardrails | Remove caching for this workflow and preserve the evidence |
Set stop-loss rules before launch. Examples include an unacceptable increase in rejected outputs, an error-rate regression, a material P95 latency increase, unexpected cross-tenant cache identity, or a daily spend increase above the team’s budget tolerance. Stop-loss thresholds should come from the product’s existing service objectives and risk policy, not from a generic blog benchmark.
The audit record to keep
Store the final decision beside the prompt and routing configuration, not in a disconnected spreadsheet. A useful audit record includes the owner, experiment dates, prompt hash, tool-schema hash, model route, rates used, raw usage-field mapping, accepted-task definition, exclusions, net savings, guardrail results, decision, and next review date.
This record becomes especially important when pricing, model versions, retention behavior, or fallback routes change. Reopen the decision when any assumption that materially affects writes, reads, storage, or accepted outcomes changes.
Prompt caching KPI dashboard
Track the following metrics by workflow and prompt version:
| KPI | Formula or definition | Decision signal |
|---|---|---|
| Cacheable token ratio | Reusable prefix tokens / total input tokens | Is the optimization surface large enough? |
| Cache-hit rate | Cache-read requests / eligible requests | Is reuse actually occurring? |
| Cached-token ratio | Cached input tokens / total input tokens | How much input receives the lower rate? |
| Savings per request | Uncached baseline cost - observed cost | Is each request cheaper? |
| Net savings | Gross savings - writes, storage, and operating cost | Is the project financially positive? |
| Cost per accepted task | Total cost / accepted tasks | Did quality-adjusted economics improve? |
| P95 latency delta | Cached P95 - baseline P95 | Did user-visible performance improve? |
| Miss reason rate | Misses by version, order, TTL, or provider | What should engineering fix next? |
| Cache write-to-read ratio | Cache writes / cache reads | Are entries being created too often? |
| Prefix cardinality | Distinct cache identities / eligible requests | Is personalization fragmenting reuse? |
| Fallback cache-loss rate | Fallback attempts that lose expected cache reuse / fallbacks | What reliability policy costs in cache locality |
A high hit rate with weak savings can occur when the repeated prefix is small. A low hit rate with a large prefix may still identify a valuable opportunity if prompt fragmentation can be fixed. Read the metrics together.
Common prompt caching failure modes
Dynamic values at the beginning
Timestamps, IDs, and per-user metadata near the front fragment the cache. Move them after the stable prefix where possible.
Tool schemas change between requests
Agents often rebuild or reorder tool definitions dynamically. Normalize order, remove irrelevant tools, and version the schema deliberately.
Cache entries are written but rarely reused
Explicit cache creation can cost more than it saves when traffic is sparse or the TTL is too long. Measure reuse per cache identity before extending retention.
Teams optimize tokens but ignore outputs
Output tokens, retries, and human review can dominate total cost. Continue measuring the full request and the accepted outcome.
Provider behavior is assumed to be portable
Automatic prefix caching, explicit breakpoints, storage billing, minimum prompt lengths, eligible models, and usage fields vary by provider. Build a provider adapter and keep the business metric provider-neutral.
Fallback destroys cache locality
Switching providers or model families can eliminate reuse because caches are not portable. This does not mean fallback should be disabled. It means reliability and cost need a shared policy: fail over when required, then attribute the miss and incremental cost correctly.
Provider implementation checklist
Use a provider adapter rather than forcing every implementation into one boolean cache_hit field:
| Provider pattern | ROI fields to capture | Main modeling risk |
|---|---|---|
| Automatic prefix caching | Cached tokens, uncached tokens, retention mode, cache key when supported | Prefix mismatch is invisible without versioned traces |
| Explicit breakpoints | Cache creation tokens, cache-read tokens, TTL | Too many breakpoints or writes can erase savings |
| Explicit stored context | Creation cost, cached token count, storage duration and charge | Idle retention can cost more than repeated input |
| Automatic hit/miss pricing | Cache-hit and cache-miss tokens | Routing or model changes reset locality |
Before enabling prompt caching for a model, confirm:
- Is caching automatic, explicit, or both?
- Which models and API endpoints support it?
- What minimum prompt length applies?
- How is a matching prefix defined?
- What TTL or retention options exist?
- Are cache writes, reads, and storage priced separately?
- Which response fields expose cached tokens or cache creation?
- Do service tier, region, data residency, or zero-retention settings change behavior?
- Are caches isolated by project, account, organization, or another boundary?
- What happens when the request falls back to another model or provider?
Use the official OpenAI prompt caching guide, Anthropic prompt caching documentation, Google Gemini context caching guide, and DeepSeek context caching guide for current implementation details. Pricing and model eligibility can change, so recheck these sources during every material cost review.
2026 migration check for existing OpenAI caching dashboards
If your dashboard predates GPT-5.6 support, verify that it does not collapse all non-cached prefix tokens into ordinary input cost. For supported GPT-5.6 requests, inspect cache-write usage separately, record the retention mode, and distinguish explicit cache writes from automatic cache reads. A dashboard that tracks only cached_tokens may overstate savings when cache creation carries a higher rate.
Where an AI gateway fits
A unified AI gateway does not make provider caches portable. Each provider still controls its own caching semantics and billing. A gateway can, however, give teams one place to normalize model identifiers, route eligible workloads, record provider-specific usage, compare cost per accepted task, and enforce fallback or budget policies.
Flatkey provides one OpenAI-compatible endpoint and a unified balance for access to multiple model families. That makes it easier to benchmark cached and uncached workflows without rebuilding every integration. Confirm the selected model's current caching support and provider behavior before treating a route as cache-enabled.
If you are consolidating existing clients first, use the OpenAI-compatible API gateway migration checklist and review Flatkey's current model access and pricing.
Frequently asked questions
How much can prompt caching save?
Savings depend on the reusable prefix, hit rate, provider pricing, write or storage charges, and implementation cost. Calculate net savings from observed cached tokens rather than applying the headline discount to all input tokens.
How many cache hits are needed to break even?
It depends on the write premium, read discount, storage fee, and operating cost. With a write charged at the normal uncached rate and no storage fee, the first successful read creates gross token savings. Premium writes or paid retention require more reuse. Calculate break-even from the exact rates and observed cache-write count for the selected model.
What cache-hit rate is good?
There is no universal target. A useful hit rate is one that produces positive net savings and improves or preserves cost per accepted task. Large prefixes can justify lower hit rates; small prefixes may need very high reuse.
Does prompt caching improve latency?
It can reduce input-processing latency for cache hits, but the effect depends on the provider, model, prompt size, network path, and workload. Track P50 and P95 latency rather than assuming a fixed improvement.
Should I cache the entire conversation?
Usually you should maximize a stable prefix, not blindly cache everything. Conversation turns grow and change. Keep stable instructions, tools, and reference content early, then append changing history and user input.
Can cached prompts be shared across providers?
No. Provider-side prompt caches are provider-specific. If routing changes the provider or model, treat the request as a likely cache miss unless the provider explicitly documents compatible reuse.
Is prompt caching safe for sensitive data?
Review the provider's data handling, cache isolation, retention, residency, and zero-retention terms for your account and model. Do not use cost optimization to bypass security, privacy, or tenant-isolation requirements.
Start with one repeated prefix
The best prompt caching workflow is deliberately narrow: choose one expensive, high-reuse workload; move stable content to the front; version it; measure hits, misses, latency, quality, and cost; then calculate net ROI.
When the result improves cost per accepted task, expand the pattern to the next workflow. When it does not, the telemetry will tell you whether the problem is prompt fragmentation, insufficient volume, short retention, provider pricing, or a workload that was never a good caching candidate.
