Prompt Caching vs a Memory Layer: Which Actually Cuts LLM Costs?

TL;DR

Prompt caching reduces the price of resending the same context. A memory layer reduces how much context you need to send at all. They solve different halves of the same bill and they compose well: cache the stable prefix, shrink the variable part. If your prompts are large and repetitive within a short window, caching is the faster win and needs no architecture change. If the same information is being reassembled from scratch by different agents, caching cannot help you, because it only rewards sending identical bytes. This guide covers the published economics of each and how to sequence them.

What does prompt caching actually do?

Both Anthropic and OpenAI offer caching that stores the processed form of a prompt prefix so repeated requests skip re-processing it. The mechanics that determine whether it helps you:

1. It matches on an exact prefix. The cached portion must be byte-identical and at the start of the prompt. Change one character near the beginning and the cache misses entirely.

2. Cached reads are billed at a reduced rate, not free. Anthropic's pricing documentation prices cache reads at a fraction of base input tokens, with a write premium the first time. Check current published rates rather than trusting a number in any article, including this one.

3. Caches expire. Anthropic documents a short default time to live with a longer option available. A prompt reused hourly benefits far less than one reused every few seconds.

4. It does nothing for output tokens. Only input is cached, so a workload dominated by generation sees little movement.

The practical shape: caching is excellent for a long system prompt or a large document that many consecutive requests share, and close to useless for a workload where each request assembles different context.

What does a memory layer do to the same bill?

A memory layer attacks the size of the payload rather than its price. Instead of retrieving the documents that mention a topic and pasting them in, it resolves the underlying facts and sends those.

The difference is concrete. A question about a customer's current contract terms might retrieve six documents totalling several thousand tokens, of which the answer occupies two sentences. A compiled fact carries the two sentences and a provenance pointer. Sentra's internal measurements put the reduction at roughly 70 percent on realistic workloads, and internal measurements are exactly that, so treat the figure as directional until independently reproduced.

Three properties make the reduction durable rather than a one-off:

1. It scales with corpus growth, not against it. Retrieval payloads grow as the corpus grows, because more documents match. Compiled facts do not.

2. It removes contradictory content from the payload. Retrieval happily returns a superseded policy alongside its replacement, and the model pays for both and sometimes picks the wrong one.

3. It is shared across agents. The same compiled fact serves the coding agent and the support agent, so the assembly cost is paid once.

Which techniques should you apply, in what order?

Ranked by effort against likely saving, for a team whose inference bill has become visible:

1. Measure where the tokens go. Split the bill into system prompt, retrieved context, conversation history and output. Most teams discover one of these dominates and it is rarely the one they assumed. This step costs a day and redirects everything after it.

2. Trim the system prompt. Long instruction files are paid on every request. Deleting rules that no longer apply is free and immediate.

3. Turn on prompt caching for stable prefixes. No architecture change, and the win is proportional to how repetitive your traffic is within the cache window.

4. Cap conversation history. Sending the full transcript every turn grows cost quadratically over a long session. Summarise or window it.

5. Route by difficulty. Send easy requests to a smaller model. This is often the single largest line-item reduction and the one teams resist longest.

6. Batch what is not interactive. Both major providers discount asynchronous batch processing substantially for work that can tolerate delay.

7. Replace retrieval with compiled facts. The largest structural reduction and the largest project. Worth it when retrieval payloads dominate and multiple agents need the same information.

How the two approaches compare

The row worth pausing on is the fourth. Caching is indifferent to whether the content it stores is correct. A cached prefix containing a policy that changed last month is served faster and cheaper, and is still wrong. Cost optimization that speeds up the delivery of stale context is not a saving.

FAQ

The decision rule

Reach for prompt caching when the same bytes are being sent repeatedly within a short window, because it is nearly free to adopt. Reach for a memory layer when the same knowledge is being reassembled differently by different agents, because that is the case caching structurally cannot address.