How to Reduce LLM Token Costs: 7 Practical Techniques for AI Agents (2026)
Seven practical techniques to cut LLM token costs for AI agents, ordered by effort against savings, from model routing and prompt caching to a write-time memory layer that shrinks what you send in the first place.
TL;DRAgent fleets running on Claude and similar models burn tokens fastest inside loops that re-read history on every step. Attack that spending in this order.
- Do this today: route simple tasks to cheaper model tiers (40-70% savings) and turn on prefix caching, where Anthropic bills cache reads at a 90% discount. Both need little code.
- Highest-leverage fix: a write-time memory layer stores resolved current facts instead of replayed history or raw retrieved chunks, cutting token spend by roughly 70% (a 50,000-token call drops to about 15,000).
- The dollar swing: on a 500-task/day fleet at Claude Sonnet rates, that shift moves monthly spend from about $4,160 to $1,250, nearly $35,000 saved a year.
Why agent token costs spiral out of control
An agent re-reads its entire accumulated history on every step. The original instructions, the tool schemas, every prior tool result, and all earlier reasoning get re-sent and re-billed with each call. Spheron calls this the "pricey context snowball", and it explains why cost climbs turn over turn, instead of staying flat.
The numbers make the stakes clear. Gartner's 2026 analysis found agentic AI burns 5 to 30 times more tokens per task than a standard chatbot exchange, and Stanford's SWE-bench work pushes that gap to 1000x on coding agents. A basic chat runs about 800 tokens. A single five-step fraud-check task runs around 13,500, a 17x jump driven almost entirely by input the model re-reads.
Every technique below intervenes at a different point in that loop. Some shrink what you send per call. Others cut how often you call at all. I have ordered them roughly by effort against savings, so the cheapest wins come first and the highest-leverage fix comes last.
1. Route tasks to the cheapest capable model
Model routing is the highest-return, lowest-effort change you can make, because most agent work never needs a frontier model. A cheap classifier reads the incoming task, scores its difficulty, and sends it to the smallest model that can handle it. Simple extraction and formatting go to fast tiers like Claude Haiku, GPT-4o Mini, or Gemini Flash at roughly $0.15 to $1 per million tokens. Moderate reasoning goes to Claude Sonnet or GPT-4o at around $2.50 to $3, and only genuinely hard analysis reaches Claude Opus or GPT-5.2 Pro at $15 to $75.
That pricing spread is where the money is. MindStudio estimates 60 to 70% of production traffic can run on the cheapest tier, cutting per-token cost 10 to 50 times on that share. Morph reports 40 to 70% savings from routing alone, and a plan-and-execute split (frontier planner, cheap executor) cut cost 57% on a 14M-token build with no drop in planner quality.
The classifier itself is nearly free. In one production example, the routing call cost $0.001 and added about 430ms. You pay a fraction of a cent to avoid overpaying by 30 times.
2. Turn on prompt and response caching
Caching is the fastest win with the least code. Every provider bills tokens it has already processed at a steep discount, so an agent that repeats the same system prompt and tool schemas on every step pays full price only once. Anthropic charges roughly 90% less on cache reads, dropping Sonnet cached input from $3 to $0.30 per million tokens, with a small write premium on the first call (Spheron).
The rule that makes it work is prefix ordering. Only a shared prefix is cacheable, so put static content first and dynamic content last. Your system prompt and reference docs go at the top, the user query and conversation history go at the bottom. Reverse that order and the cache never hits.
Agents with stable system prompts commonly see 60-80% cache hit rates once ordering is fixed (Morph). OpenAI removes even the ordering work. Its caching is automatic for shared prefixes of at least 1,024 tokens at roughly a 50% discount, so you get the saving with no code changes at all.
3. Prune context and stop over-retrieving in RAG
Most context bloat comes from two habits. Your agent keeps every prior turn in the prompt, and your retrieval step over-fetches. Both re-bill the same tokens on every call.
Fix the history first. Rolling summarization keeps the last few turns verbatim and compresses older ones, shrinking a 20-turn history from 8,000-15,000 tokens to under 2,000, a 75-85% cut (MindStudio). Verbatim compaction goes further by deleting boilerplate while leaving surviving text character-for-character intact, so you get 50-70% reduction with zero paraphrase-driven hallucination (Morph).
Then discipline retrieval. Pulling 10 chunks when 2 would do inflates input tokens 3-4x, and a logging layer that stores full prompts can double consumption outright (Silicon Data). Capping retrieval to 2-3 chunks often cuts input tokens by more than half with no precision loss. A researcher agent that re-sends a 25,000-token PDF across 5 refinement steps burns 125,000+ input tokens per task, over $600 across 1,000 documents (Fastio).
Trim proactively before every call, not when you hit the context limit. Wait, and your agent has already paid full price for 100+ turns of bloated context.
4. Budget and cap output tokens
Output tokens deserve separate discipline because they cost 3 to 10x more than input tokens, and premium reasoning models push the ratio to 8x. Claude Sonnet bills $3 per million input tokens against $15 for output. Every verbose response, every unnecessary pleasantry, gets billed at the expensive rate.
Three controls cut this cheaply. First, ask for structured JSON instead of prose. MindStudio measured a JSON response at roughly 40% fewer tokens than the equivalent description. Second, set hard max_tokens caps and explicit brevity instructions so a runaway generation cannot balloon. Third, tier the thinking budget by task difficulty. A simple lookup needs zero reasoning tokens, while hard math or code justifies 10,000 to 16,000. MindStudio reports 50 to 75% reduction in thinking-token costs from this tiering, with minimal quality loss on routine work.
5. Cut redundant and retried agent calls
Retry loops are the quiet multiplier that wrecks a token budget. When an agent botches one sub-task and self-corrects, two or three retries on that step can outweigh the token cost of the entire rest of the task combined (Spheron). The damage compounds because every retry re-reads the accumulated history that led up to it.
Early bloat pays the same tax. A mere 100 wasted tokens in turn one of a 30-turn session get re-billed on every subsequent turn, costing roughly $0.015 per session. Across 1,000 sessions a day, that single slip of 100 tokens adds ~$450 a month (Morph).
Monitoring is the guardrail that catches this before it runs away. Track cost per session, the input/output token ratio, and cache hit rate. Set soft budget alerts at 50% and 80%, then hard caps at 100% to stop a looping agent mid-run (Fastio).
6. Batch and reuse work across requests
Any workload that tolerates a delay can drop its token cost in half with almost no engineering. Anthropic and OpenAI both offer batch APIs that run requests asynchronously and return results within 24 hours, in exchange for a flat 50% discount (Morph). Overnight document processing, bulk classification, and evaluation runs fit this pattern cleanly.
The discount stacks with caching. When batched requests share a cached prefix, the 50% batch reduction combines with the roughly 90% cache-read discount for up to 95% off that portion of the spend (Morph).
The constraint is latency. Batching only works where nobody waits on the answer in real time, so route anything user-facing through the standard synchronous API and reserve batches for background jobs.
7. Replace re-sent context with a write-time memory layer
Every technique above trims what you send per call. A write-time memory layer changes how much there is to send at all. Instead of replaying raw chat history or dumping retrieved chunks into the prompt, it resolves each new exchange as it arrives. An LLM extracts the salient facts, compares them against what is already stored, and issues one of four operations: ADD a new fact, UPDATE a changed one, DELETE a retired one, or NOOP when nothing changed. Only the current resolved fact gets stored, so inference calls carry the fact and when it changed rather than the full transcript.
Query-time RAG never resolves anything at write time, and that is where its cost comes from. Because every historical version of a fact stays in the index, a search for "deployment process" can return last quarter's version alongside this week's. The model then pays full token cost to read contradicted material it has to reason around. Sentra resolves the conflict before storage, so the prompt never carries the stale version.
Sentra cuts token spend by roughly 70%, dropping a 50,000-token call to about 15,000 while holding around 88% on Terminal-Bench 2.1. Run the math on a 500-task-per-day fleet at $3 per million input tokens with a 1.85x overhead multiplier for retries and schemas. Before, 750 million tokens a month cost about $4,160. After, 225 million tokens cost about $1,250. That delta is nearly $35,000 a year on a single moderate fleet.
Mem0 follows the same extract-then-resolve loop and reports over 90% token savings on the LOCOMO benchmark versus full context, plus a 26% quality gain over OpenAI's baseline. The divergence is temporal modeling. Mem0 stores created_at and updated_at timestamps that record when a row was touched, not when a fact was true in the world. Sentra's bi-temporal model separates valid-time from transaction-time, so a shared org-wide graph can answer what the shipping policy was on March 1st even after two revisions since.
Comparison table: impact, difficulty, and best use case
The seven techniques sort cleanly by how much they cut and how much work they demand. Caching and batching return money on almost no code. The write-time memory layer sits at the top because it shrinks what you send in the first place, not just how you send it.
| Technique | Token/Cost Impact | Difficulty | Best For |
|---|---|---|---|
| Model routing | 40-70% | Low | Mixed traffic where most tasks are simple |
| Prompt/response caching | 50-90% on cache reads | Low | Stable system prompts, high repeat prefixes |
| Context pruning + RAG discipline | 50-98% | Medium | Long sessions and document-heavy retrieval |
| Output token budgeting | 40-75% on output | Low | Verbose or over-reasoning responses |
| Cutting redundant/retried calls | Varies | Medium | Looping agents and multi-step tasks |
| Batching | 50% flat (95% with cache) | Low | Non-real-time, latency-tolerant jobs |
| Write-time memory layer | ~70% | Medium | Agent fleets replaying history or chunks |
How to prioritize these techniques
If you have limited engineering time, start with model routing and caching. Both cost almost nothing to add. A router is one cheap classification call, and prefix caching often needs no code at all on OpenAI. Together they can cut costs 40 to 70% within a day, before you touch your architecture.
Once those wins are banked, move to the context and memory fixes. Prune retrieval, cap output tokens, and batch async work where latency allows. These pay off most on long-running agent loops where waste compounds turn over turn.
Techniques 1 through 6 optimize what you send on each call. They shrink the payload but leave the underlying volume intact. A write-time memory layer like Sentra changes how much there is to send in the first place by storing resolved facts instead of raw history. That makes it the ceiling on savings. Once the cheap wins are exhausted, it is the lever that keeps cutting.