A 10x reduction in LLM token spend is a target, not a guarantee. It is plausible when the baseline sends long repeated prompts to a frontier model, generates excessive output, and retries failed work without measurement. A workload already using short prompts, cached context, and a low-cost model may have nowhere near 90% waste to remove.
Provider prices and product specifications cited here are as of 2026-07-20 and may change.
The safe objective is not “use fewer tokens at any cost.” It is lower cost per accepted result while holding quality and safety thresholds constant.
1. Measure the bill by task
Start with one week of request-level records:
- task type and route;
- model and service tier;
- uncached input tokens;
- cached input tokens;
- output and reasoning tokens when exposed;
- retries and tool calls;
- latency and cost;
- pass/fail against an evaluation gate.
Rank task types by total cost, then calculate cost per accepted result. Optimizing a high-volume classification route usually matters more than trimming a rare complex analysis.
2. Route cheap-model-first, with a quality gate
Send routine extraction, classification, formatting, and short summaries to the least expensive model that passes your evals. Escalate only when the small model is uncertain, violates schema, or fails a task-specific check. Reserve the frontier model for ambiguous planning, difficult reasoning, or final review.
Do not route by prompt length alone. A short security question can be harder than a long formatting task. Use task class, required tools, risk, and eval history. AWS describes multi-model routing as selecting a model for cost and quality rather than using one model for every task, while warning that the benefit depends on the workload (AWS routing strategies).
3. Delegate narrowly instead of copying the whole context
Parallel agents can reduce wall-clock time but increase total tokens because every worker has its own context and tool loop. Delegate only independent tasks and pass each worker the smallest sufficient packet: objective, relevant files, constraints, and output schema.
Return a compact result to the parent rather than full logs. Keep one agent responsible for integration. If three agents all read the same 100,000-token repository snapshot, delegation has tripled the expensive part before producing an answer.
4. Make repeated prefixes cacheable
Put stable material first: system instructions, tool definitions, schemas, and shared reference documents. Put request-specific content later. Keep the prefix byte-for-byte stable where the provider’s cache semantics require it, and monitor cache-hit tokens rather than assuming a hit.
Google recommends placing large common content at the beginning and sending similar prefixes close together; its Gemini API reports cached tokens in usage metadata (Gemini context caching). Anthropic’s pricing documentation lists cache reads at a fraction of base input cost, while cache writes cost more than normal input, so reuse frequency matters (Anthropic pricing). OpenAI model pricing likewise separates normal and cached input, and the discount varies by model (OpenAI model comparison).
Caching is strongest when a large prefix is reused many times. It does little for unique prompts, frequently changing tool lists, or one-off documents.
5. Batch work that does not need an immediate response
Move nightly classification, embeddings, dataset labeling, and bulk summarization to asynchronous batch endpoints. OpenAI’s Batch API documents a 24-hour completion window and a 50% discount compared with synchronous processing (Batch API reference).
Batching is not appropriate for interactive chat or urgent incidents. Split jobs into retryable records, assign stable IDs, and verify partial failures before treating a batch as complete.
6. Stop paying for irrelevant context
Common context leaks include full chat histories, duplicate policies, entire repositories, raw test logs, and tool schemas that are never used. Replace them with retrieval and summaries:
- retrieve only the relevant files or chunks;
- summarize completed phases and preserve decisions, not every message;
- send failing tests, not a full successful suite log;
- load tools on demand;
- remove duplicated instructions from nested prompts;
- cap retrieved chunks and reject low-relevance matches.
Track recall on a test set. Aggressive trimming that removes the decisive fact simply converts input savings into retries and bad outputs.
7. Control output tokens
Output is often more expensive than input. Ask for the shortest artifact that the next stage needs: a JSON decision, a patch, a ranked list, or a bounded explanation. Use structured outputs and explicit field limits. Do not request chain-of-thought or long rationales when evidence spans and a concise decision are sufficient.
For agent workflows, stop after the acceptance criteria pass. Repeated self-review without a new hypothesis can burn tokens while adding no value. Set maximum turns, tool calls, retries, and wall time, then route unresolved cases to a stronger model or a person.
8. Cache final results outside the model
Prompt caching reuses model computation; application caching skips the request entirely. Cache deterministic or slow-changing results under a key that includes input hash, prompt version, model version, retrieval corpus version, and safety policy version.
Do not cache personalized, permission-sensitive, or rapidly changing answers without correct scoping and expiry. A stale answer can cost more than the saved call.
9. Prevent retry storms
Classify failures before retrying:
- transient transport or rate-limit error: exponential backoff with jitter;
- invalid structured output: one constrained repair attempt;
- missing context: retrieve the missing evidence;
- capability failure: escalate model or human review;
- policy denial: stop rather than retrying around the policy.
Deduplicate in-flight requests with idempotency keys. Cap retries globally and per task. Log the original failure so a second attempt is not mistaken for independent work.
A transparent 10x scenario
Consider a baseline costing $1,000 for a fixed evaluated workload. The following factors are illustrative and must be measured, not multiplied blindly:
| Change | Remaining cost from prior step |
|---|---|
| Route 70% of eligible work to a model averaging one-fifth the cost | $440 |
| Achieve a 60% cache discount on half of the remaining input-heavy spend | $308 |
| Batch half of eligible asynchronous work at a 50% discount | $231 |
| Cut irrelevant context and output by 40% | $139 |
| Remove duplicate calls and failed retries by 25% | $104 |
That example approaches 10x because the baseline is deliberately inefficient and the optimizations affect different portions of spend. Real savings are not independent: routing may change cache behavior, shorter prompts may reduce the cacheable share, and batch eligibility may be small. Use a cost simulator built from actual request traces.
A 30-day rollout
Week 1: instrument requests and build a representative eval set. Week 2: route one low-risk task class to a cheaper model and add fallback thresholds. Week 3: stabilize prompt prefixes, measure cache hits, and move one non-urgent job to batch. Week 4: trim context and output, add retry caps, and compare cost per accepted result with the baseline.
Roll back any change that violates the quality gate. Keep model, prompt, cache, and routing versions in the logs so the result can be reproduced.
Bottom line
The largest savings usually come from architecture: avoid unnecessary calls, use the cheapest model that passes, reuse stable context, batch non-urgent work, and stop generating text nobody consumes. The title’s 10x target is achievable for some wasteful baselines, but it should never be promised before measurement. Optimize against verified outcomes, not a lower token counter alone.