You do not always need a faster model to make an LLM application feel faster. In many systems, the larger delay comes from long answers, repeated model round trips, serial tool calls, and a user interface that waits for the entire response before showing anything.
The practical sequence is to measure the full request path, reduce unnecessary generation, stream the useful part first, and collapse or parallelize work that does not need to be sequential. Keep the model fixed while making these changes so that you can attribute any latency and quality difference to the application rather than to a model swap.
API behavior in this article was checked against official OpenAI and Anthropic documentation on 2026-07-24. Provider parameters and event formats can change, so confirm them again before production use.
Measure the latency budget before optimizing
Start with a trace that separates the stages a user experiences. A single “request duration” metric hides the location of the delay.
Record at least:
- queue and connection time before the provider accepts the request;
- time to first response byte or first meaningful output token;
- generation duration from first token to final token;
- output token count and, when supplied, reasoning-token count;
- each retrieval, database, or external tool duration;
- time spent waiting between model and tool turns;
- client rendering and post-processing time;
- cancellation, retry, and timeout status.
Use both user-facing and system-facing metrics. Time to first useful content measures when the user can begin acting. Time to completion measures when the application has the full answer. For output generation, track tokens per second as:
visible output tokens / (last-token time - first-token time)
Do not average everything into one number. Report medians plus tail latency such as p95, grouped by route, output class, tool path, and input-size band. A fast FAQ answer and a five-tool research workflow are different workloads.
Create a small, fixed test set from representative requests. Run the same cases before and after every change, using the same model, model version where pinning is available, prompt, region, and concurrency. Separate cold starts and retries rather than silently mixing them into ordinary runs. If concurrency or provider limits distort the results, use the rate-limit and concurrency guide to identify whether requests per minute, tokens per minute, or in-flight work is binding.
Match the optimization to the measured symptom
The best first change depends on where the trace shows time accumulating. Use this table as a triage guide, then confirm the result with the fixed test set rather than assuming the expected effect will appear.
| Measured symptom | First change to test | Primary metric | Main tradeoff or failure mode |
|---|---|---|---|
| Long time after the first token | Reduce requested output and remove unused fields | completion time, output tokens | required detail may be omitted |
| Long blank wait, normal completion time | Stream and place the decision first | first useful content | partial-stream and reconnect handling |
| Several independent tools dominate | Run read-only, independent calls concurrently | tool-stage wall time, p95 | throttling and partial failures |
| Many model–tool round trips | Batch tool calls or combine decisions | model turns per task | larger turns can be harder to debug |
| Fast median, slow p95 | Bound concurrency and inspect retries or queues | p95, retry rate, queue time | lower peak throughput |
| Frequent truncation at the token cap | Tighten the response contract before changing the cap | incomplete-response rate | a higher cap can hide prompt problems |
Shorten generated output without removing the answer
Output length is often the largest adjustable part of end-to-end latency. OpenAI’s official latency guide says generation is usually the highest-latency LLM step and offers a directional heuristic: cutting output tokens by 50% may cut generation latency by roughly 50%. That is not a universal benchmark; validate the relationship on your workload (official latency guide).
Replace vague requests such as “explain this thoroughly” with an explicit response contract:
- state the decision in the first sentence;
- give no more than three supporting points;
- omit background the user already supplied;
- return only the fields the application renders;
- put optional detail behind a follow-up action.
For structured output, remove fields no consumer reads. Prefer compact enum values and avoid asking the model to repeat source text, tool output, or the original question. Do not shorten field names so aggressively that observability and maintenance suffer; a few saved tokens are not worth an opaque schema.
Set an output ceiling as a guardrail, not as the primary instruction. OpenAI’s current Responses API defines max_output_tokens as an upper bound that includes visible output and reasoning tokens (official Responses API reference). Anthropic’s Messages API requires max_tokens and may stop before reaching it (official Messages API reference). A hard cap can cut off JSON, citations, or a final recommendation, so test the stop reason and incomplete-output path.
A useful pattern is progressive disclosure: return the conclusion and essential actions now, then generate optional explanation only if the user requests it. This reduces routine output without denying detail when it matters.
Stream useful content first
Streaming mainly improves perceived latency: the model may finish at a similar time, but the user does not stare at an empty interface. OpenAI documents HTTP streaming through server-sent events and notes that it lets a client print or process the beginning while the rest is still being generated (official streaming guide). Anthropic likewise documents incremental Messages API events (official streaming guide). For an implementation-focused walkthrough, see how to stream LLM responses.
Do more than switch on stream. Design the response so the first content is useful:
- Put the answer, status, or recommended action before the explanation.
- Render complete text fragments instead of repainting the whole response for every token.
- Buffer structured data until a field or object is valid.
- Show an explicit tool or retrieval status when no text can be emitted yet.
- Let the user cancel a response whose useful answer has already arrived.
Measure first-token time and first-useful-content time separately. A stream that emits punctuation, a heading, or internal scaffolding early can look healthy in provider metrics while still feeling slow.
Streaming also changes failure handling. The client must cope with a connection ending after partial text, duplicated events after a reconnect, and a final usage event arriving later than visible content. Mark partial responses as incomplete and make retries idempotent where possible. For safety-critical or schema-constrained actions, do not execute from an unfinished stream.
Tune sampling and stop rules conservatively
Sampling parameters primarily control output distribution, not application architecture. Lowering temperature does not guarantee lower latency. It can make deterministic tasks more stable, which may reduce retries or verbose variation, but that effect must be measured rather than assumed.
Change one parameter at a time. For extraction, classification, and fixed-format generation, begin with a low-variance configuration and a strict output schema. For creative text, keep enough variation to meet the product requirement. Avoid changing both temperature and top_p in the same experiment because you will not know which change caused the result.
Use stop rules only when the endpoint and task support them. Good candidates include a known record delimiter, the end of a single generated section, or a protocol marker that cannot occur in ordinary content. Poor candidates include common punctuation and brittle natural-language phrases.
Every early-stop path needs three checks:
- Was the required answer present?
- Was the output syntactically complete?
- Did the provider report a normal stop, a token limit, a stop sequence, a refusal, or an error?
Treat a token limit as an operational boundary, not a successful completion. If the response ends at the cap, either request a smaller response contract, continue through an explicit pagination protocol, or surface an incomplete state. The token-budget and circuit-breaker guide covers hard ceilings for multi-step systems.
Parallelize independent work
Draw the request as a dependency graph. Two operations can run together only when neither consumes the other’s result and they do not create unsafe side effects.
Common parallel candidates include:
- retrieving from independent indexes;
- fetching an account-safe profile while loading public product data;
- running independent policy and format checks;
- calculating deterministic metrics while the model drafts an explanation;
- requesting independent classifications that will later be combined.
If tasks A, B, and C are independent, total stage time approaches max(A, B, C) plus orchestration overhead instead of A + B + C. That is an architectural estimate, not a promise: provider queues, rate limits, connection pools, and shared databases can erase the gain.
Use bounded concurrency. An unlimited fan-out can trade one slow request for throttling, retries, and a worse p95. OpenAI documents limits across request, token, and other usage dimensions, while Anthropic documents organization-level limits and acceleration limits; the exact dimensions and headers depend on the provider and endpoint (OpenAI rate limits, Anthropic rate limits, both checked 2026-07-24). Attach a deadline and cancellation signal to every branch, and decide whether one branch failing should fail the whole request or produce a partial result. Read-only calls are usually easier to parallelize than writes.
OpenAI’s official latency guide explicitly recommends parallelizing steps that are not strictly sequential and also notes that every separate request adds round-trip latency (official latency guide). Measure both effects: splitting one model request into several parallel requests may lower wall-clock time while increasing tokens, cost, and rate-limit pressure.
Remove serial tool calls and extra model turns
Tool-enabled applications often lose more time between generations than during generation. A loop of “model → one tool → model → one tool” pays for repeated network, queue, prompt-processing, and orchestration stages.
First, remove calls whose results are already in context or can be computed locally. Next, combine dependent model decisions when one structured response can safely return both. For example, query rewriting and a boolean retrieval decision may fit in one response instead of two.
Then batch independent tool requests. Anthropic’s current tool-use documentation says one assistant turn can contain several tool_use blocks and leaves concurrent versus sequential execution to the application. It recommends parallel execution for independent read-only operations while keeping side-effecting or ordered work sequential (official parallel tool-use guide).
Do not parallelize merely because the API can represent multiple calls. Keep these serial:
- a write followed by a read that must observe it;
- authorization before a privileged action;
- payment, publication, deletion, or other approval-gated changes;
- operations that mutate the same record;
- a fallback that should run only after the primary attempt fails.
Return compact tool results. Large HTML pages, duplicate database rows, and verbose logs increase the next request’s input and can obscure the facts the model needs. Normalize errors into a stable shape with tool name, status, retryability, and a short message. If tool calls are malformed or repeatedly retried, diagnose that loop with the guide to fixing LLM tool-calling errors before adding concurrency.
Verify quality after every latency change
A latency improvement is acceptable only if the output still satisfies the task. Build a paired evaluation: the original configuration and one changed configuration run against the same cases.
Track:
| Area | Example check |
|---|---|
| Latency | first useful content, completion time, p50, p95 |
| Completeness | required sections or fields are present |
| Correctness | claims match references or deterministic checks |
| Structure | schema parses and stop reason is acceptable |
| Tools | required calls occurred, duplicates did not |
| Safety | approvals and policy gates still block correctly |
| Cost | input, output, retry, and tool usage per completed task |
Set non-negotiable gates before looking at the speed result. A 30% faster response that drops required citations or executes tools in the wrong order is a regression. For a reusable release check, adapt the self-hosted evaluation gate rather than relying on a few hand-picked prompts.
Roll out one change at a time: output contract, then streaming, then request consolidation, then bounded parallelism. Observe production-like traffic by route and keep a rollback switch. Recheck tail latency under load because an optimization that helps an isolated request may hurt the system once concurrency rises.
Frequently asked questions
Does streaming make the model generate tokens faster?
Not necessarily. Streaming mainly reduces the wait before users can see and act on partial output. Measure both first useful content and total completion time to distinguish perceived improvement from faster generation.
Does lowering temperature reduce LLM latency?
There is no general guarantee. Lower variance can reduce retries or unexpected verbosity in some constrained tasks, but test it on the same prompts and change only one sampling parameter at a time.
What should I optimize first: the prompt or max_output_tokens?
Start with the response contract: specify the required fields, order, and level of detail. Use the token limit as a safety ceiling, because lowering it alone can truncate JSON, citations, or the final recommendation.
When should LLM tool calls run in parallel?
Parallelize only calls that are independent and safe to execute together, usually read-only lookups. Keep authorization checks, dependent operations, writes to shared state, and approval-gated actions sequential.
Why did parallelization make p95 latency worse?
The extra concurrency may be exhausting request or token limits, a connection pool, or a shared downstream service, which can trigger queues and retries. Compare concurrency, retry rate, rate-limit responses, and tool-stage p95 before increasing fan-out. The timeout, retry, and idempotency guide covers safe retry behavior.
The final operating rule is simple: optimize the critical path, not the model label. Shorter useful answers, earlier rendering, fewer round trips, and dependency-aware tool execution can reduce user-visible delay while preserving the same model and a testable quality bar.