An LLM API timeout does not tell you that the request failed. It tells you that the client stopped waiting. The provider may have rejected the request, may still be generating it, or may have completed it after the connection disappeared. That ambiguity is why a production retry policy needs more than “try again three times.”
The safe pattern is to give each user operation one stable application-level idempotency key, use explicit connect and read timeouts, retry only transient failures within a total deadline, and commit external side effects through a separate duplicate-proof execution layer. Treat generated text as repeatable work, but never assume that sending the same prompt will produce byte-for-byte identical output.
Provider and infrastructure behavior in this article was checked against official documentation on 2026-07-24. In particular, the current OpenAI Python SDK documentation says that it retries selected failures twice by default and uses a ten-minute default request timeout. These are library defaults, not recommended values for every application, and should be rechecked before publication (official OpenAI Python SDK).
Identify failure modes before writing retry code
Start by separating failures according to what the client actually knows:
| Observation | What may have happened | Default action |
|---|---|---|
| DNS, connection, or TLS failure before any request bytes were sent | The provider probably did not receive the operation | Retry within the operation deadline |
HTTP 408, 429, or 5xx response | A transient timeout, capacity limit, or server failure may clear | Retry if the request is safe and the budget remains |
HTTP 400, 401, 403, or validation error | The same request is unlikely to succeed unchanged | Do not retry; fix input, credentials, or permissions |
| Read timeout after the request was sent | The provider may still be processing or may have completed | Mark the outcome unknown, then reconcile or retry idempotently |
| Stream disconnect after partial output | Some output reached the client, but completion state is unknown | Preserve the partial state; resume only if the API supports it, otherwise restart deliberately |
| Model returned a tool call and the client lost the next response | The tool may or may not have run | Query the tool-operation ledger before executing |
Google Cloud’s retry guidance classifies socket timeouts, TCP disconnects, HTTP 408, 429, and 5xx responses as generally transient, while stressing that response class and operation idempotency must both be considered (official retry strategy). An LLM provider’s documented error contract remains authoritative for its API.
Set connect, read, and total timeouts separately
A single timeout hides three different controls:
- Connect timeout: how long to wait for DNS, TCP, TLS, and connection-pool acquisition.
- Read timeout: how long the connection may be silent while waiting for response data.
- Total operation deadline: the maximum wall-clock time for the original attempt, all retry delays, and every retry.
Keep the connect timeout relatively short because a connection that cannot be established rarely benefits from occupying a worker for minutes. Set the read timeout from measured latency for the specific route, model, prompt size, output limit, and streaming mode. Then put a total deadline above the whole retry loop so several individually valid attempts cannot outlive the user request or background-job lease.
Do not copy an SDK default without examining it. As verified on 2026-07-24, the OpenAI Python SDK documents a ten-minute default request timeout, supports a simple numeric timeout or granular httpx.Timeout values, raises APITimeoutError on timeout, and notes that timed-out requests are retried under its default retry policy (official SDK timeout documentation). An interactive endpoint may need a much smaller application deadline, while a long background generation may legitimately need more time.
Choose thresholds from telemetry rather than averages alone. Measure p50, p95, and p99 latency by workload class, with separate budgets for interactive, batch, and tool-using requests.
Timeout and retry controls at a glance
| Control | Scope | Set it from | Guardrail |
|---|---|---|---|
| Connect timeout | One connection attempt | Normal DNS, TCP, TLS, and pool-acquisition latency | Keep it below the total deadline |
| Read timeout | One request or stream | Measured tail latency for the specific workload | Do not mistake silence for confirmed failure |
| Total deadline | Original attempt plus every delay and retry | User or job latency budget | Stop starting attempts that cannot finish inside it |
| Maximum attempts | All attempts across SDK, application, and queue layers | Failure rate, cost budget, and deadline | Count hidden SDK retries |
| Backoff cap | Delay before another attempt | Provider guidance and service-recovery behavior | Add jitter and cap aggregate retry traffic |
| Idempotency retention | Lifetime of one logical-operation record | Longest client, queue, and reconciliation window | Reject the same key with different input |
These are design controls, not universal numeric defaults. Record the chosen values by workload class and revisit them when models, providers, prompt sizes, or output limits change.
Retry only safe requests with bounded backoff
Retry at one layer. If an SDK retries twice, an application wrapper retries three times, and a queue retries the job, one user operation can multiply into many provider calls. Either configure the SDK as the retry owner or disable its automatic retries and own the policy in your application.
The OpenAI Python SDK currently documents two automatic retries for connection errors, 408, 409, 429, and 5xx responses, using short exponential backoff (official SDK retry documentation). If you add application retries, count those hidden SDK attempts in the total attempt and cost budget.
Use capped exponential backoff with jitter:
delay = random(0, min(cap, base × 2^retry_number))Honor a valid provider retry delay when documented, then apply your total deadline. The HTTP Semantics standard defines Retry-After as either an HTTP date or a non-negative number of seconds; for 503, it indicates how long the service is expected to remain unavailable. A client still needs to reject malformed values and avoid waiting past its own deadline (checked 2026-07-24, IETF RFC 9110, section 10.2.3).
Jitter prevents a fleet of clients from retrying in lockstep. AWS’s Builders’ Library explains that retries can increase load on an already overloaded dependency, recommends backoff and jitter, and warns that retries at multiple layers multiply dramatically (official AWS guidance).
Define a retry matrix in code and tests. Do not retry authentication, permission, invalid-parameter, safety-policy, or context-limit errors with unchanged input. Cap attempts, elapsed time, and retry traffic. When the budget ends, return failed_transient or outcome_unknown; do not label every timeout as a clean failure.
For broader capacity planning around throttling, see the AI API rate-limit calculator guide. For system-level degradation beyond one request, use the separate LLM fallback strategy.
Add idempotency keys at the application boundary
An idempotency key identifies one logical operation across repeated HTTP attempts. Generate it when accepting the user action, persist it before calling the model, and reuse it for every retry of that same operation. A refresh or queue redelivery should recover the stored key rather than create a new one.
A minimal operation record can contain:
key, request_fingerprint, status, lease_owner, lease_expires_at,
provider_request_id, result_location, created_at, completed_atInsert the key under a unique database constraint. On a duplicate:
- Return the stored result if the operation is complete.
- Return or poll the existing operation if it is still running.
- Reclaim it only if its lease expired and the recovery policy permits another attempt.
- Reject the key if the new request fingerprint differs from the original.
The fingerprint should cover normalized inputs that define the operation: tenant-safe scope, prompt-template version, model policy, relevant parameters, and tool permissions. It should not contain secrets or raw personal data.
This design follows the established idempotent-request contract: Stripe’s official API reference describes a client-generated unique key that lets the server recognize retries and avoid performing the same operation twice (official idempotency documentation). The exact header, retention period, and response behavior are API-specific. Do not send an arbitrary Idempotency-Key header to an LLM provider and assume it is honored unless that endpoint’s current official documentation says so. [VERIFY] provider-level idempotency support for the exact endpoint before relying on it.
Idempotency means repeated execution reaches one intended state; it does not mean deterministic text. Two LLM calls may produce different valid responses. Your application should select and persist one canonical result for the logical operation.
Handle streaming failures as incomplete transactions
Streaming improves perceived latency but exposes a new failure boundary. Once the client has rendered tokens, silently restarting and appending a second generation can duplicate phrases or combine two different completions.
Track the stream as an ordered sequence associated with the operation key. Persist its state as started, streaming, complete, cancelled, or unknown. If the provider supplies event IDs, sequence numbers, or a documented resume mechanism, store and use them. Otherwise, assume a broken stream cannot be resumed exactly.
After a disconnect, either discard the uncommitted partial output and restart, show it as incomplete, or produce a new complete answer that clearly replaces it.
Do not present partial JSON, half a tool call, or a truncated structured response as valid. Buffer structured output until it passes schema validation. For plain text, the interface may display provisional tokens, but the backend should commit the answer only after receiving the normal completion signal and passing required validation.
For implementation patterns focused on event handling and user-interface recovery, see how to stream LLM responses.
Prevent duplicate tool actions with a second idempotency layer
The most expensive retry bug is often outside the model call. A model can propose “send the email,” “create the ticket,” or “charge the card” twice after a timeout. Prompt instructions such as “do this only once” are not a transaction boundary.
Give every proposed side effect a deterministic action key derived from the parent operation, the logical tool step, and a normalized action fingerprint. Before execution, insert an action record under a unique constraint. Use a state machine such as proposed → approved → executing → committed, with unknown for a lost acknowledgement.
If a worker receives the same action again, return the committed receipt or inspect the external system before proceeding. Pass the stable key to downstream APIs that support idempotency. Otherwise use a natural unique identifier, conditional write, or outbox. Stop for review when a non-idempotent action cannot be reconciled.
Keep generation retries separate from tool retries. The model may be safe to call again while the external action is not. Persist the model’s tool proposal before approval and persist the external receipt before reporting success. A timeout or failure does not prove that a side effect did not occur; AWS makes the same warning in its guidance on safe retries with idempotent APIs.
If malformed arguments or schema mismatches are the source of repeated tool failures, diagnose them before widening the retry policy; see how to fix LLM tool-calling errors.
Test recovery paths, not just successful requests
Build deterministic fault injection around each boundary. Your test suite should cover:
- connection failure before sending;
- read timeout after the provider accepted the request;
429with and without a retry-delay header;- transient
5xxfollowed by success; - invalid input that must not retry;
- retry budget exhaustion;
- process crash after provider completion but before local commit;
- stream disconnect after partial text;
- duplicate queue delivery;
- tool success followed by a lost acknowledgement;
- two workers racing on the same operation key;
- the same key reused with different request content.
Assert outcomes, not only call counts. The operation must have one canonical result, every external action must have at most one committed effect, retry delay must stay inside the deadline, and terminal records must distinguish known failure from unknown outcome. Verify that logs contain correlation fields without leaking prompt content, credentials, or sensitive tool payloads.
Store both your application operation key and the provider request identifier when available; they answer different questions. OpenAI’s API reference documents the response x-request-id, recommends logging it in production, and supports a caller-supplied X-Client-Request-Id for supported endpoints so a request can be investigated even when a timeout prevents receipt of the response header. That client request ID is for tracing, not documented as an idempotency guarantee (checked 2026-07-24, official OpenAI API reference).
Run a load test in which many requests fail together. Confirm that jitter spreads retries, concurrency remains capped, and new work is not starved. Additional attempts consume rate-limit and token budgets even when the user sees one operation. The token budgets and circuit breakers guide covers wider system budgets.
Finally, write an operator runbook for outcome_unknown. It should explain how to look up the application key, provider request ID, stream state, and tool receipts; how to decide whether replay is safe; and who can resolve a stuck operation. Reliability comes from making ambiguity visible and recoverable, not from pretending retries remove it.
Frequently asked questions
Should I retry an LLM API request after a timeout?
Only when the error is transient, the total deadline and attempt budget still allow it, and the logical operation is idempotent. A read timeout means the outcome may be unknown, so reconcile stored state or reuse the same application idempotency key instead of assuming the first attempt failed.
How many times should an LLM API call be retried?
There is no universal count. Include SDK, application, and queue retries in one budget, then choose the smallest cap that improves measured success rates without exceeding the latency, rate-limit, or cost budget. Test simultaneous failures because retries that work in isolation can amplify an outage.
What is the difference between a request ID and an idempotency key?
A request ID identifies one HTTP attempt for tracing and support. An idempotency key identifies one logical user operation across multiple attempts and lets the application return one canonical result. Do not treat a provider request ID or client trace ID as duplicate prevention unless the provider explicitly documents that behavior.
Can a streaming LLM response resume after the connection drops?
Only if the exact API documents resumable streams and provides the required cursor or event identifier. Otherwise preserve the partial output as incomplete and deliberately replace or restart it; do not append a fresh generation as though it were a continuation.