An LLM API benchmark is useful only when it predicts how your application will behave. A leaderboard made from one short prompt, one successful response, and one average latency number does not do that. It hides output-length differences, slow tail requests, rate-limit pressure, retries, and answers that arrive quickly but fail the task.
A defensible benchmark treats speed as one part of a contract. It fixes the workload, records each request, separates latency phases, rejects low-quality outputs, and reports cost and reliability beside throughput. The result is not a universal winner. It is evidence about which API configuration fits a specific production workload.
Verified and updated: 2026-07-23. Metric definitions and provider fields in this article were checked against the official sources linked in the text. Provider behavior can change, so recheck those sources before publishing or rerunning a long-lived benchmark.
Benchmark scorecard at a glance
Use this scorecard to keep the comparison tied to useful work. Every latency or cost figure should carry its sample count, and every aggregate should be available by workload bucket.
| Decision dimension | Primary measure | Minimum context to publish |
|---|---|---|
| Responsiveness | TTFT p50 and p95 | Streaming mode, event definition, valid sample count |
| Completion speed | End-to-end latency p50 and p95; TPOT | Input/output token buckets, TPOT formula |
| Capacity | Qualified requests/s and output tokens/s | Arrival pattern, concurrency, test duration |
| Quality | Quality pass rate | Dataset version, gate, threshold, pass/attempt counts |
| Reliability | Qualified final results / initial tasks | Timeouts, API errors, retries, invalid outputs |
| Cost efficiency | Cost per qualified result | Billed units, retry cost, official price source and date |
1. Define the workload before choosing metrics
Start with the application decision, not the provider list. “Which model is fastest?” is too vague. “Which configuration can classify our support tickets with at least 95% schema-valid responses while keeping p95 end-to-end latency under our product target?” is testable.
Build a frozen workload manifest with these fields:
| Field | What to record | Why it matters |
|---|---|---|
| Task class | Extraction, classification, summarization, chat, coding, or tool use | Different tasks stress different capabilities |
| Prompt version | Exact system and user templates, including whitespace | Small prompt changes can alter tokens, quality, and caching |
| Input buckets | For example, short, medium, and long documents | A single mean input length conceals prefill costs |
| Output contract | Maximum length, stop conditions, schema, and required fields | Output length strongly affects total latency and cost |
| Model configuration | Exact model ID and supported sampling/reasoning settings | Aliases and defaults may change |
| Traffic shape | Arrival rate, concurrency, ramp pattern, and test duration | Queueing appears only under realistic load |
| Region and client | Client location, runtime, SDK version, and connection policy | Network and client overhead are part of observed latency |
| Success rule | HTTP success, complete stream, valid schema, and task-specific quality | A fast invalid response is not a success |
Use production-like examples, but remove secrets and personal data. Keep a held-out set that was not used to tune prompts or thresholds. If the application has several request classes, report each class separately rather than mixing them into one attractive average. The workflow in building an LLM evaluation dataset from real failures explains how to version cases and protect a holdout from repeated tuning.
For a provider comparison, send the same semantic task to every candidate. Exact token counts can differ because tokenizers differ, so save both the source text and each provider’s reported input and output usage. If one model needs a substantially different prompt to pass the quality gate, treat that as a separate configuration rather than quietly changing the test.
2. Separate TTFT, TPOT, end-to-end latency, and throughput
No single “latency” number describes a streamed LLM response. Record at least four measures:
- Time to first token (TTFT): client-observed time from sending the request until the first non-empty streamed content arrives.
- Time per output token (TPOT): the generation interval after the first token, normalized by the remaining output-token count.
- End-to-end latency: client-observed time from request start until the final response event is received.
- Throughput: completed requests per second and output tokens per second across the whole test.
For a response with at least two output tokens, a practical TPOT calculation is:
TPOT = (end-to-end latency - TTFT) / (output tokens - 1)
NVIDIA’s official NIM benchmarking guide calls the same measure inter-token latency and documents this formula without including TTFT. It also warns that benchmarking tools do not all calculate the metric identically (NVIDIA NIM LLM metrics, verified 2026-07-23). State your formula, streaming parser, and definition of a non-empty event in the report.
Reasoning-capable APIs create another definition trap. NVIDIA AIPerf distinguishes the first token of any type from the first non-reasoning output token; for models without reasoning, those points are equivalent. Its TTFT also includes client-observed network time, queueing, prompt processing, and generation up to the first token (AIPerf metrics reference, verified 2026-07-23). If an API hides reasoning tokens, label the measured event “first visible output” rather than claiming access to an internal phase.
Measure with a monotonic high-resolution clock on the client. Do not substitute a provider’s server-processing header for end-to-end latency. For example, OpenAI documents openai-processing-ms as API processing time, while x-request-id identifies a request; those fields are valuable diagnostic metadata but do not include every client-observed delay (OpenAI API reference, verified 2026-07-23).
Report distributions, not only means. At minimum, publish p50, p95, and the number of valid samples for TTFT and end-to-end latency. The p95 value is the observation at or below which 95% of sorted valid results fall under your stated percentile method. Use one percentile implementation throughout the study and disclose it, because small samples and different interpolation rules can produce different values.
3. Control prompt length, output length, and streaming
Input and output tokens are benchmark variables, not incidental metadata. A long input increases prompt-processing work; a long output gives decoding speed more influence over total latency. Comparing a 30-token answer with a 500-token answer makes end-to-end latency nearly meaningless.
Create fixed length buckets and keep their mix stable:
- Select real examples for each task class.
- Record provider-reported input tokens for every request.
- Set the same semantic output limit and stop rule where APIs permit it.
- Validate that responses are complete rather than cut off at the limit.
- Report latency by input and output bucket as well as in aggregate.
This control is not merely cosmetic. OpenAI’s official latency guide says token generation is usually the highest-latency step and gives a rough heuristic that halving output tokens may roughly halve latency; it also notes that reducing input tokens often has a smaller effect outside very large contexts (OpenAI latency optimization, verified 2026-07-24). Treat those statements as provider guidance, not as a guaranteed ratio for another model or workload. The benchmark should measure the relationship for each output-length bucket.
Run separate streaming and non-streaming tests. Streaming changes perceived responsiveness and exposes TTFT, but it also adds event parsing and potentially many chunks. OpenAI’s Responses API, for example, emits server-sent events when stream is true and supplies event types and sequence numbers in the stream (OpenAI streaming events, verified 2026-07-23). Your collector should timestamp non-empty content events, retain event order, and ignore terminal markers that carry no output.
Do not assume one network chunk equals one token. A chunk may contain several tokens, and token usage may arrive only near completion. Save raw timestamps and final provider-reported token counts, then calculate TPOT consistently. If token usage is unavailable for a failed or interrupted request, keep the request in reliability statistics but mark token-normalized latency and cost as unavailable.
Determinism is also a control, not a guarantee. Fix supported sampling parameters and random seeds when available, but still expect output variation. Store the exact request body after defaults are resolved, without credentials, so another operator can reconstruct what was sent.
4. Test warm, cold, steady-state, and concurrent traffic separately
A warm serial loop answers only one question: how repeated requests behave without meaningful competition. Production traffic may encounter connection setup, cache state, autoscaling, queueing, and shared capacity.
Use four labeled phases:
- Cold-path probe: a small, separately reported set after an idle period or new client process. Do not claim this forces a provider-side cold start unless the provider exposes that state.
- Warm-up: requests excluded from the scored sample, used to establish connections and catch configuration errors.
- Steady-state serial: concurrency of one, useful for a clean view of per-request behavior.
- Steady-state load: a fixed arrival process or concurrency ladder that resembles production.
Do not mix warm-up results into the scored distribution. Randomize or rotate provider order so time-of-day effects do not always favor the same candidate. Run more than one window, and keep the client host, region, network path, and timeout policy constant.
There is no universal minimum sample count for an LLM API benchmark. Choose the sample size before looking at winners, make it large enough to expose tail behavior for your risk level, and publish the count. A p95 based on 20 requests is effectively describing one of the slowest observations and should not be presented with false precision. For important decisions, repeat the test on different days and show between-run variation.
Concurrency must be planned against both requests and tokens. A test can be limited by requests per minute, tokens per minute, client sockets, or downstream work. OpenAI’s official API reference lists separate request and token limit, remaining, and reset headers; it also recommends logging request IDs for troubleshooting (OpenAI API reference, verified 2026-07-23). Capture supported rate-limit headers without assuming that every provider uses the same names.
Provider limit semantics can differ as well as header names. Anthropic documents separate requests-per-minute, input-tokens-per-minute, and output-tokens-per-minute limits for its Messages API. It also warns that limits can be enforced over shorter intervals, so a nominal 60 RPM allowance can still reject a one-second burst; a 429 response includes retry-after guidance (Anthropic API rate limits, verified 2026-07-24). Preserve the attempted arrival times so burst-related throttling is not mistaken for random provider instability.
Use the AI API rate limit calculator to estimate whether request rate or token rate will bind first, then add retry headroom. The calculation is a capacity hypothesis, not benchmark evidence: the load test must still record throttles, timeouts, retries, and achieved throughput. For a deeper provider-level comparison, see how Groq and Cerebras API trade-offs were evaluated.
5. Put quality gates before speed rankings
Benchmark outputs should pass a gate before their latency contributes to a “qualified performance” result. Otherwise, a model that returns short, incomplete, or malformed answers can appear faster precisely because it did less useful work.
Define the gate per task:
- Structural validity: required JSON schema, fields, types, citations, or tool-call shape.
- Completeness: no truncation, missing section, or unresolved placeholder.
- Correctness: exact match, executable tests, reference facts, or a scored rubric.
- Safety and policy: task-specific prohibited content or data leakage checks.
- Human review: blinded pairwise or rubric review when automated checks cannot capture usefulness.
Freeze thresholds before running the final comparison. If you tune the prompt after seeing failures, version it and rerun every candidate on the full held-out set. Record inter-rater agreement or adjudication rules for subjective scoring.
Publish two related measures: quality pass rate and qualified latency. Qualified latency is the latency distribution among responses that passed, accompanied by the pass count and total attempts. Also publish failure categories. Never delete timeouts or invalid outputs as “outliers” merely because no final token arrived.
The gate should mirror the actual release process. A code-generation workload might require compilation, focused tests, and a diff policy; an extraction workload might require schema validation plus field-level accuracy. The practical patterns in building a self-hosted evaluation gate and using machine gates for AI output can help turn vague quality judgments into explicit acceptance checks.
6. Normalize results by successful work and cost
Raw tokens per second rewards volume, not value. Add cost-normalized measures that include failures and retries:
cost per qualified result = total benchmark cost / number of quality-passing results
qualified throughput = quality-passing results / measured test duration
retry-adjusted success rate = qualified final results / all initial tasks
Use provider-reported billed token usage where available and reconcile it with the invoice or usage export. Keep cached input, uncached input, output, tool calls, and other billed units separate. Apply the official price effective on the test date, save its source URL, currency, and retrieval date, and do not silently reuse an old price table. This article intentionally contains no vendor price constants because prices are not needed to define the methodology.
For a repeatable way to separate input, output, cache, and batch charges before applying current rates, use the LLM API cost calculator. Do not copy its example prices into a benchmark report without rechecking the provider page on the test date.
Report rate-limit and reliability outcomes beside cost: HTTP status, provider error type, timeout stage, retry count, backoff time, and final disposition. A retry can make a task succeed while doubling latency and increasing cost. Conversely, an aggressive retry policy can create extra load and make throttling worse.
Do not combine service tiers, model versions, or regions in one row. If a provider routes an alias to an undisclosed revision, record the alias, response metadata, and exact test time, then describe the limitation. A benchmark is a dated observation, not a permanent product specification.
The final decision should use constraints, not a universal score. One application may select the lowest cost per qualified result under a p95 latency ceiling; another may prioritize the highest pass rate under a monthly budget. Publish the rule before naming the selected configuration.
7. Publish a reproducible benchmark report
A reproducible report lets another operator rerun the workload and understand every exclusion. Use one row per configuration and preserve a request-level machine-readable file privately, with secrets and sensitive prompts removed.
| Report field | Example format |
|---|---|
| Benchmark ID | support-extract-v3-2026-07-23 |
| Decision question | Configuration meeting quality and p95 targets at lowest measured cost |
| Provider/model | Exact API provider, model ID, service tier, and region |
| Test window | Start/end time with UTC offset |
| Client | Host region, runtime, SDK version, HTTP/connection settings |
| Workload | Dataset hash, task mix, input buckets, prompt version |
| Generation settings | Output cap, sampling/reasoning controls, streaming mode |
| Load | Warm-up count, scored count, arrival pattern, concurrency |
| Quality gate | Validators, thresholds, reviewer procedure |
| Latency | TTFT and end-to-end p50/p95; TPOT formula and valid count |
| Capacity | Requests/s, output tokens/s, rate-limit events |
| Reliability | Successes, timeouts, API errors, invalid outputs, retries |
| Cost | Billed units, price source/date, total cost, cost per qualified result |
| Exclusions | Every removed request and the predeclared reason |
| Artifacts | Harness commit, configuration hash, sanitized raw-result hash |
Before accepting the report, confirm that every candidate used the same workload version, quality gate, load phase, and client environment. Check that slow failures remain visible, percentile sample counts are present, and cost includes retries. Rerun any row affected by a harness bug; do not patch the summary by hand.
The honest conclusion may be that the result is inconclusive. Overlapping run-to-run variation, too few qualified samples, or an uncontrolled provider change are reasons to gather more evidence, not reasons to crown a winner. A benchmark earns trust by making uncertainty inspectable.
FAQ
How many requests do I need for an LLM API benchmark?
There is no universal minimum. Choose the count before comparing results, publish it, and use enough requests and repeated test windows to make the percentile and failure rate meaningful for your risk level. Small samples can support a smoke test, but they do not justify precise tail-latency claims.
Should I benchmark with streaming enabled?
Use the mode your application will ship, and run streaming and non-streaming as separate configurations if both matter. Streaming exposes TTFT and perceived responsiveness; non-streaming gives a simpler completion-time measure. Do not mix the two modes in one latency distribution.
Is tokens per second enough to compare LLM APIs?
No. Tokens per second describes generation throughput, but it does not capture queueing, network delay, time to first token, answer quality, failures, or retries. Pair it with TTFT, end-to-end latency, quality pass rate, reliability, and cost per qualified result.
How do I compare models with different tokenizers?
Freeze the source prompts and semantic output contract, then record each provider’s reported input and output usage. Report results by source-text bucket and provider token bucket rather than assuming token counts are interchangeable.
Should failed or rate-limited requests be removed from latency results?
Keep them in reliability and retry-cost statistics. A request without a completed response may not have a valid TPOT or end-to-end completion time, so mark those fields unavailable instead of inventing a duration. Publish both qualified-latency sample counts and all-attempt failure categories.