Groq vs Cerebras API for Low-Latency Inference

Choose Cerebras first when your latency-critical path can use GPT-OSS 120B and maximum advertised generation speed matters most. Choose Groq first when you need a broader public model catalog, a smaller and cheaper model, or access to Groq’s Compound tool-using systems. That is the practical starting point for Groq vs Cerebras API for low-latency inference.

On the one production model both public APIs share, Cerebras lists GPT-OSS 120B at about 3,000 tokens per second; Groq lists about 500. Those are provider-published generation rates, not end-to-end latency promises. All details below were checked against official sources as of 2026-07-20.

Groq vs Cerebras at a glance

Decision factor	Groq API	Cerebras API
Inference hardware	Language Processing Unit (LPU) with compiler-scheduled, deterministic execution	Wafer-Scale Engine (WSE), a wafer-sized processor with compute and memory on one piece of silicon
Shared production model	openai/gpt-oss-120b	gpt-oss-120b
Advertised speed for GPT-OSS 120B	About 500 tokens/s	About 3,000 tokens/s
GPT-OSS 120B price per 1M tokens	$0.15 input / $0.60 output	$0.35 input / $0.75 output
GPT-OSS 120B context	131,072 tokens	131,072 tokens
Developer limit for GPT-OSS 120B	1,000 RPM / 250,000 TPM	1,000 RPM / 1,000,000 TPM
Public model choice	Several production text models, Compound systems, guard models, speech recognition, and text-to-speech	One production text model plus two public preview models; more families on dedicated endpoints
Best initial fit	Model choice, low token cost, and mixed text or audio workloads	Highest listed generation rate on the shared 120B model and a larger self-serve token allowance

The table uses on-demand or pay-as-you-go public API figures. Enterprise capacity, dedicated endpoints, and negotiated service commitments are separate products.

Speed: Cerebras leads the shared-model comparison

The cleanest comparison is GPT-OSS 120B because it removes model size as an obvious confounder. Groq’s supported-models page lists the model at approximately 500 tokens per second. Cerebras’s public model catalog lists approximately 3,000 tokens per second. On the vendors’ current figures, Cerebras generates output at roughly six times Groq’s rate for that model.

Do not translate that ratio directly into “six times faster requests.” Tokens per second measures generation after decoding begins. Users also wait for network transit, queues, prompt processing, and tool calls. Time to first token can dominate short answers; generation rate matters more for long answers.

Both APIs support streaming. Measure p50 and p95 time to first token, total time, output rate, and errors from the app’s region. Vendor throughput screens providers; a workload replay shows whether users feel the difference.

LPU versus wafer-scale architecture

GroqCloud runs on Groq’s LPU, a compiler-controlled architecture with deterministic compute and networking. It uses on-chip memory and statically scheduled data movement. Groq’s LPU explanation provides the technical overview.

Cerebras places compute, memory, and communication fabric on its wafer-sized Wafer-Scale Engine. Its WSE-3 product page lists 900,000 AI-optimized cores and 4 trillion transistors.

These designs help explain the vendors’ positioning, but application teams buy an API rather than a chip diagram. Architecture does not substitute for testing queue behavior, regional network paths, concurrency, or model output quality.

Model catalog: Groq offers more public production choices

Groq’s public catalog is broader. As of 2026-07-20, its pricing page lists GPT-OSS 20B and 120B, Llama 3.1 8B Instant, Llama 3.3 70B Versatile, and Qwen 3.6 27B. Groq also exposes Compound and Compound Mini systems that can select built-in web-search and code-execution tools. Separate endpoints cover Whisper speech recognition, Orpheus text-to-speech, and prompt-guard models.

That breadth matters when one provider must serve extraction, chat, guardrails, and audio. GPT-OSS 20B is listed at about 1,000 tokens per second and Llama 3.1 8B Instant at 840. A smaller model may reduce cost but still needs a task-specific quality test.

Cerebras’s public catalog is narrower. Its official catalog currently marks GPT-OSS 120B as the sole production model, at about 3,000 tokens per second. Gemma 4 31B, at about 1,850 tokens per second, and Z.ai GLM 4.7, at about 1,000 tokens per second, are preview models. Cerebras says preview models are for evaluation and can be discontinued on short notice; GLM 4.7 is scheduled for deprecation on August 17, 2026. Do not build a production dependency around either preview without a migration plan.

Cerebras offers more model families through dedicated endpoints, a separate route from the public pay-as-you-go catalog.

Pricing: Groq costs less on the shared model

For GPT-OSS 120B, Groq charges $0.15 per million input tokens and $0.60 per million output tokens as of 2026-07-20, according to its official pricing page. Cerebras’s official public-model API documentation shows $0.35 per million prompt tokens and $0.75 per million completion tokens as of 2026-07-20.

For 1 million input and 1 million output tokens, the listed bill is $0.75 on Groq and $1.10 on Cerebras, excluding caching, tools, and retries. Groq favors cost-sensitive volume; Cerebras may fit when faster decoding has more value than the price difference.

Groq also has lower-cost public models. As of 2026-07-20, GPT-OSS 20B costs $0.075 input and $0.30 output per million tokens, while Llama 3.1 8B Instant costs $0.05 input and $0.08 output. Compare quality before treating the smaller bill as a saving; failed tool calls and retries can erase it.

Context windows and output limits

Both providers list a 131,072-token context window for GPT-OSS 120B as of 2026-07-20. Groq permits up to 65,536 completion tokens on that model, according to its supported-models table. Cerebras’s public-model documentation lists a 40,960-token maximum completion.

A context limit is not a recommended prompt size. Long inputs add processing and irrelevant material. Test realistic documents rather than filling the window because it is available.

Groq’s main production text models also list 131,072-token contexts. Catalog-wide statements are less useful for Cerebras because its two other public models are previews and its dedicated catalog is contract-dependent.

Rate limits: compare the tier, not just the provider

On the paid developer tiers, both vendors list 1,000 requests per minute for GPT-OSS 120B as of 2026-07-20. The token limits differ: Groq lists 250,000 tokens per minute, while Cerebras lists 1,000,000 tokens per minute. Cerebras therefore provides four times the listed developer-tier TPM allowance for the shared model.

Free access reverses part of the shape. Groq’s rate-limit documentation lists GPT-OSS 120B at 30 RPM and 8,000 TPM, with daily caps. Cerebras’s rate-limit documentation lists 5 RPM and 30,000 TPM for its free trial, also with hourly and daily caps. Groq allows more small requests per minute; Cerebras allows more tokens per minute but fewer requests.

Limits apply at the organization level, and the first exhausted limit wins. Cerebras also uses separate uncached and total-token buckets; total TPM is three times uncached TPM during its rollout. Check the account console before load testing because exceptions may apply.

Pick Groq if…

• You need a public production model other than GPT-OSS 120B.
• Token price matters more than the highest advertised output rate on the shared model.
• A smaller model can meet your quality threshold at lower cost.
• You want text, tool-using Compound systems, speech recognition, or text-to-speech from one provider.
• Your workload has many short free-tier requests rather than a small number of token-heavy calls.

Pick Cerebras if…

• GPT-OSS 120B fits the task and long-output generation time is the primary constraint.
• The listed 1 million TPM developer allowance better matches your concurrency plan.
• You can pay a higher per-token rate in exchange for the provider’s higher advertised generation speed.
• You want to evaluate Gemma 4 multimodal input, while accepting its preview status.
• Dedicated endpoints and reserved enterprise capacity are relevant to your production plan.

The practical recommendation

Start with Cerebras for a latency-first GPT-OSS 120B pilot and Groq for a catalog-first or cost-first pilot. If the workload permits it, test both behind the same OpenAI-style client abstraction, using identical prompts, reasoning settings, maximum output lengths, and retry policies.

Run at least three cases: a short interactive answer, a long streamed generation, and the largest realistic prompt. Record time to first token, output rate, total latency, p95 latency, 429 responses, model correctness, and cost per successful task. Also test at expected concurrency; a fast single request does not establish production tail latency.

Cerebras has the clearer published speed advantage on the shared 120B model. Groq has the clearer advantages in public catalog breadth and listed token price. The right provider is the one that meets your quality threshold and latency objective under your own traffic pattern, not the one with the larger isolated tokens-per-second number.