The practical CPU-versus-GPU decision for a local LLM is not “which chip is faster?” It is which memory system can hold your model and context, then move the weights quickly enough for the response time you need.
Use a CPU first if you already own a capable machine, want to explore small or heavily quantized models, value large and inexpensive system RAM, or run occasional background jobs. Prefer a supported GPU when interactive latency, long prompts, repeated coding assistance, multiple users, or higher throughput matters. Consider unified-memory hardware or CPU–GPU hybrid offload when a model is too large for an affordable discrete GPU but still needs some acceleration.
There is no hardware-independent tokens-per-second promise. Runtime, model, quantization, context, backend, and memory configuration all affect the result. Compare a controlled benchmark of your workload, not a generic leaderboard.
Short answer
Start with model fit. If weights, KV cache, runtime buffers, and headroom fit in device memory, a GPU normally offers the more responsive path. If they do not, a CPU with ample RAM may run a model that a smaller GPU cannot. Partial offload occupies the middle ground, with less acceleration than a full GPU fit.
For occasional use of small quantized models, test the CPU you already own. For daily chat, coding, long prompts, or multiple sessions, prioritize a supported GPU. For models larger than affordable VRAM, compare large-RAM CPU, unified-memory, and partial-offload configurations. A successful load is not proof of useful speed.
For a capacity calculation before buying anything, use How Much VRAM Does a Local LLM Need?. It separates weight storage from KV cache, buffers, and headroom.
CPU, GPU, unified memory, or hybrid: decision table
The table below is a planning summary, not a performance guarantee. “Fit” includes the model weights, target context, runtime buffers, and operating headroom.
| Option | Best fit | Main advantage | Main limitation | Verify before buying |
|---|---|---|---|---|
| CPU with system RAM | Occasional use, small quantized models, or models that need inexpensive RAM capacity | Reuses existing hardware and can access large system-memory pools | Interactive generation is often slower once memory bandwidth becomes the bottleneck | RAM channels and speed, instruction-set support, measured PP/TG, and wall power |
| Discrete GPU | Daily chat, coding, long prompts, batching, or multiple users | High memory bandwidth and strong prompt-processing throughput when the workload fits in VRAM | VRAM capacity, platform cost, power, cooling, and backend compatibility | Full-fit memory budget, supported runtime/backend, driver, PSU, and case clearance |
| Unified-memory system | Models that benefit from one large CPU/GPU-accessible memory pool | Avoids a small, separate VRAM ceiling and can accelerate models that fit the shared pool | Published bandwidth and capacity do not guarantee a particular token rate | Usable memory after the OS, runtime support, sustained thermals, and a representative benchmark |
| CPU–GPU hybrid offload | A model exceeds VRAM but some GPU acceleration is still useful | Trades inexpensive system RAM capacity for partial acceleration | Transfers and CPU-resident layers can limit gains; results depend on the offload split | Actual offloaded layer count, PCIe path, context size, and end-to-end latency |
Memory bandwidth versus compute
Local LLM inference has two materially different phases. During prefill, the runtime processes the known prompt in parallel. During decode, it generates one new token at a time. NVIDIA’s inference guide describes prefill as highly parallelized matrix–matrix work and decode as a comparatively memory-bound matrix–vector operation. In decode, the system repeatedly reads model weights and attention state to produce each token.
Core counts, TOPS, and FLOPS therefore do not predict local chat speed by themselves. A processor can advertise substantial compute yet wait on memory during single-stream generation. Effective bandwidth also differs from a theoretical specification because memory channels, cache behavior, kernels, and contention matter.
A simplified ceiling illustrates the relationship:
ideal decode tokens/s ≈ sustainable memory bandwidth / bytes read per generated token
This planning model ignores cache reuse, KV-cache traffic, mixture-of-experts behavior, runtime overhead, and compute limits. It still explains why quantization can improve fit and speed, and why more CPU cores stop helping after bandwidth binds.
Memory topology matters as much as capacity:
- A desktop CPU reads weights from system RAM, so memory channels and configured speed matter.
- A discrete GPU uses dedicated VRAM with a high-bandwidth local path. A model that fits fully in VRAM avoids repeatedly pulling weights across PCIe during inference.
- Unified-memory systems give CPU and GPU one pool, allowing large allocations without a separate full model copy.
- Hybrid inference places only part of the model on a GPU. The llama.cpp project officially supports CPU–GPU hybrid inference and layer offload, including models larger than total VRAM.
As a concrete specification example, Apple’s 2025 Mac Studio page lists 410 GB/s for one M4 Max configuration, 546 GB/s for another, and 819 GB/s for M3 Ultra configurations. Those figures are not tokens-per-second results; they show why “RAM amount” alone is an incomplete description of a unified-memory machine.
Check software support before comparing hardware
A GPU is only useful if the intended runtime can use its accelerator backend on the exact GPU, operating system, and driver combination. Do not infer support from the vendor name or from successful display output.
For NVIDIA on Windows, the official CUDA installation guide separates the checks for a CUDA-capable GPU, system requirements, driver model, and installation verification. AMD likewise publishes current Radeon and Ryzen compatibility matrices by device and operating system. These lists change, so verify the exact model rather than assuming that an entire GPU generation is supported.
Runtime support is narrower than hardware support in some workflows. Confirm all four layers:
- the GPU or integrated accelerator appears in the vendor’s current compatibility documentation;
- the operating system and driver version are supported;
- the chosen runtime was built with the intended backend; and
- startup logs and a benchmark show that layers or tensors actually moved to the accelerator.
If you want a practical backend comparison, llama.cpp vs Ollama explains how a low-level inference engine and a packaged local-model workflow differ. Hardware results from one backend should not be treated as universal.
Prompt processing and generation speed
Two machines can produce the same decode rate yet feel different. A short chat exposes generation speed; a codebase summary or long document makes time to first token and prompt-processing throughput equally important.
Measure at least:
- Prompt processing (PP), in tokens per second: how fast the existing input is evaluated.
- Text generation (TG), in tokens per second: how fast new output tokens appear.
- Time to first token: the caller’s delay through loading, queueing, tokenization, prompt processing, and first-token production.
- End-to-end latency: the complete time for the output length you normally request.
- Throughput under concurrency: completed tokens or requests per second when more than one session is active.
GPUs usually gain more on large prompts because prefill exposes parallel work. Decode for one user can leave GPU compute underused while memory remains busy. Batching several sequences may improve utilization but increase individual latency.
Do not compare token rates across unrelated models without qualification. Tokenizers divide text differently, output lengths differ, and one model’s token is not a fixed amount of useful text. NVIDIA’s official guide explicitly cautions that similar tokens-per-second values may not be equivalent across different tokenizers.
llama.cpp’s llama-bench reports prompt processing and generation separately, but excludes tokenization and sampling time. A server adds prompt templates, retrieval, tool calls, and queueing, so also record an end-to-end request.
Model size, quantization, and context
Hardware selection begins with a memory budget:
required memory ≈ model weights + KV cache + runtime buffers + headroom
Weight precision provides a rough lower bound. Seven billion parameters at 16 bits require approximately 14 GB for weights alone; NVIDIA uses this example in its official guide. A nominal 4-bit representation suggests about 3.5 GB before metadata, higher-precision tensors, KV cache, and runtime overhead. Measure the actual file and loaded allocation.
Quantization can reduce weight traffic and speed decode, but methods differ in accuracy, kernel support, and hardware behavior. llama.cpp supports integer quantization from 1.5 through 8 bits; that is not a recommendation for every format. Local LLM Quantization Explained gives a Q4, Q5, and Q8 test method.
Context adds KV-cache memory based on architecture, length, active sequences, and cache types. A larger context can turn full offload into partial offload or cause allocation failure.
Use these fit states in purchase planning:
- Full device-memory fit: weights, target context, buffers, and headroom remain in GPU or unified memory.
- Partial GPU offload: some layers or tensors run on the GPU and the remainder on the CPU.
- CPU-resident inference: weights and cache use system RAM, with CPU vector instructions and optimized kernels.
- Storage paging: the working set exceeds practical RAM. Treat this as a failed interactive configuration, even if the process eventually runs.
Never buy against the model file size with zero margin. Test the maximum context and concurrency you intend to use, not the runtime’s smallest successful load.
Power, noise, and hardware cost
Purchase price is only one cost. A discrete GPU may also require a larger power supply, a compatible case, cooling, and enough airflow to avoid sustained clock reduction. A CPU-only system can reuse existing parts, but a heavily loaded high-core-count CPU is not automatically quiet or efficient.
Manufacturer figures are boundaries, not direct energy measurements. NVIDIA lists 250 W total graphics power and 650 W required system power for its reference GeForce RTX 5070; AMD lists 170 W default TDP for the Ryzen 9 9950X. These use different definitions and are not whole-system wall-power comparisons.
Measure at the outlet for the actual task. Record idle watts, loading watts, steady generation watts, and wall-clock completion time:
energy per job (Wh) = average wall power (W) × elapsed hours
A faster device can draw more power yet use less energy per completed job. Buying a large GPU for two queries a day may still be hard to justify. Include:
- acquisition cost and any required platform upgrades;
- electricity at your local rate;
- heat and fan noise in the intended room;
- time waiting for prompts and generations;
- whether the machine must remain available for other work; and
- driver and runtime maintenance.
Do not substitute current street prices without a region and date. Retail pricing changes too quickly for an evergreen CPU-versus-GPU rule.
Benchmark your real workload
Use one model file, one runtime commit or release, and one fixed configuration for both paths. The official llama.cpp benchmark tool can vary prompt length, generated length, thread count, GPU-offloaded layers, batch size, and context depth. It repeats tests five times by default and can emit Markdown, CSV, JSON, JSONL, or SQL.
A useful test matrix is:
| Test | Suggested workload |
|---|---|
| Short chat | 128 prompt tokens, 128 generated tokens |
| Coding assistant | 2,048 prompt tokens, 256 generated tokens |
| Long-document question | Your representative context, 128 generated tokens |
| Sustained session | Five repeated turns with prompt-cache behavior recorded |
| Concurrent service | 1, 2, and 4 active sequences |
Capture runtime version, model hash, quantization, context, cache type, threads, batch settings, offloaded layers, backend, temperatures, wall power, PP, TG, time to first token, and end-to-end time. Repeat tests and report variance, thermal throttling, and cold-start behavior.
Test CPU and offload settings explicitly. llama-bench accepts multiple thread counts and --n-gpu-layers values in one run. Check startup diagnostics to confirm actual offload; a GPU-enabled build that leaves weights on the CPU is not a GPU benchmark.
Keep quality fixed. If the GPU result uses a smaller quantization merely to fit, compare output quality on a held-out prompt set before calling it faster. If one configuration changes context, sampling, or prompt caching, label it as a different workload.
Buying decision examples
Reuse-first personal setup: For private summarization or occasional chat, start with a small quantized model on your existing CPU. Buy only if measured waiting time is a problem.
Daily interactive coding: Favor a supported GPU with memory for the model, context, and headroom. Measure prompt latency as well as generation.
Large model, low request rate: Large-RAM CPU, unified memory, or partial offload can be rational when the model exceeds affordable VRAM and slower generation is acceptable. Capacity does not guarantee responsiveness.
Local team service: Test concurrency, queueing, and cooling. Include KV cache for every active sequence and set an explicit latency target.
Undecided between local runtimes: Hardware support and operational controls vary by backend. Compare the workflow in Ollama vs LM Studio before treating a benchmark from one runtime as universal.
The buying rule is simple: choose the least expensive complete system that fits the model and context, meets your measured latency target, and remains acceptable for power, noise, and maintenance. CPU is a valid starting point, GPU is usually the interactive-performance path, and hybrid or unified memory is a capacity compromise—not an automatic winner.
FAQ
Can I run a local LLM without a GPU?
Yes. CPU inference is supported by runtimes such as llama.cpp, and sufficient system RAM can hold models that exceed a small GPU’s VRAM. Whether it is useful for you depends on measured prompt latency and generation speed, not merely whether the model loads.
How much RAM do I need for CPU inference?
Budget for the model weights, KV cache at your target context, runtime buffers, the operating system, and headroom. The model file size alone is not enough. Measure the loaded process at the context and concurrency you plan to use.
Is more VRAM always better than a faster GPU?
No. More VRAM may allow a larger model or longer context to fit fully, while greater bandwidth and compute may improve latency for a model that already fits. Choose against both the memory budget and a measured response-time target.
Does partial GPU offload make a model as fast as full GPU inference?
Usually not. Partial offload can outperform CPU-only inference, but CPU-resident work and transfers can remain bottlenecks. Benchmark several offloaded-layer settings and confirm the actual split in startup diagnostics.
Are CPU and GPU tokens-per-second results directly comparable?
Only under a controlled setup. Keep the model file, quantization, context, prompt, output length, runtime version, and sampling settings fixed. Also compare time to first token and end-to-end latency, because decode tokens per second alone can hide slow prompt processing.
Official primary sources
Sources below were checked on the date shown:
- llama.cpp project documentation: supported CPU, GPU, quantization, and hybrid-inference paths. Checked 2026-07-23.
- llama-bench documentation: PP/TG test definitions, options, repetition behavior, outputs, and measurement exclusions. Checked 2026-07-23.
- llama.cpp token-generation performance guide: offload verification and CPU-thread troubleshooting. Checked 2026-07-23.
- NVIDIA LLM inference optimization guide: prefill/decode behavior, memory-bound decode, tokenizer caution, and memory composition. Checked 2026-07-23.
- Apple Mac Studio (2025) specifications: unified-memory capacities and published memory bandwidth. Checked 2026-07-23.
- NVIDIA GeForce RTX 5070 specifications: reference total graphics power and required system power. Checked 2026-07-23.
- AMD Ryzen 9 9950X specifications: default TDP. Checked 2026-07-23.
- NVIDIA CUDA Installation Guide for Microsoft Windows: CUDA-capable GPU, system, driver-model, and installation-verification checks. Checked 2026-07-24.
- AMD ROCm Radeon and Ryzen compatibility matrices: device- and operating-system-specific compatibility references. Checked 2026-07-24.