Quantization stores a model’s weights at lower precision so the model takes less disk space and memory. For most people running a local LLM, the practical choice is not simply “4-bit versus 8-bit.” It is a choice among a specific model, a specific GGUF quantization method, a context length, and a hardware path.
The short answer is: start with a well-supported Q4_K_M or Q5_K_M GGUF, then test it against Q8_0 on your own prompts. Q4 usually offers the most capacity per gigabyte, Q5 spends more memory for a smaller compression penalty, and Q8 is a useful high-precision reference when it fits. None is universally superior. Model architecture, the quantizer, runtime version, prompt type, and CPU/GPU backend can change the result.
The specifications and command-line behavior below were checked on 2026-07-24 against the official llama.cpp and GGML project documentation listed in Primary sources.
What quantization changes
An unquantized or lightly quantized model represents weights with relatively high-precision numeric formats such as FP32, BF16, or FP16. Weight quantization replaces many of those values with lower-bit representations plus the scale and other metadata needed to approximate the original values during inference.
That trade produces three immediate effects:
- Smaller weight files. Fewer effective bits per weight reduce disk and memory requirements.
- Different compute behavior. A runtime may use optimized kernels for a quant type, so a smaller file can improve speed. It can also become slower if the hardware or backend handles that format poorly.
- Some information loss. The reconstructed weights are approximations. The effect can be negligible for one workload and visible for another.
Quantization usually changes the stored weights, not the model’s advertised parameter count or context limit. It also does not automatically shrink every runtime allocation. The KV cache, compute buffers, multimodal projector, and backend overhead remain separate memory costs. If the goal is to determine whether a model fits, use the full method in How Much VRAM Does a Local LLM Need? rather than comparing download sizes alone.
GGUF is a container format, not a single quantization method. It can hold tensors and standardized model metadata, while its tensor data may use different encodings. A filename containing Q4_K_M is therefore describing the quantization within a GGUF file; “GGUF” by itself does not tell you its precision or likely memory use.
How to read common GGUF labels
The leading number in labels such as Q4, Q5, and Q8 is a useful precision family, but it is not an exact whole-file bits-per-weight figure. Block scales, metadata, tensor alignment, and mixed tensor types add overhead. Two files with “Q4” in their names can consequently differ in size, speed, and quality.
| Label | Core encoding described by llama.cpp | Relative weight size | Sensible role |
|---|---|---|---|
Q4_K_M | Q4_K is 4.5 bits/weight; _M applies a mixed-tensor recipe | Low | First fit-and-speed candidate |
Q5_K_M | Q5_K is 5.5 bits/weight; _M applies a mixed-tensor recipe | Medium | Quality-oriented default when it fits |
Q8_0 | 8-bit values in 32-weight blocks, plus a block scale | High | High-precision local reference |
Q4_0 | 4-bit values in 32-weight blocks, plus a block scale | Low | Compatibility or hardware-specific comparison |
The bits-per-weight values in this table describe the core tensor encodings documented by the project, not the exact average for a complete GGUF. A mixed recipe, metadata, alignment, and tensors stored at other precisions make the final file different. The official tensor encoding reference was checked on 2026-07-24 for these block sizes and encoding values.
The suffixes matter. K identifies the K-quant family. In the commonly distributed *_K_M recipes, the file may use a mixture of tensor encodings rather than forcing every tensor into one uniform type. This is why reading Q4_K_M as “exactly four bits for every parameter” is misleading. The official quantizer exposes a --pure option specifically to disable K-quant mixtures, and it separately allows output and embedding tensor types to be selected.
Q8_0 also carries overhead beyond eight raw bits per weight. The official GGUF documentation describes it as round-to-nearest block quantization with 32 weights per block and a block scale. Treat the label as an encoding name, not as a file-size equation.
Use the exact filename and byte size in the model repository. Avoid choosing by a publisher’s generic “Q4 recommended” note without checking whether the file is Q4_0, Q4_K_S, Q4_K_M, an importance-matrix quant, or another scheme. Also confirm that your runtime version supports the model architecture and quant type.
Estimate memory before downloading
A quick estimate for weight storage is:
weight bytes ≈ parameter count × effective bits per weight ÷ 8This is a screening calculation, not a replacement for the published GGUF file size. For example, a nominal 8-billion-parameter model at a hypothetical four effective bits per weight would start near 4 GB of raw weight data. The finished file will differ because “8B” is rounded, and because quantization metadata, higher-precision tensors, and container overhead consume space.
The official llama.cpp quantization guide provides a concrete Llama 3.1 example: an 8B source is listed at 32.1 GB and its Q4_K_M result at 4.9 GB. The same table lists 70B at 280.9 GB before quantization and 43.1 GB at Q4_K_M. These are project examples checked on 2026-07-23, not universal ratios for every 8B or 70B architecture.
For a reliable capacity check, add:
required memory ≈ GGUF weight size
+ KV cache
+ compute and runtime buffers
+ safety headroomContext length can erase the apparent saving from selecting a smaller quant. The KV cache normally grows with active context and parallel sequences, and its size depends on architecture as well as cache precision. A Q4 model loaded at a very long context can require more total memory than a Q5 model at a shorter practical context.
Do not confuse decimal GB, binary GiB, file size, and observed VRAM. Download pages may use one unit while operating-system and GPU tools display another. Record the actual file bytes and the runtime’s startup allocations. If the model is multimodal, include its projector or encoder file too; the official llama.cpp guide notes that multimodal components can be separate from the LLM GGUF.
Quality and speed are workload-dependent
More bits usually reduce the amount of weight approximation, but that does not create a universal quality ranking for complete systems. A stronger base model at Q4 may outperform a weaker model at Q8 on your task. Even within one model, a quant that preserves general language behavior can still fail on code syntax, structured JSON, rare terminology, multilingual text, or small numeric distinctions.
Perplexity is useful for detecting broad distribution changes, but it is not a product acceptance test. A small perplexity difference does not say whether the model preserved your required tool-call schema or stopped making a particular factual error. Conversely, one attractive chat transcript is not enough evidence that a quant is safe for repeated use.
Speed is similarly conditional. Smaller weights reduce memory traffic, which can help on bandwidth-limited systems. Yet kernel support, CPU instruction sets, GPU backend, number of offloaded layers, prompt batch size, and context depth all affect throughput. The official llama.cpp quantization tables show that quant methods differ in both size and measured prompt-processing and generation speed; they do not establish one format as fastest on every device.
Test prompt processing and text generation separately. Prompt processing matters when ingesting long documents or retrieved context. Generation speed matters during interactive output. Time to first token, tokens per second, peak memory, and total task completion time answer different questions.
The model’s purpose also changes the acceptable loss. A creative drafting assistant may tolerate occasional wording changes. A code transformation, extraction pipeline, or local evaluation gate may require exact syntax and consistent decisions. For a newer local architecture, such as the one discussed in DiffusionGemma and local text generation, verify that the runtime and quantization workflow actually support that architecture before applying assumptions learned from conventional decoder-only models.
CPU offload changes the trade
If a quant does not fully fit in VRAM, llama.cpp can keep fewer layers on the GPU and use system memory for the rest. Its CLI currently exposes --n-gpu-layers with an exact count, auto, or all, plus device and multi-GPU split controls. This makes partial acceleration possible, but it is not equivalent to having enough VRAM for full placement.
Offload introduces a new comparison:
- A larger Q5 or Q8 model with substantial CPU participation may preserve more weight precision but generate more slowly.
- A smaller Q4 model that fits fully on the GPU may deliver better interactive latency.
- A CPU-only run can be practical when latency is secondary and sufficient RAM and memory bandwidth are available.
- A unified-memory machine needs one combined budget for model weights, cache, runtime, operating system, and other applications.
Do not assume that one additional quantization bit causes a fixed speed penalty. Measure the exact layer placement. llama-bench supports testing different GPU-layer counts, while its structured output records the model file, backend, threads, batch sizes, cache types, GPU layers, prompt tokens, generated tokens, average throughput, and standard deviation.
For purchase planning, prefer a quant that fits with margin over a larger file that barely loads. Swapping, memory pressure, or aggressive offload may turn a technically successful startup into an unpleasant session. For architecture-level context on why open-weight releases differ beyond their quant files, see Kimi K3 and the open-weight model trade.
Benchmark quants on your machine
Compare files produced from the same base model and, where possible, the same conversion source. Changing the model revision, chat template, tokenizer, or quantizer at the same time makes the result hard to interpret.
Create a small but representative test suite:
- Fit test: Load each quant at the intended context and record weight, KV-cache, compute-buffer, RAM, and VRAM allocations.
- Speed test: Measure prompt processing and generation independently after a warm-up run.
- Task test: Run the same deterministic or tightly controlled prompts for code, extraction, writing, or domain questions.
- Format test: Validate JSON, citations, tool arguments, or other required structures mechanically.
- Stability test: Repeat runs and record variation instead of keeping only the fastest result.
A basic performance comparison can use the official benchmark utility:
llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -r 5 -o json
llama-bench -m model-Q5_K_M.gguf -p 512 -n 128 -r 5 -o json
llama-bench -m model-Q8_0.gguf -p 512 -n 128 -r 5 -o jsonAs documented on 2026-07-23, -p sets prompt tokens, -n sets generated tokens, -r controls repetitions, and JSON is one supported output format. llama-bench reports average tokens per second and standard deviation, but its documented measurements exclude tokenization and sampling time. If end-user latency matters, wrap the complete application path with a separate wall-clock measurement.
Keep model settings constant: context, batch and micro-batch sizes, threads, GPU layers, cache types, sampling parameters, prompt text, and maximum output. Record the llama.cpp build commit and driver environment. Then inspect failures, not only averages. A quant that is 10% faster but breaks a required schema is not a 10% improvement.
Starting recommendations
Choose the smallest quant that passes your quality gate, not the smallest quant that merely loads.
- Begin with Q4_K_M when memory capacity is the binding constraint or when fitting the full model on the faster device is likely to matter more than a modest precision increase.
- Try Q5_K_M next when it fits comfortably and your workload is sensitive to code, structured output, specialized terminology, or subtle instruction following.
- Use Q8_0 as a reference when memory allows. It helps show whether an observed failure is associated with stronger weight compression, although it is not identical to the original FP16 or BF16 model.
- Test Q4_0 separately only when compatibility or a specific backend makes it relevant. Do not assume its results transfer to Q4_K_M.
If Q4 quality is insufficient, first compare Q5 from the same base model. If both are weak, changing to a better-suited base model may matter more than escalating to Q8. If Q5 quality is acceptable but does not fit, reduce context, parallelism, or cache memory before accepting heavy CPU offload. Re-quantizing an already quantized file should be avoided when a high-precision source is available: the official llama.cpp quantizer warns that re-quantization can severely reduce quality compared with quantizing from 16-bit or 32-bit input.
The durable decision rule is simple: file size determines whether a candidate enters the test; your hardware measures its speed; your workload decides whether its quality is sufficient. Q4, Q5, and Q8 are useful starting labels, not final verdicts.
Frequently asked questions
Is Q5 always better than Q4?
No. Q5 usually retains more weight precision, but the practical result depends on the base model, quantizer, runtime, prompt, and hardware. A Q4 file that stays fully on the GPU can also be more responsive than a Q5 file that requires substantial CPU offload. Compare both with the same settings and task suite.
How much VRAM does a Q4 model need?
The GGUF file size is only the starting point. Add KV-cache memory, compute buffers, backend overhead, and headroom for the operating system or display. Context length and parallel requests can materially increase the total. Use the worked method in How Much VRAM Does a Local LLM Need? before downloading a large model.
Does Q8_0 have the same quality as FP16 or BF16?
No. Q8_0 is still a quantized representation and should not be treated as identical to the higher-precision source. It is useful as a relatively high-precision local reference when it fits, but a direct comparison is needed for quality-sensitive workloads.
Does a smaller GGUF always run faster?
No. Smaller weights can reduce memory traffic, but throughput also depends on the available CPU or GPU kernels, offloaded layers, batch sizes, context depth, and memory bandwidth. Measure prompt processing and token generation separately with the same runtime build. For the broader hardware trade, see CPU vs. GPU for Local LLMs.
Can Ollama and llama.cpp use the same quantization labels?
They may expose GGUF-based models with familiar labels, but the surrounding defaults and model packaging can differ. Confirm the exact model file, template, context, and runtime settings before comparing results. llama.cpp vs. Ollama explains the runtime-level differences.
Primary sources
All sources below were checked on 2026-07-24:
- llama.cpp quantization documentation: quantizer options, re-quantization warning, multimodal conversion, and example Q4_K_M sizes.
- llama.cpp tensor encoding schemes: block layouts and the documented Q4_K, Q5_K, Q4_0, and Q8_0 encoding values.
- GGML GGUF specification: GGUF structure, metadata, tensor types, alignment, and quantization-version fields.
- llama.cpp llama-bench documentation: benchmark parameters, output fields, repetitions, and the exclusion of tokenization and sampling time.
- llama.cpp project documentation: GGUF runtime usage, perplexity tooling, and benchmark entry points.