If you are choosing a GPU, do not match its advertised VRAM to the model’s download size. The practical answer to how much VRAM for a local LLM is the sum of four parts: model weights, KV cache, runtime buffers, and headroom. Estimate each part for the model and context you will actually use.
Use this purchase-planning formula:
required VRAM ≈ weight file size + KV cache + runtime buffers + 10–20% headroomThe 10–20% is a planning allowance, not a runtime specification. Drivers, display use, backend choice, batch size, and implementation changes can move the observed total. A model that only fits with a few megabytes free is not a comfortable fit.
1. Estimate the model weights
For an unquantized model, start with parameter count times bytes per parameter:
weight GB ≈ parameters × bytes per parameter ÷ 1,000,000,000FP16 uses 2 bytes per parameter, so a nominal 7B model is about 14 GB and a 13B model about 26 GB before metadata or other allocations. Quantization stores most weights at lower precision, but Q4, Q5, and Q8 labels are not exact file-size multipliers. Block scales, metadata, and tensors kept at higher precision add overhead.
Use the actual quantized file size whenever it is published. A llama.cpp maintainer notes that loaded weight memory should be very close to the model file size in most cases (llama.cpp memory discussion). The figures and behavior in this article are checked as of 2026-07-20.
| Weight format | Planning bytes per parameter | Rough weight size, 7B | Rough weight size, 13B |
|---|---|---|---|
| FP16 | 2.00 | 14.0 GB | 26.0 GB |
| Q8_0 | 1.05–1.10 | 7.4–7.7 GB | 13.7–14.3 GB |
| Q5_K_M | 0.70–0.75 | 4.9–5.3 GB | 9.1–9.8 GB |
| Q4_K_M | 0.60–0.65 | 4.2–4.6 GB | 7.8–8.5 GB |
These are screening ranges, not promises for every architecture. For a real check, the published Llama 3.1 8B GGUF files are 4.92 GB at Q4_K_M, 5.73 GB at Q5_K_M, and 8.54 GB at Q8_0 (8B GGUF model card). The Llama 2 13B files are 7.87 GB, 9.23 GB, and 13.83 GB respectively (13B GGUF model card). Those are weight files, not complete VRAM requirements.
2. Calculate the KV cache
The KV cache stores attention keys and values for tokens in the active context. Its size grows roughly linearly with context length and concurrent sequences. For a standard transformer, use:
KV bytes = 2 × layers × context tokens × KV heads × head dimension × cache bytes
head dimension = hidden size ÷ attention headsThe leading 2 represents keys and values. cache bytes is 2 for FP16, 1 for an approximate Q8 estimate, or 0.5 for an approximate Q4 estimate; block quantization adds some overhead. Find num_hidden_layers, num_key_value_heads, hidden_size, and num_attention_heads in the model’s configuration. Do not substitute attention heads for KV heads when the model uses grouped-query attention (GQA).
Worked 8B example
Llama 3.1 8B has 32 layers, 8 KV heads, a hidden size of 4,096, and 32 attention heads. Its head dimension is 128. With an FP16 cache:
2 × 32 × 4,096 × 8 × 128 × 2 = 536,870,912 bytes = 512 MiBThat becomes about 1 GiB at 8K context and 4 GiB at 32K context. Add those numbers to the weight file. An 8B Q4_K_M model at 8K context therefore starts near 4.92 GB of weights plus roughly 1.07 GB of FP16 KV cache, before buffers and headroom.
Architecture matters. Llama 2 13B has 40 layers and 40 KV heads, so its FP16 cache is about 3.125 GiB at 4K context. This is why two models with similar weight-file sizes can need different VRAM. It is also why multiplying “gigabytes per billion parameters” cannot answer the whole question.
Current llama.cpp server documentation lists FP16 as the default KV type and supports Q8_0 and Q4 variants for both K and V caches (server options). Cache quantization can reduce memory, but confirm model and backend compatibility and compare output quality for your workload.
3. Add runtime buffers and headroom
Inference also needs output and compute buffers. llama.cpp’s logs separate weights, KV cache, output buffers, and compute buffers; the project notes that batch size, micro-batch size, Flash Attention, model architecture, and backend affect these allocations (memory calculation discussion). There is no single buffer constant that is accurate across runtimes.
For buying hardware, reserve at least 10–20% after weights and KV cache. Use the higher end when the GPU also drives a display, you want large prompt batches, or you have not tested the exact runtime. Once hardware is available, replace the allowance with measured startup logs at the intended context and batch settings.
4. Decide whether CPU offload is acceptable
CPU offload places some model layers or the KV cache in system RAM. It can make a model run when full GPU placement does not fit, but it does not make the missing memory disappear. You need enough RAM for the offloaded data, and transfers across PCIe can reduce token generation speed. Prompt processing may also slow, especially with long contexts.
Use offload as a deliberate compromise:
- Prefer full GPU placement when low latency is the priority.
- Partial layer offload is reasonable when buying the next VRAM tier is not justified.
- KV offload saves VRAM but makes long-context traffic more dependent on system memory bandwidth and transfers.
- Integrated-memory systems do not have a separate VRAM pool; budget unified memory for the model, cache, runtime, operating system, and other applications.
As of 2026-07-20, llama.cpp exposes layer/device placement controls and enables KV offload by default when applicable; it also provides --no-kv-offload for keeping the cache off the GPU. Exact behavior remains backend-dependent, so treat a successful load as the start of testing, not the end.
A practical buying checklist
- Choose the exact model repository and quant file, then record its file size.
- Choose a realistic total context: prompt, conversation history, retrieved text, and expected output all consume tokens.
- Calculate FP16 KV cache from the model configuration. Reduce it only if you plan to test quantized cache.
- Multiply cache needs by the number of simultaneous sequences your server reserves.
- Add runtime headroom, then round up to an available VRAM tier.
- If the result barely misses, compare a smaller quant, shorter context, or CPU offload before changing hardware.
The durable rule is simple: file size tells you the weight cost; architecture and context tell you the cache cost; the runtime tells you the buffer cost. Buying against all three is safer than shopping by parameter count alone.