# How much VRAM do you need to run LLMs locally?

> At Q4_K_M, the default 4-bit quantization, a model's weights take about 0.6GB of VRAM per billion parameters. Add the KV cache for your context and about 1GB of runtime overhead. A 9B model needs about 7GB, a 27B dense model about 18GB, and gpt-oss-120b about 68GB at 8K context. 16GB covers 9B to 20B models, 24GB covers 27B to 35B, 32GB adds headroom for quality and long context, and 96GB or more is where 120B-class models fit without offloading.

For most people the number to remember is about 0.6GB of VRAM per billion parameters. That is what a model's weights take at Q4_K_M, the 4-bit format most local tools default to, and you add the KV cache for your context plus roughly a gigabyte of runtime overhead on top. In practice that comes to about 7GB for [Qwen3.5 9B](/models/qwen3-5-9b), about 18GB for the dense [Qwen3.8 27B](/models/qwen3-8-27b), and about 68GB for [gpt-oss-120b](/models/gpt-oss-120b) at 8K context.

Those numbers line up with hardware tiers. An 8GB card handles small models, 16GB covers 9B to 20B models, 24GB is the practical floor for 27B to 35B models, 32GB buys room for higher quality and long context, and you need 96GB or more before 120B-class models fit without spilling into system RAM. The rest of this guide shows where the numbers come from so you can work them out for any model. If you'd rather skip the math, the [VRAM calculator](/tools/vram-calculator) does it for every model and GPU in our database.

## What fills your VRAM

Three things take up memory when a model runs on a GPU.

**Weights.** These are the model's parameters, and their size is the parameter count times the bits stored per weight, divided by eight. Q4_K_M averages about 4.85 bits per weight in our estimates, so a 27B model needs 27 × 4.85 ÷ 8, or about 16.4GB. llama.cpp's own measurements agree: its [quantization docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) list Llama 3.1 8B at 4.89 bits per weight and 4.58GiB for Q4_K_M, 7.95GiB for Q8_0, and 14.96GiB at 16-bit. When we know a model's real file size, we use that instead of the formula.

**KV cache.** For every token in the context window, the model stores attention keys and values so it doesn't have to recompute them for each new token. This grows in a straight line with context length, and the cost per token depends on the model's architecture. Qwen3.8 27B stores 64KiB per token, so 8K tokens take about 0.5GB and 128K tokens take about 8.6GB.

**Overhead.** The CUDA or Metal runtime, compute buffers and scratch space need room too. We budget 0.6GB plus 3% of the weights. If a monitor is plugged into the same GPU, the desktop takes some VRAM as well.

> **The formula:** Total VRAM ≈ weights + KV cache + overhead. At Q4_K_M, the weights take about 0.6GB per billion parameters. Leave 10 to 15% of the card free so the model loads reliably and has room for longer prompts. Our [methodology](/methodology) page explains each term.

## VRAM by model and quantization

The table below comes from the same estimator that powers our GPU and model pages. It assumes 8K of context, a reasonable working size for chat and short coding sessions, with a 16-bit KV cache. The gpt-oss models ship in their own 4-bit format called MXFP4, so they show a single number.

| Model | Size | Q4_K_M | Q6_K | Q8_0 |
| --- | --- | --- | --- | --- |
| Qwen3.5 9B | 9B | 7.2GB | 9.5GB | 12.0GB |
| gpt-oss-20b | 20.9B (3.6B active) | 15.0GB (MXFP4) | - | - |
| Gemma 4 26B A4B | 26B (3.8B active) | 17.3GB | 23.2GB | 29.5GB |
| Qwen3.8 27B | 27B | 18.1GB | 24.8GB | 31.6GB |
| Gemma 4 31B | 31B | 22.5GB | 29.7GB | 35.9GB |
| Qwen3.6 35B-A3B | 35B (3B active) | 23.7GB | 32.0GB | 41.0GB |
| gpt-oss-120b | 117B (5.1B active) | 68.3GB (MXFP4) | - | - |
| Qwen3.5 122B-A10B | 122B (10B active) | 77.0GB | 104.5GB | 134.3GB |

_Total memory with 8K context, estimated._

For models that aren't in our database, the weights alone scale with parameter count. Add the KV cache and about 1GB of overhead to these.

| Parameters | Q4_K_M | Q8_0 | 16-bit (BF16) |
| --- | --- | --- | --- |
| 4B | about 2.4GB | about 4.3GB | about 8GB |
| 8B | about 4.8GB | about 8.5GB | about 16GB |
| 14B | about 8.5GB | about 15GB | about 28GB |
| 32B | about 19GB | about 34GB | about 64GB |
| 70B | about 42GB | about 74GB | about 140GB |
| 120B | about 73GB | about 128GB | about 240GB |
| 235B | about 143GB | about 250GB | about 470GB |

Real files land close to these figures. The Q4_K_M file for [Llama 3.3 70B](/models/llama-3-3-70b) is 42.5GB, for example. If you want to understand what Q4_K_M and Q8_0 actually do to a model, our [quantization guide](/guides/llm-quantization-explained) covers the formats and the measured quality loss at each level.

## Context length is the hidden cost

Weights are a fixed cost. The KV cache grows with every token you keep in context, and long-context work can make it the largest item on the bill. Here is the 16-bit KV cache alone for a few models:

| Model | 8K tokens | 32K tokens | 128K tokens |
| --- | --- | --- | --- |
| Qwen3.6 35B-A3B | about 0.2GB | about 0.7GB | about 2.7GB |
| Qwen3.5 9B | about 0.3GB | about 1.1GB | about 4.3GB |
| gpt-oss-120b | about 0.3GB | about 1.2GB | about 4.8GB |
| Qwen3.8 27B | about 0.5GB | about 2.1GB | about 8.6GB |
| Gemma 4 31B | about 1.7GB | about 3.7GB | about 11.7GB |
| GLM-4.5-Air | about 1.5GB | about 6.2GB | about 24.7GB |
| Llama 3.3 70B (2024) | about 2.7GB | about 10.7GB | about 42.9GB |

The spread comes from architecture: how many layers keep a cache for the full context, how many key and value heads each layer has, and whether some layers only look at a sliding window of recent tokens. Llama 3.3 70B keeps a full-length cache on all 80 of its layers, which is why 128K of context costs more than its 4-bit weights. Most of the 2026 models in our database are far cheaper per token, but the differences still matter. Qwen3.8 27B at Q4_K_M needs about 18GB at 8K and about 22GB at 64K, both fine on a 24GB card, but about 26GB at 128K.

| Model | Size | Q4_K_M | Q8_0 |
| --- | --- | --- | --- |
| Qwen3.8 27B | 27B | 18.1GB | 31.6GB |
| Gemma 4 31B | 31B | 22.5GB | 35.9GB |
| gpt-oss-120b | 117B (5.1B active) | 68.3GB (MXFP4) | - |
| GLM-4.5-Air | 106B (12B active) | 68.3GB | 118.1GB |

_Total memory with 8K context, estimated._

### Shrinking the KV cache

You can store the cache at lower precision. In llama.cpp the flags are `-ctk` and `-ctv` (cache type for keys and values), which default to `f16` and accept `q8_0` or `q4_0`, according to the [llama-server docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). Ollama uses the `OLLAMA_KV_CACHE_TYPE` environment variable and needs flash attention, which it turns on automatically when the GPU supports it. [Ollama's FAQ](https://docs.ollama.com/faq) describes `q8_0` as about half the memory of `f16` with usually no noticeable quality impact, and `q4_0` as a quarter of the memory with a loss that can become more noticeable at long context.

With an 8-bit cache, Qwen3.8 27B at Q4_K_M and 128K of context drops from about 26GB to about 22GB, which brings it back inside a 24GB card.

> **Check your context setting:** Ollama defaults to a 4,096-token context window and only raises it if you set `OLLAMA_CONTEXT_LENGTH` or `num_ctx`. Running several requests in parallel multiplies the context memory too: Ollama's docs say required memory scales with `OLLAMA_NUM_PARALLEL` times the context length.

## Dense vs mixture-of-experts models

A dense model reads every weight for every token. A mixture-of-experts (MoE) model routes each token through a small subset of its weights, so only the "active" parameters are read per token. The whole model still has to sit in memory, though.

The short version is that memory follows total parameters and speed follows active parameters. [Qwen3.6 35B-A3B](/models/qwen3-6-35b-a3b) has 35B parameters in total but about 3B active per token. At Q4_K_M it needs about 24GB, more than the dense Qwen3.8 27B's 18GB, yet on an RTX 3090 it generates at about 138 tok/s by our estimate against about 39 tok/s for the 27B. OpenAI's gpt-oss-120b has 117B total parameters and 5.1B active, according to [Hugging Face's gpt-oss announcement](https://huggingface.co/blog/welcome-openai-gpt-oss).

This changes how you shop. For MoE models, memory capacity matters more than raw bandwidth, which is why large unified-memory machines handle them well.

## When the model doesn't fit: offloading to system RAM

llama.cpp, Ollama and LM Studio can keep part of a model in system RAM and run it from there. How well that works depends on the model type.

For dense models, it's painful. Every token has to read the CPU-side weights through system memory, which is many times slower than GDDR6X or GDDR7. Qwen3.8 27B at Q4_K_M on an 8GB [RTX 5060](/gpus/rtx-5060) runs at about 4 tok/s by our estimate.

For MoE models, it works surprisingly well. You keep the attention layers and shared weights on the GPU and push the expert weights, which are read sparsely, into RAM. llama.cpp's `--cpu-moe` and `--n-cpu-moe` flags do exactly this, and recent llama-server builds try to fit the model to your available memory automatically with `--fit`, which is on by default. By our estimate, gpt-oss-120b runs at about 21 tok/s on a single 24GB card with 64GB of RAM, and Qwen3.6 35B-A3B runs at about 28 tok/s on an 8GB card.

Our offload estimates assume 64GB of system RAM. If you plan to offload, system RAM capacity and speed become part of your GPU budget. In Ollama, `ollama ps` shows the split; anything other than "100% GPU" means part of the model is running from system memory.

## Unified memory: Macs, Ryzen AI Max and DGX Spark

Apple silicon Macs, AMD's Ryzen AI Max chips and NVIDIA's [DGX Spark](/gpus/dgx-spark) share one pool of memory between the CPU and GPU. That makes big models fit cheaply. We count about 75% of a 128GB machine, or about 96GB, as usable for the model.

The catch is bandwidth. Generation speed depends on how fast the chip can read the weights, and unified memory is slower than high-end GDDR7. By our estimates, gpt-oss-120b runs at about 64 tok/s on a [Mac Studio M5 Max 128GB](/gpus/mac-studio-m5-max-128gb) (614 GB/s), about 35 tok/s on a [Ryzen AI Max+ 395](/gpus/ryzen-ai-max-plus-395) (256 GB/s), and about 172 tok/s on an [RTX PRO 6000 Blackwell](/gpus/rtx-pro-6000-blackwell) (1,792 GB/s). Dense models suffer more: Qwen3.8 27B at Q8_0 runs at about 7 tok/s on the Ryzen AI Max+ 395. Our [Mac and mini PC guide](/best/macs-and-mini-pcs-for-local-llms) goes deeper on these machines.

## What fits at each VRAM tier

All speeds below are our estimates for one user generating text at 8K context. "Tight" means the model fits but uses more than 88% of memory, leaving little room for longer prompts.

| VRAM | Example hardware | Runs well | Stretch |
| --- | --- | --- | --- |
| 8GB | [RTX 5060](/gpus/rtx-5060) | Gemma 4 E4B at Q4_K_M (about 6GB, ~62 tok/s) | Qwen3.5 9B at Q4_K_M (about 7GB, tight); Qwen3.6 35B-A3B with offload (~28 tok/s) |
| 12GB | [RTX 3060 12GB](/gpus/rtx-3060-12gb) | Qwen3.5 9B at Q6_K (about 9.5GB, ~30 tok/s) | gpt-oss-20b with offload (~37 tok/s) |
| 16GB | [RTX 5060 Ti 16GB](/gpus/rtx-5060-ti-16gb), [RTX 5070 Ti](/gpus/rtx-5070-ti) | Qwen3.5 9B at Q8_0 (about 12GB); gpt-oss-20b (about 15GB, tight, ~74 tok/s on the 5060 Ti) | Qwen3.8 27B at Q3_K_M (about 15GB, tight) |
| 24GB | [RTX 3090](/gpus/rtx-3090), [RTX 4090](/gpus/rtx-4090), [RX 7900 XTX](/gpus/rx-7900-xtx) | Qwen3.8 27B at Q5_K_M (about 21GB, ~34 tok/s on a 3090); Qwen3.6 35B-A3B at Q3_K_M (~159 tok/s) | Gemma 4 31B at Q4_K_M (about 22.5GB, tight) |
| 32GB | [RTX 5090](/gpus/rtx-5090), [Radeon AI PRO R9700](/gpus/radeon-ai-pro-r9700) | Qwen3.8 27B at Q6_K (~60 tok/s on a 5090); Qwen3.6 35B-A3B at Q5_K_M (~196 tok/s) | Qwen3.8 27B at Q4_K_M with 128K context (about 26GB) |
| 48GB | Two RTX 3090s | Qwen3.8 27B at Q8_0 (~22 tok/s); Gemma 4 31B at Q8_0 | gpt-oss-120b with offload (~32 tok/s) |
| 96GB+ | RTX PRO 6000, four RTX 3090s, 128GB unified memory | gpt-oss-120b (about 68GB, ~172 tok/s on an RTX PRO 6000, ~56 tok/s on four 3090s); Qwen3.5 122B-A10B at Q4_K_M (about 77GB) | GLM-4.5-Air at Q5_K_M (about 80GB) |

Two things to keep in mind with multiple GPUs. First, we count 97% of the combined memory as usable, because splitting a model duplicates some buffers, so two 24GB cards give about 46.6GB. Second, splitting by layer adds capacity but not speed. The cards take turns working through the layers, so two 3090s are no faster than one on a model that fits on one card, and usually a little slower.

Above 96GB you're into server territory. [DeepSeek V4 Flash](/models/deepseek-v4-flash) needs about 168GB at its native FP4, and [GLM-5.3](/models/glm-5-3) needs about 466GB at Q4_K_M. Those call for multi-GPU servers or 256GB to 512GB Mac Studios.

## Checking what you're actually using

Estimates get you to the right card; real usage varies with the runtime version, batch size and how many requests you serve at once. On NVIDIA cards, `nvidia-smi` shows memory in use per GPU. llama.cpp prints the size of each buffer when it loads a model, and Ollama's `ollama ps` shows the total size and the CPU and GPU split.

If you're buying hardware, size it for the model you want next year rather than the one you run today. Models keep getting more capable at the same size, but context windows and agent workloads keep pushing memory use up. Our [GPU buying guide](/best/gpus-for-local-llms) turns these tiers into specific picks, and each GPU page has a best-models list, like this one for the [RTX 3090](/gpus/rtx-3090/best-models).

## Frequently asked questions

### How much VRAM do I need for a 7B to 9B model?

About 7GB at Q4_K_M with 8K of context, or about 12GB at Q8_0. An 8GB card runs a 9B model at 4-bit with little room left for long context, while a 12GB card gives comfortable headroom or room for Q6_K.

### Is 24GB of VRAM enough for local LLMs?

For most single-user work, yes. 24GB holds 27B to 35B models at 4 to 5 bits with 8K to 64K of context; Qwen3.8 27B at Q5_K_M needs about 21GB. It is not enough for 70B dense or 120B mixture-of-experts models unless you offload part of the model to system RAM.

### How much VRAM does context length use?

It depends on the model. With a 16-bit KV cache, Qwen3.8 27B uses about 0.5GB at 8K tokens and about 8.6GB at 128K, while the older Llama 3.3 70B uses about 43GB at 128K. Storing the cache in 8-bit halves those numbers.

### Can I run a model that is bigger than my VRAM?

Yes. llama.cpp, Ollama and LM Studio can keep part of the model in system RAM. Dense models slow to a few tokens per second when that happens, but mixture-of-experts models stay usable: gpt-oss-120b on a 24GB card with 64GB of RAM runs at about 21 tok/s by our estimate.

### How much of a Mac's unified memory can a model use?

We count about 75% of unified memory as usable for models, so a 128GB Mac Studio or Ryzen AI Max system gets about 96GB. That fits gpt-oss-120b (about 68GB at 8K context), but generation is slower than on a high-end NVIDIA card because unified memory has less bandwidth.

---
Source: https://llamabuilds.ai/guides/how-much-vram-do-you-need · Updated October 14, 2026 · Llama Builds. Speeds marked ~ are estimates from our bandwidth model (https://llamabuilds.ai/methodology); measured results are labeled as such.