# LLM quantization explained: what Q4_K_M, Q8_0 and the rest mean > Quantization stores model weights in fewer bits so models take less memory and run faster. Q8_0 (about 8.5 bits per weight) is close to lossless, Q6_K and Q5_K_M are hard to tell apart from it, and Q4_K_M (about 4.9 bits) is the usual default because it cuts the model to under a third of its 16-bit size for a small, measurable quality loss. Below 4 bits quality falls faster, though a bigger model at 3 or 4 bits usually still beats a smaller one at 8. Run the largest model that fits at Q4_K_M or better, with room left for context. Quantization stores a model's weights with fewer bits, so the model takes less memory and runs faster. Open models are usually released at 16 bits per weight. Q8_0 cuts that to about 8.5 bits and is close to lossless, while Q4_K_M uses about 4.9 bits, a little under a third of the original size, for a small but measurable drop in quality. That trade is why Q4_K_M is the default in most local tools. The practical rule is to run the largest model that fits your memory at Q4_K_M or better, with room left for context. If you have memory to spare, step up to Q5_K_M, Q6_K or Q8_0. Drop below 4 bits only when it lets you run a much larger model. The rest of this guide explains the names, shows what published measurements say about quality at each level, and covers the newer 4-bit floating point formats (MXFP4 and NVFP4) and Apple's MLX. ## Why smaller also means faster When a model generates a token, it reads every active weight from memory once. On a GPU that read is the bottleneck, so generation speed is roughly memory bandwidth divided by the size of the weights. Halve the bytes and you come close to doubling the tokens per second. llama.cpp's own reference numbers for Llama 3.1 8B show the effect: generation ran at about 29 tok/s at 16-bit, 51 tok/s at Q8_0 and 72 tok/s at Q4_K_M on the same machine, according to its [quantization README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). Prompt processing, where the model reads your input, depends more on raw compute and behaves differently. For chat, generation speed is the part you feel. ## How to read GGUF quant names GGUF is the model file format used by llama.cpp, Ollama and LM Studio. The quant name in a GGUF file tells you how the weights were packed: - **Q plus a number** means quantized, at roughly that many bits per weight. Q4 is about 4 bits, Q8 about 8. - **_0 and _1** are llama.cpp's original formats. Weights are split into small blocks that share a scale (and for _1, an offset). Q8_0 is the one still in wide use, because at 8 bits the simple approach is already close to lossless. - **K** marks the k-quants added to llama.cpp in 2023. Weights are grouped into super-blocks of 256, split into smaller blocks whose scales are themselves quantized, as described in the [original k-quants pull request](https://github.com/ggml-org/llama.cpp/pull/1684). - **S, M and L** (small, medium, large) describe how many sensitive tensors get extra bits. Per that same pull request, Q4_K_M stores half of the attention value tensors and half of the feed-forward down-projection tensors at 6-bit, and everything else at 4-bit. - **IQ** marks the i-quants, newer formats built for low bit rates. They work best with an importance matrix, a measurement of which weights matter most on a sample of calibration text. The real bits per weight run higher than the names suggest, because the per-block scales take space and the mixes keep some tensors at higher precision. Here is what llama.cpp measured for Llama 3.1 8B: | Quant | Bits per weight | Size vs 16-bit | | --- | --- | --- | | F16 | 16.00 | 100% | | Q8_0 | 8.50 | 53% | | Q6_K | 6.56 | 41% | | Q5_K_M | 5.70 | 36% | | Q4_K_M | 4.89 | 31% | | IQ4_XS | 4.46 | 28% | | Q3_K_M | 4.00 | 25% | | IQ3_XXS | 3.25 | 20% | | Q2_K | 3.16 | 20% | | IQ2_XXS | 2.38 | 15% | | IQ1_S | 2.00 | 13% | Source: [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). Your model's numbers will differ a little, but the ratios hold. ### Importance matrices and dynamic quants Two refinements show up in file names on Hugging Face. An importance matrix ("imatrix") is measured by running the full-precision model over calibration text; llama.cpp's quantizer then uses it to decide where rounding errors hurt least, and its docs recommend it for reducing quality loss. Unsloth's Dynamic quants go a step further and pick a different quant type for each layer based on how sensitive it is ([Unsloth docs](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs)). Both help most at 4 bits and below. ## What you lose at each level Three kinds of evidence point the same way: almost nothing down to 5 or 6 bits, a small loss at 4, a clearer loss at 3, and a steep drop at 2 and below. ### Perplexity from the original k-quant work Perplexity measures how surprised a model is by real text; lower is better. The k-quants pull request reported these numbers for the original LLaMA 7B: | Quant | Perplexity | Change vs F16 | | --- | --- | --- | | F16 | 5.9066 | none | | Q6_K | 5.9110 | +0.07% | | Q5_K_M | 5.9208 | +0.24% | | Q4_K_M | 5.9601 | +0.91% | | Q3_K_M | 6.1503 | +4.1% | | Q2_K | 6.7764 | +14.7% | The same pull request found that a 30B model at Q2_K had lower perplexity than a 13B model at full 16-bit precision. That 2023 result is the origin of the advice to prefer a bigger model at lower bits. ### Benchmarks on a modern instruct model A January 2026 study ([arXiv 2601.14277](https://arxiv.org/abs/2601.14277)) ran Llama 3.1 8B Instruct through a benchmark suite at each GGUF level: | Quant | Average score | Perplexity | | --- | --- | --- | | F16 | 69.47 | 7.32 | | Q8_0 | 69.41 | 7.33 | | Q6_K | 69.23 | 7.35 | | Q5_K_M | 69.36 | 7.40 | | Q4_K_M | 69.15 | 7.56 | | Q3_K_M | 68.07 | 7.96 | | Q3_K_S | 65.49 | 8.96 | Down to Q4_K_M the average moved by about a third of a point, and Q5_K_M even scored slightly above Q6_K, which tells you the differences at that level are within noise. Q3_K_M lost 1.4 points. Q3_K_S lost about 4, with math hit hardest: GSM8K fell from 77.63 to 68.31. ### Vendor numbers for a larger model Unsloth publishes MMLU scores for its own quants of Gemma 3 27B: 71.5 at BF16, 71.60 at Q8_0, 71.87 at Q6_K, 71.23 at Q4_K_M, 70.70 at Q3_K_M, 59.20 at IQ2_XXS and 48.10 at IQ1_M ([Unsloth docs](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs)). These are a vendor's numbers for its own product, so treat them with some caution, but the shape matches the independent results: flat down to 4 bits, mild at 3, and a cliff at 2 bits and below. ### Averages can hide changed answers Benchmark averages can stay flat while individual answers change. The paper [Accuracy is Not All You Need](https://arxiv.org/abs/2407.09141) found that compressed models can match the original's accuracy while flipping a meaningful number of answers, with right-to-wrong and wrong-to-right changes canceling out. The authors suggest KL divergence, which compares the full probability distributions of the two models, as a better measure, and llama.cpp's tools can report it. If your work needs exact output, like long code edits or strict tool calls, test the quant on your own tasks and step up a level if you see odd mistakes. ## Bigger model at lower bits, or smaller model at higher bits? Usually the bigger model, down to about 4 bits and often to 3. On a 24GB card, [Qwen3.8 27B](/models/qwen3-8-27b) at Q5_K_M (about 21GB) is a better choice than [Qwen3.5 9B](/models/qwen3-5-9b) at Q8_0 (about 12GB) for almost any task. Below 3 bits the losses grow fast, and 1-bit and 2-bit files are best kept for very large mixture-of-experts models you couldn't run any other way. Unsloth, which publishes 1-bit quants, says in its docs that they should not be used for agentic work. Quantization also sets speed. This table shows the memory each level needs for three popular mid-size models: | Model | Size | Q3_K_M | Q4_K_M | Q5_K_M | Q6_K | Q8_0 | | --- | --- | --- | --- | --- | --- | --- | | Qwen3.8 27B | 27B | 14.8GB | 18.1GB | 21.6GB | 24.8GB | 31.6GB | | Gemma 4 31B | 31B | 18.5GB | 22.5GB | 26.0GB | 29.7GB | 35.9GB | | Qwen3.6 35B-A3B | 35B (3B active) | 19.2GB | 23.7GB | 27.7GB | 32.0GB | 41.0GB | _Total memory with 8K context, estimated._ On an [RTX 5090](/gpus/rtx-5090), our estimates for Qwen3.8 27B run from about 98 tok/s at Q3_K_M to about 80 tok/s at Q4_K_M, 69 at Q5_K_M, 60 at Q6_K and 47 at Q8_0. That's the other half of why Q4_K_M is the default. Going from Q8_0 to Q4_K_M buys about 70% more speed for a quality loss that is hard to measure, while going lower costs quality faster than it adds speed. ## Beyond GGUF: MLX, MXFP4 and NVFP4 ### MLX on Apple silicon Apple's MLX framework has its own quantization. By default it stores weights at 4 bits in groups of 64, each group with its own scale and bias, and it supports 1 to 8 bits plus the mxfp4, mxfp8 and nvfp4 modes, according to the [MLX documentation](https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html). With 16-bit scales and biases, 4-bit MLX works out to about 4.5 bits per weight, close to Q4_K_M. On a Mac, LM Studio can run both GGUF and MLX models, and either format at 4 bits is a sensible default. ### MXFP4 MXFP4 stores each weight as a 4-bit floating point number and gives every block of 32 values a shared 8-bit power-of-two scale, for about 4.25 bits per weight. OpenAI released [gpt-oss-20b](/models/gpt-oss-20b) and gpt-oss-120b with their mixture-of-experts weights in MXFP4 and the rest in BF16. [Hugging Face's gpt-oss announcement](https://huggingface.co/blog/welcome-openai-gpt-oss) notes this lets the 120B model fit on a single 80GB GPU and the 20B model in 16GB. llama.cpp runs MXFP4 natively, so use the MXFP4 GGUF as released instead of requantizing it. ### NVFP4 NVFP4 is NVIDIA's 4-bit format for Blackwell GPUs. Like MXFP4 it stores 4-bit floating point values, but it shares an FP8 scale across every 16 values instead of 32 and adds a per-tensor FP32 scale, for about 4.5 bits per weight. [NVIDIA's NVFP4 write-up](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) describes it as about 3.5 times smaller than FP16 and reports an accuracy loss of 1% or less against FP8 on DeepSeek-R1-0528. Blackwell cards like the RTX 50 series and the [RTX PRO 6000 Blackwell](/gpus/rtx-pro-6000-blackwell) have FP4 math in hardware; older cards like the RTX 3090 and 4090 do not. Runtime support varies, so check yours before downloading NVFP4 checkpoints. ## Quantizing the KV cache The KV cache, the memory that holds your context, is quantized separately from the weights. llama.cpp stores it at 16-bit by default; the `-ctk` and `-ctv` flags switch keys and values to `q8_0` or `q4_0` ([llama-server docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)). In Ollama, set `OLLAMA_KV_CACHE_TYPE`. [Ollama's FAQ](https://docs.ollama.com/faq) says `q8_0` uses about half the memory with usually no noticeable impact, while `q4_0` uses a quarter with a loss that may be more noticeable at long context. For Qwen3.8 27B at 128K tokens, that is the difference between about 8.6GB of cache at 16-bit, 4.3GB at 8-bit and 2.1GB at 4-bit. Our [VRAM guide](/guides/how-much-vram-do-you-need) has more on how context adds up. ## How to choose a quant 1. **Find out what fits.** Use the [VRAM calculator](/tools/vram-calculator) or a model's requirements page, like the one for [Qwen3.8 27B](/models/qwen3-8-27b/requirements). 2. **Start at Q4_K_M** for the largest model that fits with 10 to 15% of memory free. 3. **Spend spare memory on context first** if you work with long documents or code, then on Q5_K_M or Q6_K. Q8_0 makes sense when memory is plentiful. 4. **Below 4 bits, use imatrix or dynamic quants**, and only when that gets you a much larger model. 5. **Use native formats as released.** For gpt-oss, that means MXFP4. In practice, on a 16GB card like the [RTX 5060 Ti 16GB](/gpus/rtx-5060-ti-16gb), you can choose between Qwen3.8 27B at Q3_K_M (about 15GB and tight, ~26 tok/s estimated) and Qwen3.5 9B at Q8_0 (about 12GB); try both on your own tasks. On a 24GB [RTX 3090](/gpus/rtx-3090), Qwen3.8 27B at Q5_K_M (about 21GB, ~34 tok/s estimated) is the sweet spot. On a 32GB RTX 5090, Q6_K (about 24GB) leaves room for long context. For which card gets you to each of those tiers, see our [GPU buying guide](/best/gpus-for-local-llms). ## Frequently asked questions ### Is Q4_K_M good enough for everyday use? For most chat and coding work, yes. In a January 2026 study of Llama 3.1 8B Instruct, Q4_K_M averaged 69.15 across the benchmark suite against 69.47 at 16-bit, and perplexity rose from 7.32 to 7.56. The loss becomes more noticeable at Q3 and below. ### What is the difference between Q4_K_M and Q4_K_S? Both are 4-bit k-quants from llama.cpp. The M (medium) version stores half of the attention value tensors and feed-forward down-projection tensors at 6-bit instead of 4-bit, so it is slightly larger and slightly more accurate than the S (small) version. ### Should I use Q8_0 or Q4_K_M? Use Q8_0 when the model fits with room for context, since it is close to lossless: it scored 69.41 against 69.47 at 16-bit on Llama 3.1 8B Instruct. If Q8_0 doesn't fit, a bigger model at Q4_K_M usually beats a smaller one at Q8_0, and Q4_K_M also generates tokens faster because there is less data to read. ### What are I-quants like IQ4_XS and IQ3_XXS? I-quants are llama.cpp formats designed for low bit rates that work best with an importance matrix built from calibration text. IQ4_XS averages about 4.46 bits per weight against 4.89 for Q4_K_M, and the IQ2 and IQ1 levels make files of about 2 to 2.4 bits per weight possible, at a real quality cost. ### What is MXFP4 in gpt-oss? MXFP4 is a 4-bit floating point format where each block of 32 values shares one 8-bit scale, for about 4.25 bits per weight. OpenAI released gpt-oss with its mixture-of-experts weights in MXFP4, which is why gpt-oss-120b fits on a single 80GB GPU and gpt-oss-20b fits in 16GB without further quantization. ### Does quantizing the KV cache hurt quality? An 8-bit KV cache halves context memory, and Ollama's documentation says it usually has no noticeable impact on quality. A 4-bit cache uses a quarter of the memory but the loss can become noticeable at long context, so test it on your own work first. --- Source: https://llamabuilds.ai/guides/llm-quantization-explained · Updated October 14, 2026 · Llama Builds. Speeds marked ~ are estimates from our bandwidth model (https://llamabuilds.ai/methodology); measured results are labeled as such.