Alibaba Qwen · Released April 2025
Qwen3 8B
What it takes to run Qwen3 8B locally: memory by quantization, the GPUs that handle it, and how fast it runs.
Updated
- Parameters
- 8.2B
- Architecture
- Dense
- Context
- 32K tokens
- License
- Apache 2.0 license
- Memory at Q4_K_M
- ~7GB
What do you need to run Qwen3 8B?
Qwen3 8B needs about 7.0GB of VRAM at Q4_K_M with 8K context (5.0GB for the weights plus KV cache and overhead). The smallest GPU that runs it comfortably is a 8GB card like the RTX 5060.
What to buy
GPUs for Qwen3 8B
Speeds are estimates at the quantization shown with 8K context. We may earn a commission when you buy through these links. Learn more.
Memory
Qwen3 8B VRAM requirements by quantization
Lower-bit quantizations shrink the model at some cost to quality. Q4_K_M is the usual sweet spot. Totals include the KV cache for the context length shown plus runtime overhead.
| Quantization | Quality | Weights | Total @ 8K ctx | Total @ 32K ctx |
|---|---|---|---|---|
| Q2_K | Rough | 3.1 GB | 5.0 GB | 8.6 GB |
| Q3_K_M | Usable | 4.0 GB | 6.0 GB | 9.6 GB |
| Q4_K_M | RecommendedDefault | 5.0 GB | 7.0 GB | 11 GB |
| Q5_K_M | Very good | 5.9 GB | 7.9 GB | 12 GB |
| Q6_K | Near-lossless | 6.8 GB | 8.9 GB | 12 GB |
| Q8_0 | Practically lossless | 8.7 GB | 11 GB | 14 GB |
| BF16 | Full precision | 16 GB | 19 GB | 22 GB |
KV cache note: 2 x 36 layers x 8 KV heads x 128 dims x 2 bytes = 147,456 bytes (144 KiB) per token: about 1.2GB at 8K and 4.8GB at the native 32K context. YaRN scaling to 128K grows it to about 19GB.
Performance
How fast Qwen3 8B runs
Single-user generation speed on popular GPUs that fit the model entirely in memory, at the best quantization each can hold.
RTX 5090
Q8_0 · 32GB
~140tok/s
RTX PRO 6000
Q8_0 · 96GB
~140tok/s
RTX PRO 5000 72GB
Q8_0 · 72GB
~110tok/s
RTX 4090
Q8_0 · 24GB
~85tok/s
RTX 5080
Q8_0 · 16GB
~81tok/s
RTX 6000 Ada
Q8_0 · 48GB
~81tok/s
RTX 5070 Ti
Q8_0 · 16GB
~76tok/s
RTX 5070
Q6_K · 12GB
~73tok/s
RTX 3090
Q8_0 · 24GB
~72tok/s
M5 Ultra 512GB
Q8_0 · 512GB
~69tok/s
Measured
Qwen3 8B benchmarks
- RTX PRO 5000 72GBQ4_K_XL · llama.cpp160 tok/s Hardware Corner: RTX Pro 5000 Blackwell LLM Performance
- RTX 4060 Ti 16GBQ4_K_XL · llama.cpp46 tok/s Hardware Corner: RTX 4060 Ti 16GB LLM Performance
Overview
About Qwen3 8B
Qwen3 8B brought switchable thinking to the 8B class in April 2025 and was one of the most capable models you could run on an 8GB GPU. It has since been superseded by Qwen3.5 9B, which adds vision, a 262K context, and much stronger reasoning at a similar size. At Q4_K_M it needs about 7GB with 8K context, so it fits 8GB cards, and 12GB cards can run Q8_0. The native context is 32K, extendable to 128K with YaRN at the cost of a large KV cache.
Strengths
- +Thinking mode can be switched on for hard problems and off for quick answers
- +Fits 8GB GPUs at Q4_K_M with 8K context
- +Broad multilingual coverage (119 languages and dialects) and solid tool calling
- +Apache 2.0 license with no usage restrictions
Weaknesses
- −Superseded by Qwen3.5 9B, which is better on nearly every task
- −Native context is only 32K tokens
- −Thinking mode is verbose, which slows answers on low-bandwidth hardware
- −Limited world knowledge, typical of 8B models
Setup
How to run Qwen3 8B
$ ollama run qwen3:8b$ llama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M -c 8192 -ngl 99$ mlx_lm.chat --model mlx-community/Qwen3-8B-4bitWeights: Qwen/Qwen3-8B · GGUF: Qwen/Qwen3-8B-GGUF
Alternatives
Models like Qwen3 8B
Frequently asked questions
How much VRAM do you need to run Qwen3 8B?
About 7GB at Q4_K_M with 8K tokens of context: 5.0GB of weights, 1.2GB of KV cache, and 0.8GB of runtime overhead. Higher-precision quants and longer contexts need more; see the table above.
What is the cheapest GPU that can run Qwen3 8B?
The smallest card in our database that runs it comfortably is the RTX 5060 (8GB), at Q4_K_M and ~67 tokens/sec.
Is Qwen3 8B free for commercial use?
Qwen3 8B is released under the Apache 2.0 license. That is a permissive license, so commercial use is allowed.
How long a context does Qwen3 8B support?
Up to 32,768 tokens (32K). Long contexts need a lot of extra memory for the KV cache, so most local setups run 8K to 32K.
Sources: Qwen3 8B model card (Hugging Face), Qwen3 launch post. Capability score basis: Placed between the Llama 3.1 8B and Qwen3 32B anchors from its Artificial Analysis Intelligence Index v3 standing in reasoning mode (October 2025).