Google · Released March 2025
Gemma 3 12B
What it takes to run Gemma 3 12B locally: memory by quantization, the GPUs that handle it, and how fast it runs.
Updated
- Parameters
- 12.2B
- Architecture
- Dense
- Context
- 128K tokens
- License
- Gemma Terms of Use
- Memory at Q4_K_M
- ~9GB
What do you need to run Gemma 3 12B?
Gemma 3 12B needs about 9.0GB of VRAM at Q4_K_M with 8K context (7.3GB for the weights plus KV cache and overhead). The smallest GPU that can hold it is a 8GB card like the RTX 5060 at Q3_K_M, with little memory to spare. For a good balance of price and speed we recommend the RTX 3060 12GB (~29 tokens/sec).
What to buy
GPUs for Gemma 3 12B
Speeds are estimates at the quantization shown with 8K context. We may earn a commission when you buy through these links. Learn more.
Memory
Gemma 3 12B VRAM requirements by quantization
Lower-bit quantizations shrink the model at some cost to quality. Q4_K_M is the usual sweet spot. Totals include the KV cache for the context length shown plus runtime overhead.
| Quantization | Quality | Weights | Total @ 8K ctx | Total @ 32K ctx |
|---|---|---|---|---|
| Q2_K | Rough | 4.5 GB | 6.1 GB | 7.7 GB |
| Q3_K_M | Usable | 5.9 GB | 7.5 GB | 9.1 GB |
| Q4_K_M | RecommendedDefault | 7.3 GB | 9.0 GB | 11 GB |
| Q5_K_M | Very good | 8.6 GB | 10 GB | 12 GB |
| Q6_K | Near-lossless | 9.9 GB | 12 GB | 13 GB |
| Q8_0 | Practically lossless | 13 GB | 14 GB | 16 GB |
| BF16 | Full precision | 24 GB | 26 GB | 27 GB |
KV cache note: 48 layers in a 5:1 pattern of 1,024-token sliding-window and global layers. Only the 8 global layers grow with context: 2 x 8 x 8 KV heads x 256 dims x 2 bytes = 65,536 bytes per token, about 8.6GB at the full 128K. The 40 sliding layers add a fixed buffer of about 0.3GB.
Performance
How fast Gemma 3 12B runs
Single-user generation speed on popular GPUs that fit the model entirely in memory, at the best quantization each can hold.
RTX 5090
Q8_0 · 32GB
~105tok/s
RTX PRO 6000
Q8_0 · 96GB
~105tok/s
RTX PRO 5000 72GB
Q8_0 · 72GB
~79tok/s
RTX 5080
Q6_K · 16GB
~72tok/s
RTX 5070 Ti
Q6_K · 16GB
~67tok/s
RTX 4090
Q8_0 · 24GB
~61tok/s
RTX 5070
Q5_K_M · 12GB
~59tok/s
RTX 6000 Ada
Q8_0 · 48GB
~58tok/s
RTX 4080 Super
Q6_K · 16GB
~56tok/s
RTX 4070 Ti Super
Q6_K · 16GB
~51tok/s
Overview
About Gemma 3 12B
Gemma 3 12B was Google's mid-size open model for 2025, with image input, support for over 140 languages, and a 128K context window. Gemma 4 has replaced it: Gemma 4 E4B covers the smaller tier and Gemma 4 26B A4B is far stronger for modestly more memory, both under the more permissive Apache 2.0 license. At Q4_K_M it needs about 9GB with 8K context, so it fits 12GB GPUs comfortably and 16GB cards at Q6_K or Q8_0. Its interleaved sliding-window attention keeps long-context memory low.
Strengths
- +Good conversational quality and writing for its size
- +Image input included
- +Sliding-window attention keeps the KV cache small at long context
Weaknesses
- −Superseded by Gemma 4, which is stronger at every size
- −No reasoning mode; weak at math and multi-step problems
- −Gemma license is more restrictive than the Apache 2.0 license used by Gemma 4
Setup
How to run Gemma 3 12B
$ ollama run gemma3:12b$ llama-server -hf bartowski/google_gemma-3-12b-it-GGUF:Q4_K_M -c 8192 -ngl 99$ mlx_lm.chat --model mlx-community/gemma-3-12b-it-4bitWeights: google/gemma-3-12b-it · GGUF: bartowski/google_gemma-3-12b-it-GGUF
Alternatives
Models like Gemma 3 12B
Frequently asked questions
How much VRAM do you need to run Gemma 3 12B?
About 9GB at Q4_K_M with 8K tokens of context: 7.3GB of weights, 0.8GB of KV cache, and 0.8GB of runtime overhead. Higher-precision quants and longer contexts need more; see the table above.
What is the cheapest GPU that can run Gemma 3 12B?
The smallest card in our database that can hold it is the RTX 5060 (8GB), at Q3_K_M with little memory to spare, generating ~58 tokens/sec.
Is Gemma 3 12B free for commercial use?
Gemma 3 12B is released under the Gemma Terms of Use. Commercial use is allowed with some conditions; read the license before shipping a product.
How long a context does Gemma 3 12B support?
Up to 131,072 tokens (128K). Long contexts need a lot of extra memory for the KV cache, so most local setups run 8K to 32K.
Sources: Gemma 3 12B IT model card (Hugging Face), Gemma 3 technical report. Capability score basis: Placed above the Llama 3.1 8B anchor from its Artificial Analysis Intelligence Index v3 standing (October 2025), adjusted up slightly for chat and vision quality.