Mistral AI · Released June 2025
Mistral Small 3.2 24B Instruct
What it takes to run Mistral Small 3.2 24B Instruct locally: memory by quantization, the GPUs that handle it, and how fast it runs.
Updated
- Parameters
- 24B
- Architecture
- Dense
- Context
- 128K tokens
- License
- Apache 2.0 license
- Memory at Q4_K_M
- ~17GB
What do you need to run Mistral Small 3.2 24B Instruct?
Mistral Small 3.2 24B Instruct needs about 17GB of VRAM at Q4_K_M with 8K context (14GB for the weights plus KV cache and overhead). The smallest GPU that runs it comfortably at Q3_K_M is a 16GB card like the RTX 4060 Ti 16GB.
See the full hardware guide for Mistral Small 3.2 24B Instruct
What to buy
GPUs for Mistral Small 3.2 24B Instruct
Speeds are estimates at the quantization shown with 8K context. We may earn a commission when you buy through these links. Learn more.
Memory
Mistral Small 3.2 24B Instruct VRAM requirements by quantization
Lower-bit quantizations shrink the model at some cost to quality. Q4_K_M is the usual sweet spot. Totals include the KV cache for the context length shown plus runtime overhead.
| Quantization | Quality | Weights | Total @ 8K ctx | Total @ 32K ctx |
|---|---|---|---|---|
| Q2_K | Rough | 8.9 GB | 11 GB | 15 GB |
| Q3_K_M | Usable | 12 GB | 14 GB | 18 GB |
| Q4_K_M | RecommendedDefault | 14 GB | 17 GB | 21 GB |
| Q5_K_M | Very good | 17 GB | 19 GB | 23 GB |
| Q6_K | Near-lossless | 20 GB | 22 GB | 26 GB |
| Q8_0 | Practically lossless | 25 GB | 28 GB | 32 GB |
| BF16 | Full precision | 47 GB | 51 GB | 55 GB |
KV cache note: Standard grouped-query attention: 2 x 40 layers x 8 KV heads x 128 dims x 2 bytes = 163,840 bytes (160 KiB) per token, about 1.3GB at 8K and 21GB at the full 128K.
Performance
How fast Mistral Small 3.2 24B Instruct runs
Single-user generation speed on popular GPUs that fit the model entirely in memory, at the best quantization each can hold.
RTX 5080
Q3_K_M · 16GB
~62tok/s
RTX 5070 Ti
Q3_K_M · 16GB
~59tok/s
RTX 5090
Q8_0 · 32GB
~54tok/s
RTX PRO 6000
Q8_0 · 96GB
~54tok/s
RTX 4080 Super
Q3_K_M · 16GB
~49tok/s
RTX 4090
Q5_K_M · 24GB
~46tok/s
RTX 4070 Ti Super
Q3_K_M · 16GB
~45tok/s
RTX PRO 5000 72GB
Q8_0 · 72GB
~41tok/s
RTX 3090
Q5_K_M · 24GB
~38tok/s
RX 7900 XTX
Q5_K_M · 24GB
~35tok/s
Overview
About Mistral Small 3.2 24B Instruct
Mistral Small 3.2 is the June 2025 update of Mistral's dense 24B model, with image input, reliable function calling, and a 128K context under Apache 2.0. Mistral replaced the line in March 2026 with Mistral Small 4, a 119B mixture-of-experts that needs roughly 75GB at 4-bit, so 24GB GPU owners are now better served by Gemma 4 31B or Qwen3.8 27B. At Q4_K_M it needs about 17GB with 8K context, which fits a 24GB card with room for longer context, or a 16GB card at Q3_K_M. It has no reasoning mode.
Strengths
- +Fast, reliable function calling and structured output
- +Image input and 128K context
- +Apache 2.0 license
- +Fits 24GB GPUs with room for long context
Weaknesses
- −No reasoning mode; behind 2026 models on math and hard coding tasks
- −Superseded by Mistral Small 4, which is far larger
- −Large KV cache for a 24B model (160 KiB per token)
Setup
How to run Mistral Small 3.2 24B Instruct
$ ollama run mistral-small3.2:24b$ llama-server -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M -c 8192 -ngl 99Weights: mistralai/Mistral-Small-3.2-24B-Instruct-2506 · GGUF: unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF
Alternatives
Models like Mistral Small 3.2 24B Instruct
Frequently asked questions
How much VRAM do you need to run Mistral Small 3.2 24B Instruct?
About 17GB at Q4_K_M with 8K tokens of context: 14.3GB of weights, 1.3GB of KV cache, and 1.0GB of runtime overhead. Higher-precision quants and longer contexts need more; see the table above.
What is the cheapest GPU that can run Mistral Small 3.2 24B Instruct?
The smallest card in our database that runs it comfortably is the RTX 4060 Ti 16GB (16GB), at Q3_K_M and ~20 tokens/sec.
Is Mistral Small 3.2 24B Instruct free for commercial use?
Mistral Small 3.2 24B Instruct is released under the Apache 2.0 license. That is a permissive license, so commercial use is allowed.
How long a context does Mistral Small 3.2 24B Instruct support?
Up to 131,072 tokens (128K). Long contexts need a lot of extra memory for the KV cache, so most local setups run 8K to 32K.
Sources: Mistral Small 3.2 24B Instruct model card (Hugging Face), Mistral Small 4 model card (Hugging Face). Capability score basis: Placed between the Llama 3.1 8B and Qwen3 32B anchors from its Artificial Analysis Intelligence Index v3 standing (October 2025).