# NVIDIA GeForce RTX 5060 for local LLMs

> Only 8GB, which is fine for 7B to 8B models; spend a little more for 16GB.

The RTX 5060 has 8GB of GDDR7 and 448 GB/s of memory bandwidth, which decides how big a model fits and how fast it generates text. It comfortably runs models up to about 8B parameters: the largest in our catalog is Gemma 4 E4B at Q4_K_M (6.3GB with 8K context, ~62 tokens/sec). Our top all-round pick for it is Qwen3.5 9B (Q4_K_M, ~55 tokens/sec).

## Key specs

- Memory: 8GB GDDR7
- Memory bandwidth: 448 GB/s
- Architecture: Blackwell (2025)
- Power: 145W
- Typical price: ~$450 (checked 2026-10)
- Software: CUDA, Vulkan

## Best models for this GPU

- **Best overall (also coding, reasoning, vision, largest that fits):** Qwen3.5 9B at Q4_K_M, ~7.2GB with 8K context, ~55 tokens/sec
- **Runner-up (also fastest):** Gemma 4 E4B at Q4_K_M, ~6.3GB with 8K context, ~62 tokens/sec
- **Stretch pick (CPU offload):** Qwen3.6 35B-A3B at Q4_K_M, ~23.7GB with 8K context, about 28 tokens/sec with CPU offload

## Compatibility (best quantization that fits, 8K context)

| Model | Params | Quant | Memory | Speed | Verdict |
| --- | --- | --- | --- | --- | --- |
| Gemma 4 E4B | 8B | Q4_K_M | 6.3GB | ~62 tok/s | Runs well |
| Qwen3.5 9B | 9B | Q4_K_M | 7.2GB | ~55 tok/s | Fits, tight |
| Phi-4 | 14.7B | Q4_K_M | 11.6GB | ~9.4 tok/s | Partial offload |
| gpt-oss-20b | 20.9B (3.6B active) | MXFP4 | 15.0GB | ~28 tok/s | Partial offload |
| Gemma 4 26B A4B | 26B (3.8B active) | Q4_K_M | 17.3GB | ~26 tok/s | Partial offload |
| Qwen3.8 27B | 27B | Q4_K_M | 18.1GB | ~4.0 tok/s | Partial offload |
| Gemma 4 31B | 31B | Q4_K_M | 22.5GB | ~2.9 tok/s | Partial offload |
| Qwen3.6 35B-A3B | 35B (3B active) | Q4_K_M | 23.7GB | ~28 tok/s | Partial offload |
| GLM-4.5-Air | 106B (12B active) | - | - | - | Won't fit |
| Llama 4 Scout | 109B (17B active) | - | - | - | Won't fit |
| gpt-oss-120b | 117B (5.1B active) | - | - | - | Won't fit |
| Qwen3.5 122B-A10B | 122B (10B active) | - | - | - | Won't fit |
| DeepSeek V4 Flash | 284B (13B active) | - | - | - | Won't fit |
| GLM-5.3-Flash | 320B (18B active) | - | - | - | Won't fit |
| GLM-5.3 | 744B (40B active) | - | - | - | Won't fit |

## Pros
- Low price for a new Blackwell card
- Fast 448 GB/s memory for its class
- 145W board power

## Cons
- 8GB is too little for most current models
- Long context quickly runs out of memory

Full page: https://llamabuilds.ai/gpus/rtx-5060 · Best models: https://llamabuilds.ai/gpus/rtx-5060/best-models

Sources: [NVIDIA GeForce RTX 5060 family product page](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5060-family/)

---
Source: https://llamabuilds.ai/gpus/rtx-5060 · Updated October 14, 2026 · Llama Builds. Speeds marked ~ are estimates from our bandwidth model (https://llamabuilds.ai/methodology); measured results are labeled as such.