The best local LLMs for the RTX 5060

Qwen3.5 9B (9B) fits, with little room to spare in the RTX 5060's memory at Q4_K_M, using about 7.2GB with 8K context and generating ~55 tokens/sec. It is the most capable model in our catalog that runs well on this card. It is also our top pick for coding, reasoning and working with images. It is also the largest model that runs fully on this GPU.

Qwen3.5 9B is the largest of the small Qwen3.5 models released in March 2026, with built-in image understanding, a 262K context, and switchable thinking. It uses Qwen's hybrid attention, where most layers are linear-attention layers with a fixed-size state, so long contexts cost far less memory than on older 8B models. It is one of the strongest models that fits an 8GB GPU: at Q4_K_M it needs about 7GB with 8K context, and 12GB cards can run Q8_0. It replaces Qwen3 8B and, for most uses, Qwen3 14B.

Gemma 4 E4B (8B) fits comfortably in the RTX 5060's memory at Q4_K_M, using about 6.3GB with 8K context and generating ~62 tokens/sec. It is the next-best model that runs well here, and the one to try if the top pick doesn't suit you. It is also our top pick for speed.

Gemma 4 E4B is the larger of Google's two on-device Gemma 4 models: about 4.5B effective parameters (8B including per-layer embeddings) with text, image, and audio input and an optional thinking mode. It replaces Gemma 3n and the small Gemma 3 models, and it ships under Apache 2.0 instead of the old Gemma license. At Q4_K_M it needs about 6GB with 8K context, so it runs on any 8GB GPU, on 16GB laptops and Macs, and even on CPU-only machines at usable speed. Its KV cache is tiny, so the full 128K context adds only about 2GB.

Qwen3.6 35B-A3B (35B (3B active)) runs with part of the model in system RAM in the RTX 5060's memory at Q4_K_M, using about 24GB with 8K context and generating about 28 tokens/sec with CPU offload. Because it is a mixture-of-experts model, only 3B parameters are active per token, so offloading experts to system RAM still gives usable speed (assumes 64GB of system RAM).

Qwen3.6 35B-A3B is the April 2026 refresh of Qwen's fast mixture-of-experts model, focused on agentic coding and on preserving its reasoning across multi-turn tool use. With only 3B active parameters it generates text very quickly, and its hybrid linear attention keeps the KV cache tiny: the full 262K context needs about 5GB. At Q4_K_M it needs about 24GB with 8K context, which is tight on a 24GB card and comfortable on 32GB GPUs and Macs with 36GB or more; 12GB to 16GB GPUs run it well with experts offloaded to system RAM. It replaces Qwen3 30B-A3B and Qwen3 Coder 30B-A3B for most local users.