NVIDIA · Ada Lovelace · 2022
NVIDIA GeForce RTX 4090 for local AI
Still one of the fastest 24GB cards, though used prices now exceed its launch MSRP.
Updated · Speeds marked ~ are estimates; seemethodology
- VRAM
- 24GB
- Bandwidth
- 1,008 GB/s
- Power
- 450W
- Runs well
- 8 models
RTX 4090
Typically about $2,500 used
Can the RTX 4090 run LLMs?
The RTX 4090 has 24GB of GDDR6X and 1,008 GB/s of memory bandwidth, which decides how big a model fits and how fast it generates text. It comfortably runs models up to about 35B parameters: the largest in our catalog is Qwen3.6 35B-A3B at Q3_K_M (19GB with 8K context, ~175 tokens/sec). Our top all-round pick for it is Qwen3.8 27B (Q4_K_M, ~47 tokens/sec). Llama 3.3 70B Instruct only runs with partial CPU offload, at about 1.8 tokens/sec with CPU offload.
Recommendations
Best models for the RTX 4090
Best overall · also coding, reasoning, vision
Qwen3.8 27BQ4_K_M · 18GB · ~47 tokens/sec
Runs wellRunner-up
Gemma 4 31BQ4_K_M · 22GB · ~40 tokens/sec
Fits, tightFastest good model · also largest that fits
Qwen3.6 35B-A3BQ3_K_M · 19GB · ~175 tokens/sec
Runs well
Compatibility
What runs on the RTX 4090
Every current model in our catalog, at the best quantization that fits in 24GB with 8K tokens of context. "Partial offload" means part of the model runs from system RAM (we assume 64GB), which is much slower.
| Model | Parameters | Best quant | Memory | Speed | Verdict |
|---|---|---|---|---|---|
| Gemma 4 E4B | 8B | Q8_0 | 10 GB | ~79 tok/s | Runs well |
| Qwen3.5 9B | 9B | Q8_0 | 12 GB | ~69 tok/s | Runs well |
| Phi-4 | 14.7B | Q8_0 | 18 GB | ~49 tok/s | Runs well |
| gpt-oss-20b | 20.9B (3.6B active) | MXFP4 | 15 GB | ~135 tok/s | Runs well |
| Gemma 4 26B A4B | 26B (3.8B active) | Q5_K_M | 20 GB | ~125 tok/s | Runs well |
| Qwen3.8 27B | 27B | Q4_K_M | 18 GB | ~47 tok/s | Runs well |
| Gemma 4 31B | 31B | Q4_K_M | 22 GB | ~40 tok/s | Fits, tight |
| Qwen3.6 35B-A3B | 35B (3B active) | Q3_K_M | 19 GB | ~175 tok/s | Runs well |
| GLM-4.5-Air | 106B (12B active) | Q4_K_M | 68 GB | ~8.3 tok/s | Partial offload |
| Llama 4 Scout | 109B (17B active) | Q4_K_M | 70 GB | ~5.9 tok/s | Partial offload |
| gpt-oss-120b | 117B (5.1B active) | MXFP4 | 68 GB | ~21 tok/s | Partial offload |
| Qwen3.5 122B-A10B | 122B (10B active) | — | — | — | Won't fitWhat it needs |
| DeepSeek V4 Flash | 284B (13B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3-Flash | 320B (18B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3 | 744B (40B active) | — | — | — | Won't fitWhat it needs |
Performance
RTX 4090 vs. alternatives on Qwen3.8 27B
Generation speed in tokens per second for a single user. Token generation is limited mostly by memory bandwidth, which is why bandwidth predicts these results so well.
RTX 5090
Q6_K · 32GB
~59tok/s
RTX 4090
Q4_K_M · 24GB
~47tok/s
RTX 3090
Q4_K_M · 24GB
~39tok/s
RTX 6000 Ada
Q8_0 · 48GB
~25tok/s
Measured
RTX 4090 benchmark results
| Model | Quant | Runtime | Generation | Prompt | Source |
|---|---|---|---|---|---|
| Llama 2 7B Chat | Q4_0 | llama.cpp (Vulkan) | 190 tok/s | 9450 tok/s | llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · October 2026 |
| Llama 2 7B Chat | Q4_0 | llama.cpp (CUDA) | 185 tok/s | 11995 tok/s | llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · October 2026 |
| Llama 3.1 8B Instruct | Q4_K_M | llama.cpp (CUDA) | 130 tok/s | 7140 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| Llama 3.1 8B Instruct | F16 | llama.cpp (CUDA) | 55 tok/s | 9380 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| 4× · Llama 3.3 70B Instruct | Q4_K_M | llama.cpp (CUDA) | 20 tok/s | 835 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| 2× · Llama 3.3 70B Instruct | Q4_K_M | llama.cpp (CUDA) | 19 tok/s | 840 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| 8× · Llama 3.3 70B Instruct | F16 | llama.cpp (CUDA) | 6.5 tok/s | 1185 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
Our take
Should you buy the RTX 4090 for AI?
The RTX 4090 has 24GB of GDDR6X on a 384-bit bus for 1,008 GB/s, which made it the top consumer LLM card until the RTX 5090 arrived. It runs 32B models at 4 to 5 bit on one card, while 70B models need heavy quantization, a second card or CPU offload. NVIDIA wound down production ahead of the RTX 50 launch, so most cards today are used, and they sold for $2,500 to $3,000 in October 2026. At those prices, compare it carefully against a used RTX 3090, which offers the same 24GB for less than half the money.
What we like
- 1,008 GB/s and strong compute for prompt processing
- 24GB fits 32B models at 4 to 5 bit
- Mature CUDA support and wide software compatibility
Watch out for
- Used prices above the $1,599 launch MSRP
- 450W and a 16-pin connector that needs careful seating
- No NVLink for pooling memory across cards
Best for: Fast single-card inference for models up to 32B · LoRA fine-tuning of small models
Specifications
RTX 4090 specs that matter for AI
- Memory
- 24GB GDDR6X
- Memory bandwidth
- 1,008 GB/s
- Memory bus
- 384-bit
- Architecture
- Ada Lovelace
- Power
- 450W
- Interface
- PCIe 4.0 x16
- Released
- 2022
- Launch price
- $1,599
- Street price (used)
- ~$2,500 (2026-10)
- Software
- CUDA, Vulkan
Sources: NVIDIA GeForce RTX 4090 product page, TechPowerUp GPU Database, GeForce RTX 4090
Cross-shop
Alternatives to the RTX 4090
Frequently asked questions
How much VRAM does the RTX 4090 have?
The NVIDIA GeForce RTX 4090 has 24GB of GDDR6X with 1,008 GB/s of memory bandwidth on a 384-bit bus.
What is the largest LLM the RTX 4090 can run?
Fully in GPU memory, the largest model in our catalog that runs comfortably is Qwen3.6 35B-A3B (35B (3B active)) at Q3_K_M, using about 19GB with 8K context at ~175 tokens/sec.
Can the RTX 4090 run Llama 3.3 70B?
Only with partial CPU offload. At Q4_K_M the model needs about 47GB with 8K context, so part of it spills into system RAM and speed drops to about 1.8 tokens/sec with CPU offload. For usable 70B speeds you want 48GB of VRAM or more (for example two 24GB cards).
What is the best LLM to run on the RTX 4090?
Our top pick is Qwen3.8 27B at Q4_K_M, which fits in 24GB with room for 8K of context and runs at ~47 tokens/sec. See our full list of the best models for the RTX 4090 for coding, reasoning, and vision picks.
Is the RTX 4090 good for local AI in 2026?
Still one of the fastest 24GB cards, though used prices now exceed its launch MSRP.