NVIDIA · Volta · 2018
NVIDIA Tesla V100 32GB for local AI
900 GB/s HBM2 and tensor cores for mid-range money, with adapter and cooling caveats.
Updated · Speeds marked ~ are estimates; seemethodology
- VRAM
- 32GB
- Bandwidth
- 900 GB/s
- Power
- 300W
- Runs well
- 8 models
V100 32GB
Typically about $700 used
Can the V100 32GB run LLMs?
The V100 32GB has 32GB of HBM2 and 900 GB/s of memory bandwidth, which decides how big a model fits and how fast it generates text. It comfortably runs models up to about 35B parameters: the largest in our catalog is Qwen3.6 35B-A3B at Q5_K_M (28GB with 8K context, ~105 tokens/sec). Our top all-round pick for it is Qwen3.8 27B (Q6_K, ~25 tokens/sec). Llama 3.3 70B Instruct only runs with partial CPU offload, at about 2.5 tokens/sec with CPU offload.
Recommendations
Best models for the V100 32GB
Best overall · also coding, reasoning, vision
Qwen3.8 27BQ6_K · 25GB · ~25 tokens/sec
Runs wellRunner-up · also fastest, largest that fits
Qwen3.6 35B-A3BQ5_K_M · 28GB · ~105 tokens/sec
Runs well
Compatibility
What runs on the V100 32GB
Every current model in our catalog, at the best quantization that fits in 32GB with 8K tokens of context. "Partial offload" means part of the model runs from system RAM (we assume 64GB), which is much slower.
| Model | Parameters | Best quant | Memory | Speed | Verdict |
|---|---|---|---|---|---|
| Gemma 4 E4B | 8B | Q8_0 | 10 GB | ~58 tok/s | Runs well |
| Qwen3.5 9B | 9B | Q8_0 | 12 GB | ~51 tok/s | Runs well |
| Phi-4 | 14.7B | Q8_0 | 18 GB | ~36 tok/s | Runs well |
| gpt-oss-20b | 20.9B (3.6B active) | MXFP4 | 15 GB | ~100 tok/s | Runs well |
| Gemma 4 26B A4B | 26B (3.8B active) | Q6_K | 23 GB | ~84 tok/s | Runs well |
| Qwen3.8 27B | 27B | Q6_K | 25 GB | ~25 tok/s | Runs well |
| Gemma 4 31B | 31B | Q5_K_M | 26 GB | ~25 tok/s | Runs well |
| Qwen3.6 35B-A3B | 35B (3B active) | Q5_K_M | 28 GB | ~105 tok/s | Runs well |
| GLM-4.5-Air | 106B (12B active) | Q4_K_M | 68 GB | ~9.6 tok/s | Partial offload |
| Llama 4 Scout | 109B (17B active) | Q4_K_M | 70 GB | ~6.8 tok/s | Partial offload |
| gpt-oss-120b | 117B (5.1B active) | MXFP4 | 68 GB | ~24 tok/s | Partial offload |
| Qwen3.5 122B-A10B | 122B (10B active) | Q4_K_M | 77 GB | ~11 tok/s | Partial offload |
| DeepSeek V4 Flash | 284B (13B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3-Flash | 320B (18B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3 | 744B (40B active) | — | — | — | Won't fitWhat it needs |
Performance
V100 32GB vs. alternatives on Qwen3.8 27B
Generation speed in tokens per second for a single user. Token generation is limited mostly by memory bandwidth, which is why bandwidth predicts these results so well.
RTX 3090
Q4_K_M · 24GB
~39tok/s
V100 32GB
Q6_K · 32GB
~25tok/s
MI50 32GB
Q6_K · 32GB
~19tok/s
Tesla P40
Q4_K_M · 24GB
~11tok/s
Measured
V100 32GB benchmark results
| Model | Quant | Runtime | Generation | Prompt | Source |
|---|---|---|---|---|---|
| Llama 2 7B Chat | Q4_0 | llama.cpp (Vulkan) | 130 tok/s | 1390 tok/s | llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · October 2026 |
| Llama 2 7B Chat | Q4_0 | llama.cpp (CUDA) | 130 tok/s | 3045 tok/s | llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · October 2026 |
Our take
Should you buy the V100 32GB for AI?
The Tesla V100 32GB pairs Volta tensor cores with 32GB of HBM2 on a 4096-bit bus for 900 GB/s. That bandwidth beats many current consumer cards, so it generates tokens quickly on models that fit in 32GB. Many used units are SXM2 modules that need an adapter board, heatsink and fan to run in a desktop, while the rarer PCIe version is passively cooled. CUDA 13 dropped Volta, so stick to CUDA 12 builds. Used units sold for about $700 in October 2026.
What we like
- 900 GB/s HBM2 bandwidth
- 32GB per card
- Tensor cores give faster prompt processing than Pascal
Watch out for
- SXM2 modules need adapters and custom cooling
- Not supported by CUDA 13
- No display outputs
Best for: Tinkerers building cheap high-bandwidth rigs · Running 32B models at 4 to 5 bit
Specifications
V100 32GB specs that matter for AI
- Memory
- 32GB HBM2
- Memory bandwidth
- 900 GB/s
- Memory bus
- 4096-bit
- Architecture
- Volta
- Power
- 300W
- Interface
- SXM2 or PCIe 3.0 x16
- Released
- 2018
- Street price (used)
- ~$700 (2026-10)
- Software
- CUDA, Vulkan
SXM2 modules are rated at 300W and the PCIe card at 250W.
Sources: NVIDIA Tesla V100 product page
Builds
PC builds with the V100 32GB
Cross-shop
Alternatives to the V100 32GB
Frequently asked questions
How much VRAM does the V100 32GB have?
The NVIDIA Tesla V100 32GB has 32GB of HBM2 with 900 GB/s of memory bandwidth on a 4096-bit bus.
What is the largest LLM the V100 32GB can run?
Fully in GPU memory, the largest model in our catalog that runs comfortably is Qwen3.6 35B-A3B (35B (3B active)) at Q5_K_M, using about 28GB with 8K context at ~105 tokens/sec.
Can the V100 32GB run Llama 3.3 70B?
Only with partial CPU offload. At Q4_K_M the model needs about 47GB with 8K context, so part of it spills into system RAM and speed drops to about 2.5 tokens/sec with CPU offload. For usable 70B speeds you want 48GB of VRAM or more (for example two 24GB cards).
What is the best LLM to run on the V100 32GB?
Our top pick is Qwen3.8 27B at Q6_K, which fits in 32GB with room for 8K of context and runs at ~25 tokens/sec. See our full list of the best models for the V100 32GB for coding, reasoning, and vision picks.
Is the V100 32GB good for local AI in 2026?
900 GB/s HBM2 and tensor cores for mid-range money, with adapter and cooling caveats.