Benchmarks

How fast does it run?

Token generation speed for popular open models on every GPU and Mac we track. Measured where we have data, clearly marked estimates where we don't.

How to read these numbers

Tokens per second for one user generating text, which is what you feel in chat. Each GPU runs the best quantization that fits with 8K context. Solid bars are measured results with a linked source; hatched bars are estimates from memory bandwidth, calibrated against those measurements.

Explorer

Speed by model

Measured onlyInclude Macs and unified memory

Qwen3 32B 32.8B

Generation speed, tokens/sec, single user

Measured (click the number for the source) Estimated from memory bandwidth

Measured results

108 measured runs

Every result links to where it was measured. Click a column to sort.

GPU	Model	Quant	Runtime	Generation	Prompt	Source
RTX 5090	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	290	14075	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX PRO 6000	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	275	14855	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 5090	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	265	10380	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 4090	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	190	9450	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 4090	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	185	11995	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 5080	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	185	7445	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 7900 XTX	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	185	3725	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 5080	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	180	8295	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
M5 Ultra 256GB	Llama 2 7B Chat	Q4_0	llama.cpp · Metal	180	4945	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
RTX 5070 Ti	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	175	6950	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 6000 Ada	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	175	9230	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 3090	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	170	5155	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RX 7900 XTX	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	165	3550	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RTX 3090	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	165	4665	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 3090	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	160	5175	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RX 7900 XTX	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	155	4145	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
4× RTX 3090	Qwen3.6 35B-A3B	Q5_K_M	llama.cpp · CUDA	150	—	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 4080 Super	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	150	8125	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 4080 Super	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	145	7100	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
R9700	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	145	5610	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX A6000	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	140	4915	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RX 9070 XT	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	135	5035	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 5070 Ti	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	135	6215	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 4070 Ti Super	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	130	6925	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
V100 32GB	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	130	1390	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 4070 Ti Super	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	130	6100	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
V100 32GB	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	130	3045	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
RTX 5070	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	130	5185	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
R9700	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	125	5035	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
M5 Max 128GB	Llama 2 7B Chat	Q4_0	llama.cpp · Metal	120	3220	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
M5 Ultra 256GB	Llama 2 7B Chat	Q8_0	llama.cpp · Metal	120	4910	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
R9700	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	115	4665	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
Arc Pro B70	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	110	3380	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
MI50 32GB	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	110	1120	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 9060 XT 16GB	gpt-oss-20b	MXFP4	llama.cpp · Vulkan	105	3465	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 9070 XT	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	100	5055	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
MI50 32GB	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	99	1055	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RTX 5060	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	97	3270	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
R9700	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	94	4445	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RTX 5060 Ti 16GB	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	94	3460	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 9060 XT 16GB	gpt-oss-20b	MXFP4	llama.cpp · ROCm (HIP)	93	3370	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RTX 4070	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	92	3180	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
M3 Ultra 512GB	Llama 2 7B Chat	Q4_0	llama.cpp · Metal	92	1470	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
RTX 5060 Ti 16GB	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	91	3735	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
M4 Max 128GB	Llama 2 7B Chat	Q4_0	llama.cpp · Metal	83	885	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
RTX 3060 12GB	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	76	1815	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RTX 3060 12GB	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	76	2140	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
M5 Ultra 256GB	Llama 2 7B Chat	F16	llama.cpp · Metal	74	5205	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
M5 Max 128GB	Llama 2 7B Chat	Q8_0	llama.cpp · Metal	72	3145	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
RX 9060 XT 16GB	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	71	2845	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
Arc B580	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	70	620	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
Arc Pro B60	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	69	520	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 9060 XT 16GB	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	68	1420	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RX 9060 XT 16GB	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	67	2640	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
M3 Ultra 512GB	Llama 2 7B Chat	Q8_0	llama.cpp · Metal	64	1490	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
RTX 4000 Ada	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	62	2780	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
Tesla P40	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	59	490	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
DGX Spark	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	59	2790	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
DGX Spark	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	57	3060	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
Tesla P40	Llama 2 7B Chat	Q4_0	llama.cpp · CUDA	55	1005	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
M4 Max 128GB	Llama 2 7B Chat	Q8_0	llama.cpp · Metal	54	890	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
Ryzen AI Max+ 395	Llama 2 7B Chat	Q4_0	llama.cpp · Vulkan	52	1485	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
Ryzen AI Max+ 395	Llama 2 7B Chat	Q4_0	llama.cpp · ROCm (HIP)	50	910	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
M3 Ultra 512GB	Llama 2 7B Chat	F16	llama.cpp · Metal	40	1540	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
4× RTX 3090	Qwen3.8 27B	Q5_K_M	llama.cpp · CUDA	39	—	llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · Oct 14, 2026
M5 Max 128GB	Llama 2 7B Chat	F16	llama.cpp · Metal	37	3160	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
2× RX 7900 XTX	Gemma 4 31B	Q8_0	llama.cpp · ROCm (HIP)	32	1115	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
M4 Max 128GB	Llama 2 7B Chat	F16	llama.cpp · Metal	32	925	llama.cpp discussion #4167: Performance of llama.cpp on Apple Silicon M-series · Oct 14, 2026
2× RX 7900 XTX	Gemma 4 31B	Q8_0	llama.cpp · Vulkan	24	850	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
2× RX 7900 XTX	Gemma 4 31B	Q8_0	llama.cpp · ROCm (HIP)	23	920	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RX 9060 XT 16GB	Qwen3.8 27B	IQ4_XS	llama.cpp · Vulkan	20	710	llama.cpp discussion #10879: Performance of llama.cpp with Vulkan · Oct 14, 2026
RX 9060 XT 16GB	Qwen3.8 27B	IQ4_XS	llama.cpp · ROCm (HIP)	20	740	llama.cpp discussion #15021: Performance of llama.cpp on AMD ROCm (HIP) · Oct 14, 2026
RTX PRO 5000 72GB	gpt-oss-20b	MXFP4	llama.cpp · CUDA	215	10340	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
RTX PRO 5000 72GB	Qwen3 30B-A3B (2507)	Q4_K_XL	llama.cpp · CUDA	190	5265	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
RTX PRO 5000 72GB	Qwen3 8B	Q4_K_XL	llama.cpp · CUDA	160	8175	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
RTX PRO 5000 72GB	Qwen3 14B	Q4_K_XL	llama.cpp · CUDA	99	4515	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
RTX 4060 Ti 16GB	gpt-oss-20b	MXFP4	llama.cpp · CUDA	63	3275	Hardware Corner: RTX 4060 Ti 16GB LLM Performance · Mar 1, 2026
RTX PRO 5000 72GB	Qwen3 32B	Q4_K_XL	llama.cpp · CUDA	47	2030	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
RTX 4060 Ti 16GB	Qwen3 8B	Q4_K_XL	llama.cpp · CUDA	46	2675	Hardware Corner: RTX 4060 Ti 16GB LLM Performance · Mar 1, 2026
RTX 4060 Ti 16GB	Qwen3 14B	Q4_K_XL	llama.cpp · CUDA	27	1645	Hardware Corner: RTX 4060 Ti 16GB LLM Performance · Mar 1, 2026
RTX PRO 5000 72GB	Llama 3.3 70B Instruct	Q4_K_XL	llama.cpp · CUDA	24	1025	Hardware Corner: RTX Pro 5000 Blackwell LLM Performance · Mar 1, 2026
DGX Spark	gpt-oss-20b	MXFP4	llama.cpp · CUDA	83	4505	llama.cpp repository: benches/dgx-spark/dgx-spark.md · Feb 5, 2026
DGX Spark	Qwen3 Coder 30B-A3B Instruct	Q8_0	llama.cpp · CUDA	61	2985	llama.cpp repository: benches/dgx-spark/dgx-spark.md · Feb 5, 2026
DGX Spark	gpt-oss-120b	MXFP4	llama.cpp · CUDA	59	2445	llama.cpp repository: benches/dgx-spark/dgx-spark.md · Feb 5, 2026
M4 Max 128GB	gpt-oss-20b	MXFP4	llama.cpp	120	1850	llama.cpp discussion #16578: Performance of llama.cpp on NVIDIA DGX Spark · Nov 16, 2025
DGX Spark	gpt-oss-20b	MXFP4	llama.cpp	86	3800	llama.cpp discussion #16578: Performance of llama.cpp on NVIDIA DGX Spark · Nov 16, 2025
Ryzen AI Max+ 395	gpt-oss-120b	MXFP4	llama.cpp	47	1000	llama.cpp discussion #16578: Performance of llama.cpp on NVIDIA DGX Spark · Nov 16, 2025
RTX 6000 Ada	Llama 3.1 8B Instruct	Q4_K_M	llama.cpp · CUDA	135	5790	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 4090	Llama 3.1 8B Instruct	Q4_K_M	llama.cpp · CUDA	130	7140	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 3090	Llama 3.1 8B Instruct	Q4_K_M	llama.cpp · CUDA	115	4030	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX A6000	Llama 3.1 8B Instruct	Q4_K_M	llama.cpp · CUDA	105	3780	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 4000 Ada	Llama 3.1 8B Instruct	Q4_K_M	llama.cpp · CUDA	59	2450	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 4090	Llama 3.1 8B Instruct	F16	llama.cpp · CUDA	55	9380	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 6000 Ada	Llama 3.1 8B Instruct	F16	llama.cpp · CUDA	52	6665	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 3090	Llama 3.1 8B Instruct	F16	llama.cpp · CUDA	47	4445	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX A6000	Llama 3.1 8B Instruct	F16	llama.cpp · CUDA	41	4510	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 4000 Ada	Llama 3.1 8B Instruct	F16	llama.cpp · CUDA	21	3120	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
4× RTX 4090	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	20	835	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
2× RTX 4090	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	19	840	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX 6000 Ada	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	19	565	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
4× RTX 6000 Ada	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	18	655	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
4× RTX 3090	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	17	300	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
2× RTX 3090	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	17	355	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
RTX A6000	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	15	480	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
4× RTX A6000	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	14	495	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
4× RTX 4000 Ada	Llama 3.3 70B Instruct	Q4_K_M	llama.cpp · CUDA	7.4	280	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
8× RTX 4090	Llama 3.3 70B Instruct	F16	llama.cpp · CUDA	6.5	1185	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024
6× RTX 3090	Llama 3.3 70B Instruct	F16	llama.cpp · CUDA	5.8	580	XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 12, 2024

Methodology

How we measure and estimate

Which runtimes and settings count, how we pick sources, and how the estimate model works, including its known blind spots.

Read the methodology

Frequently asked questions

What is a good tokens per second for a local LLM?

People read at roughly 5–10 tokens per second, so anything above about 15 tok/s feels fluid for chat. Coding agents and reasoning models generate far more tokens per answer, so 30+ tok/s makes a real difference there.

Why does memory bandwidth decide LLM speed?

To generate each token, the GPU has to read every active weight of the model from memory once. A 20GB model on a GPU with 1,000 GB/s of bandwidth can, at best, produce about 50 tokens per second. Compute matters more for prompt processing, which is why we report both when we have them.

What is the fastest consumer GPU for running LLMs?

The RTX 5090, with 1,792 GB/s of memory bandwidth. It generates roughly 145 tokens per second on Llama 3.1 8B Instruct at Q8_0 in our estimates.

Are your benchmark numbers measured or estimated?

Both, and we label them. Solid bars and plain numbers are measured results from the linked source. Hatched bars and numbers marked ~ are estimates from our bandwidth model, which we calibrate against the measured data.