Cheatsheet

Local LLM speed cheatsheet: decode tok/s by GPU & Mac

Name: LLM speed cheatsheet — every model x every rig
Creator: llm-speed
License: https://creativecommons.org/licenses/by/4.0/

Best decode tokens-per-second per (model × hardware) tuple measured by llm-speed under suite-v1. Numbers are wall-clock, batch size 1 unless the workload says otherwise. Each row links to the canonical run page; cite as llm-speed.com/r/<id>.

112 (model × hardware) combinations · sorted by decode tok/s · suite-v1

Long documents and bulk extraction

For long inputs with short answers, compare prefill, time to first token and total request time at your input length, cache state and concurrency. Check extraction accuracy too. This table selects the fastest decode row per model and hardware pair; its prefill and TTFT columns describe that row, not the best prefill speed or lowest latency.

Start with the prefill vs decode guide or our measured Gemma 4 long-context study to see how input length changes the wait before streaming.

Model	Quant	Hardware	Backend	Workload	decode tok/s	prefill tok/s	TTFT (ms)	Run
llama3.2	Q8_0	RTX 5880 (48GB) + AMD EPYC 9T24 96-Core Processor (16c) + 62GB	ollama@0.34.2	chat-short	466.9	44.3	2820.6	r_b948ehgi8g4
mlx-community/Qwen3-0.6B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	429.2	1185.2	92.8	r_6rgao6ao735
stable-code-instruct-3b	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	concurrent-decode	356.1	—	—	r_q9f15lz6831
gpt-oss-20b	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	324.2	586.8	282.9	r_dwp5fgdapmw
LFM2.5-24B-A2B	Q4_K_M	RTX 5090 (32GB) + INTEL(R) XEON(R) GOLD 6530 (64c) + 1008GB	llama.cpp@0.6.0-dev (build 1, commit a0e6dc7)	concurrent-decode	321.3	—	—	r_c8k6fleym09
gpt-oss-20b	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	318.4	—	172.5	r_b9ul-vxh9sc
DeepSeek-Coder-V2-Lite-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	309.5	—	267.6	r_0_gs1rgl2fl
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	304.9	514.5	213.8	r_vsype4pm1ko
mlx-community-Qwen2.5-0.5B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	286.5	656.2	199.6	r_akcbpx5vcqa
Qwen3-Coder-30B-A3B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	259.9	—	218.4	r_c7qyvvmmsv1
Qwen2.5-Coder-7B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	255.3	—	35.4	r_2b9o6y_49mi
Qwen2.5-7B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	246.7	—	33.9	r_3yn-4321hp-
gpt-oss-20b	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	242.6	593.8	279.5	r_t-a9v_ipdit
LFM2.5-24B-A2B	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (64c) + 1008GB	llama.cpp@0.6.0-dev (build 1, commit a0e6dc7)	concurrent-decode	236.8	—	—	r_bqmpv9kncyd
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	232.5	831.9	132.2	r_1k8942tw3zf
Llama-3.1-8B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	232.2	—	33.8	r_kfrkg-vn376
Qwen3.6-35B-A3B-Q4_K_M.gguf	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	224.0	—	169.5	r_g1aw3mizd32
gpt-oss-20b	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	207.2	658.3	252.1	r_nakw1nq6clw
Qwen3-Coder-30B-A3B-Instruct	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	199.7	612.5	179.6	r_0dh1kzbv4h0
Yi-Coder-9B-Chat	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	199.3	—	36.2	r_u4iojm6-ekg
gemma3	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB	ollama@0.31.1	chat-short	195.0	167.2	705.6	r_dlanfbgym0h
mlx-community/stable-code-instruct-3b-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	192.5	560.7	226.5	r_y2_5y8oo97d
deepseek-coder-v2	Q4_0	RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB	ollama@0.31.1	chat-short	189.5	159.2	734.9	r_o2-1w665rtq
mlx-community/Qwen3-4B-Instruct-2507-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	182.7	636.6	172.8	r_p2rpb_0iyjj
qwen3-coder	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB	ollama@0.31.1	concurrent-decode	179.9	—	—	r_wjq32z47vlp
mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	168.3	291.5	449.5	r_l_v1-zq_qaz
glm-4.7-flash	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB	ollama@0.31.1	chat-short	164.9	3.7	28105.7	r_ted0u8uyre0
Qwen3.6-35B-A3B	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@8931 (9725a313b)	concurrent-decode	163.5	—	—	r_m6j1zxwvp-f
qwen2.5-coder	Q4_K_M	RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB	ollama@0.31.1	concurrent-decode	161.1	—	—	r_mv8n8k9wu1e
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx32k	-	RTX 5090 (32GB) + AMD EPYC 9534 64-Core Processor (128c) + 756GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	157.7	—	—	r_ymx58emedxd
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 5090 (32GB) + AMD EPYC 9534 64-Core Processor (128c) + 756GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	156.0	—	—	r_z-l1yelv_67
llama3.1	Q4_K_M	RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB	ollama@0.31.1	chat-short	154.4	328.0	335.4	r_h1ub_1uxzdh
gemma-2-9b-it	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	152.7	—	58.9	r__b_bzmmab_8
mlx-community/gpt-oss-20b-MXFP4-Q4	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	152.7	239.9	692.1	r_3ijun8ltjnb
LFM2.5-24B-A2B	Q4_K_M	RTX A4000 (16GB) + AMD EPYC 7402P 24-Core Processor (24c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit b76b133)	concurrent-decode	142.7	—	—	r_j5tgvia-kjk
gemma-4-12b-it-qat	Q4_0	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp@1 (9725a31)	long-context-decay	142.6	7115.8	3354.8	r_-lh-vczf7ik
gpt-oss	MXFP4	RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB	ollama@0.31.1	chat-long	141.7	631.5	5048.2	r_iu2sfa9ykvw
phi-4	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	140.9	—	56.7	r_e-k4aea8ipr
Qwen2.5-Coder-14B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	140.0	—	42.1	r_opuj21f13-_
mlx-community-Qwen2.5-7B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	139.6	190.0	689.4	r_5r6rhiynenc
qwen2.5-coder	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB	ollama@0.31.1	concurrent-decode	139.2	—	—	r_pb0jpnbujji
mlx-community/Qwen2.5-Coder-7B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	138.6	539.6	242.8	r_uoehjq0nvc0
llama3.1	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	ollama@0.31.1	chat-short	137.3	269.1	408.8	r_2np7abzjobe
qwen2.5-coder	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	ollama@0.31.1	concurrent-decode	137.3	—	—	r_nrpc8-onxu2
Qwen2.5-Coder-14B-Instruct	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	137.3	2016.9	65.0	r_xnr_k4emoyr
llama3.1	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB	ollama@0.31.1	chat-short	136.2	2.2	49960.0	r_9hurqggbshk
glm-4.7-flash	Q4_K_M	RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB	ollama@0.31.1	chat-short	134.6	4.3	24308.6	r_kalwcyab-86
deepseek-r1	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB	ollama@0.31.1	agent-trace	133.8	8852.4	366.2	r_fg77v2hhohb
Qwen2.5-14B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	133.3	—	55.9	r_tj9bu7gvnvh
mlx-community/Llama-3.1-8B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	130.2	400.3	339.7	r_v2pbc0rq2l4
mlx-community/Qwen3-8B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	120.5	446.1	246.6	r_b1awwhkoo2v
lmstudio-community/DeepSeek-R1-0528-Qwen3-8B-MLX-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	116.8	424.5	247.4	r_yy_6jfx70jq
qwen3-coder-bench-32k	Q4_K_M	M4 Max (40-core GPU) + 128GB unified	ollama@0.30.11	chat-short	113.3	84.1	1307.7	r_roktphpc--8
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.5	308.7	356.3	r_o6wbmfegmde
mlx-community/granite-8b-code-instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.3	525.7	220.7	r_bue3bee0gw7
mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	112.2	204.0	539.3	r_fpsca03u2o_
qwen3-coder	Q4_K_M	M4 Max (40-core GPU) + 128GB unified	ollama@0.30.11	concurrent-decode	109.9	—	—	r_r0di2hkku1h
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	chat-short	107.3	255.8	469.1	r_vkjsinuhlyl
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k	-	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	chat-short	107.0	253.3	473.7	r_iu0s18cs9d8
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx16k	-	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	105.8	—	—	r_qibhq5bkmgq
mlx-community/Yi-Coder-9B-Chat-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	103.5	390.9	307.0	r_3hvui9a1yuc
llama3.2	Q8_0	M1 Pro (16-core GPU) + 16GB unified	ollama@0.32.14	chat-short	103.1	8.8	14246.7	r_w-p7v--eync
Codestral-22B-v0.1	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	concurrent-decode	100.3	—	—	r_4q040m4scic
gemma-4-12b-it-qat	Q4_0	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@8931 (9725a313b)	chat-short	97.7	1109.0	111.8	r_jfh0qkngw76
Qwen2.5-Coder-14B-Instruct	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	89.9	2292.6	57.1	r_eqnsqmgz29a
mlx-community/gemma-2-9b-it-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	89.5	170.7	691.4	r_iz137eqvuzy
qwen3.6-agent-q6k-ctx128k	Q6_K	M5 Max (40-core GPU) + 48GB unified	ollama@0.32.1	chat-short	88.3	16.2	6915.3	r_udwgba7udqu
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB	-	RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	87.8	—	—	r_-sifd5otbpy
Qwen3.8-27B-mtp-on	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	concurrent-decode	87.0	—	—	r_7j8rpk9mwhg
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k	-	RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (64c) + 252GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	80.4	—	—	r_w3lond-s75t
lmstudio-community-Qwen3-Next-80B-A3B-Instruct-MLX-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	80.3	24.5	4492.5	r_1pl79r50ofy
gemma3	Q4_K_M	RTX 3090 (24GB) + AMD Ryzen Threadripper PRO 5975WX 32-Cores (32c) + 504GB	ollama@0.31.1	chat-short	78.7	26.0	4534.1	r_3gnxkzhhmyj
Qwen2.5-Coder-14B-Instruct	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	78.2	1693.2	77.4	r_rkyf9wxzw9z
Qwen3.8-27B	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 9534 64-Core Processor (128c) + 756GB	llama.cpp@0.7.0-dev (build 11669, commit 620d2225c)	concurrent-decode	75.4	—	—	r_qfrb2zmyk0k
Qwen3.6-27B	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	concurrent-decode	75.1	—	—	r_pob-co51bd-
mlx-community/phi-4-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	74.4	255.0	443.2	r_sqzp-0rdez-
Qwen3.6-27B-Q4_K_M.gguf	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	74.0	—	190.4	r_f7ulllqu4vk
Qwen2.5-Coder-32B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	71.9	—	71.0	r_nkbs6d3-d21
Qwen2.5-32B-Instruct	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	71.9	—	115.2	r_twfs86tf_xf
mlx-community-Qwen2.5-14B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	70.9	301.7	434.1	r_v4bq1sviz4o
mlx-community/Qwen2.5-Coder-14B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	70.5	306.1	427.9	r_l36cijqxq4t
Qwen3-32B	Q4_K_M	RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB	llama.cpp@0.7.0-dev (build 1, commit 1268eda)	chat-short	69.8	1047.0	105.1	r_vhbqyt1f_xr
gemma-4-31B-it-Q4_K_M.gguf	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	69.7	—	174.8	r_pa7nuerd9tl
Qwen3-32B	-	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp	chat-short	69.5	—	327.1	r_phvxm9dcak0
qwen2.5-coder	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB	ollama@0.31.1	chat-short	69.2	402.0	325.8	r_6ahy-dq0f_0
Qwen3.8-27B	Q4_K_M	RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB	llama.cpp@1 (9725a31)	chat-short	65.7	973.6	117.1	r_c3ps9wjygi9
mlx-community/starcoder2-15b-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	64.2	220.5	489.8	r_wsxml_39dh_
mlx-community/Codestral-22B-v0.1-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	47.5	205.7	559.1	r_79dvtag5fd_
Qwen3.6-27B	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB	llama.cpp@0.6.0-dev (build 1, commit 3fb4767)	concurrent-decode	47.2	—	—	r_d_wda_rw62f
qwen3.6	Q4_K_M	RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB	ollama@0.31.1	agent-trace	44.2	1910.5	1807.9	r_h_659oy695r
Qwen3-32B	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	43.5	778.6	141.3	r_ayy9rjgha5r
Qwen3.8-27B	Q4_K_M	RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB	llama.cpp@0.6.0-dev (build 1, commit df18bee)	chat-short	43.1	428.6	359.3	r_11kn1epxewh
Qwen3.6-27B	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	42.7	236.9	472.7	r_qm1xyvjmx_r
Qwen3-32B	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB	llama.cpp@0.7.0-dev (build 1, commit ea0c4c2)	chat-short	38.5	589.5	186.6	r_km62sybxga6
Qwen3.8-27B	Q4_K_M	RTX 3090 (24GB) + AMD EPYC 7C13 64-Core Processor (128c) + 1008GB	llama.cpp@0.6.0-dev (build 1, commit 8c2134a)	chat-short	37.6	368.5	417.9	r_ptct8o01o5l
mlx-community-Qwen2.5-32B-Instruct-4bit	4bit	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.6	141.9	922.9	r_njgxtgyym1e
mlx-community/Qwen2.5-Coder-32B-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.5	144.1	909.4	r_721b4bls_oq
mlx-community/Qwen3-32B-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	34.4	95.1	1156.4	r_anmmc80-aoq
mlx-community-Qwen2.5-7B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	30.5	161.9	809.1	r_llzv_g-ymaf
mlx-community/Llama-3.1-8B-Instruct-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	29.2	203.3	669.0	r_h0-use1ypnb
Muse-Spark-Open-816B-A42B-UD-00001-of-00005	IQ1_S	3x A100 (80GB) + AMD EPYC 7742 64-Core Processor (128c) + 1008GB	llama.cpp@0.7.0-dev (build 11675, commit 2851b3bcb)	concurrent-decode	26.3	—	—	r_fsd9qjmmna0
llama3.2	Q8_0	AMD64 Family 23 Model 104 Stepping 1, AuthenticAMD (6c) + 15GB	ollama@0.34.0	chat-short	23.9	39.4	3171.6	r_b1rn5gi6b3b
mlx-community/gemma-2-9b-it-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	22.0	102.5	1151.3	r_q7t7b7dcuz5
mlx-community/stable-code-instruct-3b-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	19.4	131.3	967.0	r_pqjsvd-cub4
Llama-3.3-70B-Instruct	Q4_K_M	2x RTX 3090 (24GB) + AMD EPYC 7H12 64-Core Processor (128c) + 1008GB	llama.cpp@0.7.0-dev (build 1, commit f74e2fa)	chat-short	19.1	415.4	325.0	r_n93ukk0mkmm
mlx-community/llama-3.3-70b-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	16.8	25.1	5419.9	r_sx3a4y9n-m4
mlx-community/qwen2.5-72b-Instruct-4bit	-	M3 Ultra (60-core GPU) + 96GB unified	mlx@0.31.3	chat-short	16.3	23.2	5635.3	r_5c80gthqlh6
mlx-community/phi-4-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	15.6	120.7	936.0	r_-w8hnn61va_
mlx-community-Qwen2.5-14B-Instruct-4bit	4bit	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	14.4	117.9	1111.5	r_ia73dzeue0b
Llama-3.3-70B-Instruct	Q4_K_M	A40 (45GB) + Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz (48c) + 504GB	llama.cpp@0.7.0-dev (build 1, commit ceefd16)	chat-short	13.0	401.0	336.7	r_19f64yux97o
mlx-community/Qwen3-32B-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	7.2	25.7	4287.9	r_pnrrpcdqfo4
mlx-community/Qwen2.5-32B-Instruct-4bit	-	M3 Pro (18-core GPU) + 36GB unified	mlx@0.31.3	chat-short	7.1	33.4	3921.1	r_f4x3xaan2ay

How to read this table

Each row is the highest decode tokens-per-second measured for a unique (model, hardware) pair. When more than one workload was run, we pick the workload that produced the headline number. Lower-tps duplicates from the same machine are not shown; click through to the run page for the full per-workload breakdown.

decode tok/s is wall-clock streaming throughput (memory-bandwidth-bound). prefill tok/s is prompt ingestion throughput (compute-bound). TTFT is time-to-first-token in milliseconds. See /glossary for definitions and /methodology for the workload spec.