Qwen3-32B-4bit
19 workload results across 6 hardware configurations.
Fastest local config
69.8 decode tok/s
on RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB via llama.cpp (Q4_K_M). see full run
For a standard Q4_K_M quantization using llama.cpp, here are measured results for short-chat generation.
On llama.cpp Q4_K_M, Qwen3-32B is fastest on the RTX 5090 at 69.83 tok/s, 61% faster than the RTX 4090 (43.47 tok/s); slowest: RTX 3090 at 38.45 tok/s.
| Card | Decode tok/s | TTFT (110-token prompt) | Run |
|---|---|---|---|
| RTX 5090 | 69.83 (single run) | 105 ms | r_vhbqyt1f_xr |
| RTX 4090 | 43.47 (single run) | 141 ms | r_ayy9rjgha5r |
| RTX 3090 | 38.45 (single run) | 187 ms | r_km62sybxga6 |
Qwen3-32B, Q4_K_M, llama.cpp, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded. Runs use different llama.cpp versions (RTX 5090: 0.7.0-dev (build 1, commit 1268eda); RTX 4090: 0.6.0-dev (build 1, commit df18bee); RTX 3090: 0.7.0-dev (build 1, commit ea0c4c2)), so treat this as measured reference points, not a controlled ranking.
Local runs (19 workload results)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 5090 (32GB) + AMD EPYC 7543 32-Core Processor (120c) + 945GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp@0.7.0-dev (build 1, commit 1268eda) | Q4_K_M | 69.83tok/s | 1,047.0tok/s | 105ms | r_vhbqyt1f_xr |
| chat-long | llama.cpp@0.7.0-dev (build 1, commit 1268eda) | Q4_K_M | 64.13tok/s | 3,139.6tok/s | 1,002ms | r_vhbqyt1f_xr |
| concurrent-decode | llama.cpp@0.7.0-dev (build 1, commit 1268eda) | Q4_K_M | 68.52tok/s | no data | no data | r_vhbqyt1f_xr |
| agent-trace | llama.cpp@0.7.0-dev (build 1, commit 1268eda) | Q4_K_M | 66.46tok/s | 8,161.5tok/s | 253ms | r_vhbqyt1f_xr |
RTX 3090 (24GB) + AMD EPYC 7282 16-Core Processor (16c) + 126GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 38.45tok/s | 589.5tok/s | 187ms | r_km62sybxga6 |
| chat-long | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 36.94tok/s | 1,218.2tok/s | 2,583ms | r_km62sybxga6 |
| concurrent-decode | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 38.05tok/s | no data | no data | r_km62sybxga6 |
| agent-trace | llama.cpp@0.7.0-dev (build 1, commit ea0c4c2) | Q4_K_M | 37.57tok/s | 3,287.7tok/s | 624ms | r_km62sybxga6 |
RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 43.47tok/s | 778.6tok/s | 141ms | r_ayy9rjgha5r |
| chat-long | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 40.66tok/s | 2,333.3tok/s | 1,349ms | r_ayy9rjgha5r |
| concurrent-decode | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 42.69tok/s | no data | no data | r_ayy9rjgha5r |
| agent-trace | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 41.94tok/s | 5,982.3tok/s | 342ms | r_ayy9rjgha5r |
RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 66.51tok/s | no data | 158ms | r_-txe_hiq44n |
| chat-long | llama.cpp | - | 62.30tok/s | no data | 1,067ms | r_-txe_hiq44n |
| concurrent-decode | llama.cpp | - | 66.64tok/s | no data | no data | r_-txe_hiq44n |
| agent-trace | llama.cpp | - | 64.34tok/s | 8,394.1tok/s | 259ms | r_-txe_hiq44n |
| chat-short | llama.cpp | - | 69.45tok/s | no data | 327ms | r_phvxm9dcak0 |
M3 Pro (18-core GPU) + 36GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | mlx@0.31.3 | - | 7.16tok/s | 25.65tok/s | 4,288ms | r_pnrrpcdqfo4 |
M3 Ultra (60-core GPU) + 96GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | mlx@0.31.3 | - | 34.41tok/s | 95.12tok/s | 1,156ms | r_anmmc80-aoq |
Community folklore
24 unverified claims extracted from Reddit/HN comments. Lower trust than signed runs above; every row links to the source.
- communityconfidence 75%
30.00tok/s — qwen3 32B on RTX 3090 via vllm gptq
our signed data: RTX 3090 · qwen3 32B
“Thank you for sharing! The difference for PP between 3090 vs M3MAX is remarkable. I thought 3090 would reach 30 t/s for TG, especially with tensor parallelism. Oh actually, can you please share vLLM results? You can just share one data point if you don't”
- communityconfidence 75%
100.0tok/s — qwen 3 32b on RTX 3090 via vllm awq
our signed data: RTX 3090 · qwen 3 32b
“I have 2x3090 to run qwen 3 32b awq (4bit). Using vllm it can run ~80tok /s , using lmdeploy it much faster, maybe ~100tok/s. I like fast speed because iam using it as code agent. Fast speed definately help, it allow me to decide use the code or ask better code u”
- communityconfidence 75%
80.00tok/s — qwen 3 32b on RTX 3090 via vllm awq
our signed data: RTX 3090 · qwen 3 32b
“I have 2x3090 to run qwen 3 32b awq (4bit). Using vllm it can run ~80tok /s , using lmdeploy it much faster, maybe ~100tok/s. I like fast speed because iam using it as code agent. Fast speed definately help, it all”
- communityconfidence 65%
40.00tok/s — Qwen3 32b via llama.cpp gptq
our signed data: Qwen3 32b
“Note that you will get 40t/s for Qwen3 32b gptq 4bit with 4x tensor parallelism. Qwen3 235B Q4_1 will work with llama.cpp and 5xMI50 at 19t/s initially. But expect that”
- communityconfidence 60%
180.0tok/s — Qwen3-32B on M4 via lm-studio
our signed data: M4 · Qwen3-32B
“its 79.51% while sustaining ~64 tok/s - arguably today's best speed/quality trade-off for Mac setups. 5️⃣ The 0.6B micro-model races above 180 tok/s but tops out at 37.56% - that's why it's not even on the graph (50 % performance cut-off). All local runs were done with @lmstudi…”
- communityconfidence 60%
45.00tok/s — Qwen3-32B on Fireworks AI via hosted-api
our signed data: Fireworks AI · Qwen3-32B
“(via Fireworks API) tops the table at 83.66% with ~55 tok/s. 2️⃣ But the 30B-A3B Unsloth quant delivered 82.20% while running locally at ~45 tok/s and with zero API spend. 3️⃣ The same Unsloth build is ~5x faster than Qwen's Qwen3-32B, which scores 82.20% as well yet crawls at …”
- communityconfidence 60%
180.0tok/s — Qwen3-32B on M4 via lm-studio
our signed data: M4 · Qwen3-32B
“9.51%** while sustaining \~64 tok/s - arguably today's best speed/quality trade-off for Mac setups. 5. The **0.6B** micro-model races above 180 tok/s but tops out at **37.56%** \- that's why it's not even on the graph (50 % performance cut-off). All local runs were done with LM …”
- communityconfidence 60%
45.00tok/s — Qwen3-32B on Fireworks AI via hosted-api
our signed data: Fireworks AI · Qwen3-32B
“rks API) tops the table at **83.66%** with \~55 tok/s. 2. But the **30B-A3B Unsloth** quant delivered **82.20%** while running locally at \~45 tok/s and with zero API spend. 3. The same Unsloth build is \~5x faster than Qwen's **Qwen3-32B**, which scores **82.20%** as well yet cr…”
- communityconfidence 60%
14.00tok/s — Qwen 3 32B on M4 Max via lm-studio
our signed data: M4 Max · Qwen 3 32B
“Indeed as others have mentioned, why not configure each to optimum, I get 14 tokens / s on my MBP16 M4 Max using LM Studio Qwen 3 32B Q8 MLX, I would suspect the Studio to be faster no?”
- communityconfidence 60%
80.00tok/s — Qwen3 32B on Cerebras via hosted-api
our signed data: Cerebras · Qwen3 32B
“n3 32B at 2,496 tok/s for $0.50/1M and Llama 4 Scout at 2,808 tok/s for $0.70/1M.<p>Compare to the same models elsewhere: Often stuck at 40-80 tok/s for similar or higher prices.<p>Arbitrage alert: 50x+ throughput boosts on the same model?<p>4. Quality overpays that defy logic Hi…”