qwen2.5-coder
24 workload results across 4 hardware configurations.
Fastest local config
161.1 decode tok/s
on RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB via ollama (Q4_K_M). see full run
For a standard Q4_K_M quantization using Ollama, here are measured results for short-chat generation.
On Ollama Q4_K_M, qwen2.5-coder (14.8B) is fastest on the RTX 4090 at 81.18 tok/s, 17% faster than the RTX 3090 (69.21 tok/s).
| Card | Decode tok/s | TTFT (131-token prompt) | Run |
|---|---|---|---|
| RTX 4090 | 81.18 (single run) | 404 ms | r_ttniv2j74vk |
| RTX 3090 | 69.21 (single run) | 326 ms | r_6ahy-dq0f_0 |
qwen2.5-coder (14.8B), Q4_K_M, Ollama, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded.
Local runs (24 workload results)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 81.18tok/s | 324.4tok/s | 404ms | r_ttniv2j74vk |
| chat-long | ollama@0.31.1 | Q4_K_M | 76.65tok/s | 3,252.2tok/s | 974ms | r_ttniv2j74vk |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 80.11tok/s | no data | no data | r_ttniv2j74vk |
| agent-trace | ollama@0.31.1 | Q4_K_M | 78.63tok/s | 4,874.6tok/s | 449ms | r_ttniv2j74vk |
| chat-short | ollama@0.31.1 | Q4_K_M | 131.6tok/s | 446.2tok/s | 294ms | r_nrpc8-onxu2 |
| chat-long | ollama@0.31.1 | Q4_K_M | 133.1tok/s | 5,116.7tok/s | 619ms | r_nrpc8-onxu2 |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 137.3tok/s | no data | no data | r_nrpc8-onxu2 |
| agent-trace | ollama@0.31.1 | Q4_K_M | 135.8tok/s | 5,297.3tok/s | 352ms | r_nrpc8-onxu2 |
RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 89.45tok/s | 466.6tok/s | 281ms | r_73tnfdueq2h |
| chat-long | ollama@0.31.1 | Q4_K_M | 83.84tok/s | 3,572.3tok/s | 887ms | r_73tnfdueq2h |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 88.63tok/s | no data | no data | r_73tnfdueq2h |
| agent-trace | ollama@0.31.1 | Q4_K_M | 86.21tok/s | 5,298.4tok/s | 395ms | r_73tnfdueq2h |
| chat-short | ollama@0.31.1 | Q4_K_M | 159.4tok/s | 452.0tok/s | 290ms | r_mv8n8k9wu1e |
| chat-long | ollama@0.31.1 | Q4_K_M | 154.7tok/s | 5,998.7tok/s | 528ms | r_mv8n8k9wu1e |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 161.1tok/s | no data | no data | r_mv8n8k9wu1e |
| agent-trace | ollama@0.31.1 | Q4_K_M | 158.1tok/s | 6,421.1tok/s | 320ms | r_mv8n8k9wu1e |
RTX 3090 (24GB) + AMD EPYC 7702P 64-Core Processor (64c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 69.21tok/s | 402.0tok/s | 326ms | r_6ahy-dq0f_0 |
| chat-long | ollama@0.31.1 | Q4_K_M | 65.83tok/s | 2,070.6tok/s | 1,530ms | r_6ahy-dq0f_0 |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 67.64tok/s | no data | no data | r_6ahy-dq0f_0 |
| agent-trace | ollama@0.31.1 | Q4_K_M | 63.63tok/s | 3,518.8tok/s | 483ms | r_6ahy-dq0f_0 |
RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 39.86tok/s | 2.83tok/s | 46,337ms | r_pb0jpnbujji |
| chat-long | ollama@0.31.1 | Q4_K_M | 135.0tok/s | 3,319.4tok/s | 954ms | r_pb0jpnbujji |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 139.2tok/s | no data | no data | r_pb0jpnbujji |
| agent-trace | ollama@0.31.1 | Q4_K_M | 134.2tok/s | 4,111.1tok/s | 487ms | r_pb0jpnbujji |