RTX 4090 LLM benchmark
On the RTX 4090, measured single-user chat decode speeds are: ~7.6–8B models at 154.4 tok/s (Q4_K_M, ollama; Measured, middle of 2 results, source run) and ~14.8B models at 89.5 tok/s (Q4_K_M, ollama; Measured, 1 result, source run). Short-chat workload, local runtimes, dense models only; quantization, backend and settings vary between submissions, so check each row in the tables below.
Fastest single result in these submissions (any model size)
159.4 decode tok/s
qwen2.5-coder (7.6B) via ollama (Q4_K_M). Workload: chat-short. Reported hardware: RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB. see full run
qwen3.6
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 27.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| chat-long | 27.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| concurrent-decode | 27.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_h_659oy695r |
| agent-trace | 27.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 44.18tok/s | 1,910.5tok/s | 1,808ms | r_h_659oy695r |
qwen2.5-coder
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 14.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 89.45tok/s | 466.6tok/s | 281ms | r_73tnfdueq2h |
| chat-long | 14.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 83.84tok/s | 3,572.3tok/s | 887ms | r_73tnfdueq2h |
| concurrent-decode | 14.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 88.63tok/s | no data | no data | r_73tnfdueq2h |
| agent-trace | 14.8B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 86.21tok/s | 5,298.4tok/s | 395ms | r_73tnfdueq2h |
| chat-short | 7.6B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 159.4tok/s | 452.0tok/s | 290ms | r_mv8n8k9wu1e |
| chat-long | 7.6B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 154.7tok/s | 5,998.7tok/s | 528ms | r_mv8n8k9wu1e |
| concurrent-decode | 7.6B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 161.1tok/s | no data | no data | r_mv8n8k9wu1e |
| agent-trace | 7.6B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 158.1tok/s | 6,421.1tok/s | 320ms | r_mv8n8k9wu1e |
llama3.1
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 8.0B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 154.4tok/s | 328.0tok/s | 335ms | r_h1ub_1uxzdh |
| chat-long | 8.0B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 146.3tok/s | 5,032.5tok/s | 623ms | r_h1ub_1uxzdh |
| concurrent-decode | 8.0B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 154.2tok/s | no data | no data | r_h1ub_1uxzdh |
| agent-trace | 8.0B | RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | ollama@0.31.1 | Q4_K_M | 149.4tok/s | 5,377.5tok/s | 387ms | r_h1ub_1uxzdh |