RTX 4090 LLM benchmark
On the RTX 4090, measured single-user chat decode speeds are: ~7.6–8B models at 131.6 tok/s (Q4_K_M, ollama; Measured, middle of 2 results, source run), ~12–14.8B models at 89.9 tok/s (Q4_K_M, llama.cpp; Measured, middle of 4 results, source run) and ~27–32B models at 47.0 tok/s (Q4_K_M, ollama; Measured, middle of 6 results, source run). Short-chat workload, local runtimes, dense models only; quantization, backend and settings vary between submissions, so check each row in the tables below.
Fastest single result in these submissions (any model size)
242.6 decode tok/s
gpt-oss-20b via llama.cpp (Q4_K_M). Workload: chat-short. Reported hardware: RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB. see full run
glm-4.7-flash
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 164.9tok/s | 3.74tok/s | 28,106ms | r_ted0u8uyre0 |
| chat-long | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 146.1tok/s | 3,617.7tok/s | 866ms | r_ted0u8uyre0 |
| concurrent-decode | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 155.2tok/s | no data | no data | r_ted0u8uyre0 |
| agent-trace | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 155.5tok/s | 3,224.6tok/s | 641ms | r_ted0u8uyre0 |
| chat-short | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_o636l3cc-rr |
| chat-long | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_o636l3cc-rr |
| concurrent-decode | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_o636l3cc-rr |
| agent-trace | 29.9B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 129.9tok/s | 2,491.1tok/s | 1,262ms | r_o636l3cc-rr |
gemma-4-12b-it-qat
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_0 | 97.72tok/s | 1,109.0tok/s | 112ms | r_jfh0qkngw76 |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_0 | 86.30tok/s | 1,743.0tok/s | 1,831ms | r_jfh0qkngw76 |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_0 | 95.49tok/s | no data | no data | r_jfh0qkngw76 |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_0 | 89.10tok/s | 2,305.0tok/s | 983ms | r_jfh0qkngw76 |
Qwen3.6-35B-A3B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_K_M | 160.6tok/s | 464.3tok/s | 241ms | r_m6j1zxwvp-f |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_K_M | 160.4tok/s | 4,588.8tok/s | 693ms | r_m6j1zxwvp-f |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_K_M | 163.5tok/s | no data | no data | r_m6j1zxwvp-f |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@8931 (9725a313b) | Q4_K_M | 162.7tok/s | 6,547.5tok/s | 330ms | r_m6j1zxwvp-f |
Nemotron-3.5-Bolt-74B-A9B-24GB-ctx16k
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 105.7tok/s | 252.8tok/s | 475ms | r_qibhq5bkmgq |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 105.2tok/s | 1,746.2tok/s | 1,861ms | r_qibhq5bkmgq |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 105.8tok/s | no data | no data | r_qibhq5bkmgq |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 102.0tok/s | 3,280.7tok/s | 673ms | r_qibhq5bkmgq |
Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 107.0tok/s | 253.3tok/s | 474ms | r_iu0s18cs9d8 |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 105.5tok/s | 1,743.8tok/s | 1,863ms | r_iu0s18cs9d8 |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 106.4tok/s | no data | no data | r_iu0s18cs9d8 |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 103.0tok/s | 3,340.4tok/s | 661ms | r_iu0s18cs9d8 |
Nemotron-3.5-Bolt-74B-A9B-24GB
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 107.3tok/s | 255.8tok/s | 469ms | r_vkjsinuhlyl |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 106.1tok/s | 1,814.7tok/s | 1,790ms | r_vkjsinuhlyl |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 106.7tok/s | no data | no data | r_vkjsinuhlyl |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.7.0-dev (build 11669, commit 620d2225c) | - | 103.9tok/s | 3,318.9tok/s | 636ms | r_vkjsinuhlyl |
LFM2.5-24B-A2B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (64c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit a0e6dc7) | Q4_K_M | 231.9tok/s | 640.1tok/s | 178ms | r_bqmpv9kncyd |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (64c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit a0e6dc7) | Q4_K_M | 233.4tok/s | 9,512.4tok/s | 345ms | r_bqmpv9kncyd |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (64c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit a0e6dc7) | Q4_K_M | 236.8tok/s | no data | no data | r_bqmpv9kncyd |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7452 32-Core Processor (64c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit a0e6dc7) | Q4_K_M | 231.0tok/s | 13,801.7tok/s | 153ms | r_bqmpv9kncyd |
Qwen3.6-27B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 3fb4767) | Q4_K_M | 47.14tok/s | 349.5tok/s | 320ms | r_d_wda_rw62f |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 3fb4767) | Q4_K_M | 46.57tok/s | 2,107.9tok/s | 1,510ms | r_d_wda_rw62f |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 3fb4767) | Q4_K_M | 47.15tok/s | no data | no data | r_d_wda_rw62f |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB | llama.cpp@0.6.0-dev (build 1, commit 3fb4767) | Q4_K_M | 46.19tok/s | 3,445.6tok/s | 626ms | r_d_wda_rw62f |
qwen2.5-coder
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 14.8B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 81.18tok/s | 324.4tok/s | 404ms | r_ttniv2j74vk |
| chat-long | 14.8B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 76.65tok/s | 3,252.2tok/s | 974ms | r_ttniv2j74vk |
| concurrent-decode | 14.8B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 80.11tok/s | no data | no data | r_ttniv2j74vk |
| agent-trace | 14.8B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 78.63tok/s | 4,874.6tok/s | 449ms | r_ttniv2j74vk |
| chat-short | 7.6B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 131.6tok/s | 446.2tok/s | 294ms | r_nrpc8-onxu2 |
| chat-long | 7.6B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 133.1tok/s | 5,116.7tok/s | 619ms | r_nrpc8-onxu2 |
| concurrent-decode | 7.6B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 137.3tok/s | no data | no data | r_nrpc8-onxu2 |
| agent-trace | 7.6B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 135.8tok/s | 5,297.3tok/s | 352ms | r_nrpc8-onxu2 |
llama3.1
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 8.0B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 137.3tok/s | 269.1tok/s | 409ms | r_2np7abzjobe |
| chat-long | 8.0B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 129.2tok/s | 4,652.7tok/s | 674ms | r_2np7abzjobe |
| concurrent-decode | 8.0B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 136.0tok/s | no data | no data | r_2np7abzjobe |
| agent-trace | 8.0B | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | ollama@0.31.1 | Q4_K_M | 130.5tok/s | 3,955.2tok/s | 429ms | r_2np7abzjobe |
Qwen3-32B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 43.47tok/s | 778.6tok/s | 141ms | r_ayy9rjgha5r |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 40.66tok/s | 2,333.3tok/s | 1,349ms | r_ayy9rjgha5r |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 42.69tok/s | no data | no data | r_ayy9rjgha5r |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 41.94tok/s | 5,982.3tok/s | 342ms | r_ayy9rjgha5r |
Qwen2.5-Coder-14B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 89.87tok/s | 2,292.6tok/s | 57.1ms | r_eqnsqmgz29a |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 84.17tok/s | 4,989.1tok/s | 635ms | r_eqnsqmgz29a |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 88.90tok/s | no data | no data | r_eqnsqmgz29a |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 86.64tok/s | 14,684.0tok/s | 142ms | r_eqnsqmgz29a |
Qwen3-Coder-30B-A3B-Instruct
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 232.5tok/s | 831.9tok/s | 132ms | r_1k8942tw3zf |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 206.9tok/s | 7,688.7tok/s | 409ms | r_1k8942tw3zf |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 225.1tok/s | no data | no data | r_1k8942tw3zf |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 216.4tok/s | 19,138.1tok/s | 108ms | r_1k8942tw3zf |
gpt-oss-20b
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 242.6tok/s | 593.8tok/s | 280ms | r_t-a9v_ipdit |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 231.2tok/s | 7,864.1tok/s | 405ms | r_t-a9v_ipdit |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 239.6tok/s | no data | no data | r_t-a9v_ipdit |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 236.1tok/s | 13,439.2tok/s | 150ms | r_t-a9v_ipdit |
Qwen3.8-27B-mtp-on
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 84.51tok/s | 299.3tok/s | 515ms | r_7j8rpk9mwhg |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 73.10tok/s | 1,585.8tok/s | 2,033ms | r_7j8rpk9mwhg |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 87.04tok/s | no data | no data | r_7j8rpk9mwhg |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 77.60tok/s | 2,912.1tok/s | 787ms | r_7j8rpk9mwhg |
Qwen3.8-27B
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 43.06tok/s | 428.6tok/s | 359ms | r_11kn1epxewh |
| chat-long | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 42.63tok/s | 1,915.1tok/s | 1,684ms | r_11kn1epxewh |
| concurrent-decode | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 43.00tok/s | no data | no data | r_11kn1epxewh |
| agent-trace | not supplied | RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB | llama.cpp@0.6.0-dev (build 1, commit df18bee) | Q4_K_M | 42.07tok/s | 3,644.0tok/s | 629ms | r_11kn1epxewh |
gemma3
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 27.4B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 46.95tok/s | 144.5tok/s | 817ms | r_x23y_sg24pm |
| chat-long | 27.4B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 45.25tok/s | 1,594.1tok/s | 1,998ms | r_x23y_sg24pm |
| concurrent-decode | 27.4B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 46.17tok/s | no data | no data | r_x23y_sg24pm |
| agent-trace | 27.4B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 45.33tok/s | 2,556.8tok/s | 956ms | r_x23y_sg24pm |
| chat-short | 12.2B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 92.65tok/s | 171.4tok/s | 689ms | r_3kmrc135e0e |
| chat-long | 12.2B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 87.60tok/s | 2,206.4tok/s | 1,444ms | r_3kmrc135e0e |
| concurrent-decode | 12.2B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 90.43tok/s | no data | no data | r_3kmrc135e0e |
| agent-trace | 12.2B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 88.53tok/s | 3,123.9tok/s | 815ms | r_3kmrc135e0e |
| chat-short | 4.3B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 195.0tok/s | 167.2tok/s | 706ms | r_dlanfbgym0h |
| chat-long | 4.3B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 187.6tok/s | 3,254.1tok/s | 979ms | r_dlanfbgym0h |
| concurrent-decode | 4.3B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 193.1tok/s | no data | no data | r_dlanfbgym0h |
| agent-trace | 4.3B | RTX 4090 (24GB) + AMD EPYC 7443 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 190.0tok/s | 3,543.4tok/s | 715ms | r_dlanfbgym0h |
r1
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 14.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_zrjj-pj93q8 |
| chat-long | 14.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 81.09tok/s | 240.4tok/s | 13,070ms | r_zrjj-pj93q8 |
| concurrent-decode | 14.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_zrjj-pj93q8 |
| agent-trace | 14.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_zrjj-pj93q8 |
| chat-short | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_fg77v2hhohb |
| chat-long | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 131.9tok/s | 686.3tok/s | 4,578ms | r_fg77v2hhohb |
| concurrent-decode | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_fg77v2hhohb |
| agent-trace | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 133.8tok/s | 8,852.4tok/s | 366ms | r_fg77v2hhohb |
| chat-short | 32.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_l2ck67ls40i |
| chat-long | 32.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 3.76tok/s | 14.28tok/s | 219,986ms | r_l2ck67ls40i |
| concurrent-decode | 32.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_l2ck67ls40i |
| agent-trace | 32.8B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 3.79tok/s | 3,515.1tok/s | 971ms | r_l2ck67ls40i |
| chat-short | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_2r1p3w9ps7c |
| chat-long | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 131.0tok/s | 599.9tok/s | 5,238ms | r_2r1p3w9ps7c |
| concurrent-decode | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_2r1p3w9ps7c |
| agent-trace | 8.2B | RTX 4090 (24GB) + AMD EPYC 75F3 32-Core Processor (64c) + 504GB | ollama@0.31.1 | Q4_K_M | 132.4tok/s | 9,612.8tok/s | 337ms | r_2r1p3w9ps7c |
qwen3-coder
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 30.5B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 145.2tok/s | 195.6tok/s | 562ms | r_wjq32z47vlp |
| chat-long | 30.5B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 165.5tok/s | 3,527.2tok/s | 892ms | r_wjq32z47vlp |
| concurrent-decode | 30.5B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | 179.9tok/s | no data | no data | r_wjq32z47vlp |
| agent-trace | 30.5B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | Q4_K_M | no data | no data | no data | r_wjq32z47vlp |
gpt-oss
| Workload | Model size | Reported hardware | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|---|---|
| chat-short | 20.9B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | MXFP4 | no data | no data | no data | r_iu2sfa9ykvw |
| chat-long | 20.9B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | MXFP4 | 141.7tok/s | 631.5tok/s | 5,048ms | r_iu2sfa9ykvw |
| concurrent-decode | 20.9B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | MXFP4 | no data | no data | no data | r_iu2sfa9ykvw |
| agent-trace | 20.9B | RTX 4090 (24GB) + AMD EPYC 7352 24-Core Processor (24c) + 252GB | ollama@0.31.1 | MXFP4 | no data | no data | no data | r_iu2sfa9ykvw |
Models measured on RTX 4090
- glm-4.7-flash benchmarks
- gemma-4-12b-it-qat benchmarks
- Qwen3.6-35B-A3B benchmarks
- Nemotron-3.5-Bolt-74B-A9B-24GB-ctx16k benchmarks
- Nemotron-3.5-Bolt-74B-A9B-24GB-ctx8k benchmarks
- Nemotron-3.5-Bolt-74B-A9B-24GB benchmarks
- LFM2.5-24B-A2B benchmarks
- Qwen3.6-27B benchmarks
- qwen2.5-coder benchmarks
- llama3.1 benchmarks
- Qwen3-32B benchmarks
- Qwen2.5-Coder-14B-Instruct benchmarks
- Qwen3-Coder-30B-A3B-Instruct benchmarks
- gpt-oss-20b benchmarks
- Qwen3.8-27B-mtp-on benchmarks
- Qwen3.8-27B benchmarks
- gemma3 benchmarks
- r1 benchmarks
- qwen3-coder benchmarks
- gpt-oss benchmarks