llama3.1
12 workload results across 3 hardware configurations.
Fastest local config
154.4 decode tok/s
on RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB via ollama (Q4_K_M). see full run
For a standard Q4_K_M quantization using Ollama, here are measured results for short-chat generation.
On Ollama Q4_K_M, llama3.1 (8.0B) is fastest on the RTX 4090 at 137.3 tok/s, 1% faster than the RTX 3090 (136.2 tok/s).
| Card | Decode tok/s | TTFT (110-token prompt) | Run |
|---|---|---|---|
| RTX 4090 | 137.3 (single run) | 409 ms | r_2np7abzjobe |
| RTX 3090 | 136.2 (single run) | — | r_9hurqggbshk |
llama3.1 (8.0B), Q4_K_M, Ollama, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded. A dash marks a first-token time dropped as a cold model load.
Local runs (12 workload results)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 4090 (24GB) + AMD EPYC 7642 48-Core Processor (48c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 137.3tok/s | 269.1tok/s | 409ms | r_2np7abzjobe |
| chat-long | ollama@0.31.1 | Q4_K_M | 129.2tok/s | 4,652.7tok/s | 674ms | r_2np7abzjobe |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 136.0tok/s | no data | no data | r_2np7abzjobe |
| agent-trace | ollama@0.31.1 | Q4_K_M | 130.5tok/s | 3,955.2tok/s | 429ms | r_2np7abzjobe |
RTX 4090 (48GB) + AMD EPYC 7763 64-Core Processor (128c) + 1008GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 154.4tok/s | 328.0tok/s | 335ms | r_h1ub_1uxzdh |
| chat-long | ollama@0.31.1 | Q4_K_M | 146.3tok/s | 5,032.5tok/s | 623ms | r_h1ub_1uxzdh |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 154.2tok/s | no data | no data | r_h1ub_1uxzdh |
| agent-trace | ollama@0.31.1 | Q4_K_M | 149.4tok/s | 5,377.5tok/s | 387ms | r_h1ub_1uxzdh |
RTX 3090 (24GB) + AMD EPYC 7663 56-Core Processor (56c) + 252GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.31.1 | Q4_K_M | 136.2tok/s | 2.20tok/s | 49,960ms | r_9hurqggbshk |
| chat-long | ollama@0.31.1 | Q4_K_M | 127.5tok/s | 3,004.3tok/s | 1,044ms | r_9hurqggbshk |
| concurrent-decode | ollama@0.31.1 | Q4_K_M | 133.9tok/s | no data | no data | r_9hurqggbshk |
| agent-trace | ollama@0.31.1 | Q4_K_M | 127.5tok/s | 3,475.9tok/s | 527ms | r_9hurqggbshk |