llama3.2
12 workload results across 3 hardware configurations.
Fastest local config
466.9 decode tok/s
on RTX 5880 (48GB) + AMD EPYC 9T24 96-Core Processor (16c) + 62GB via ollama (Q8_0). see full run
Local runs (12 workload results)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 5880 (48GB) + AMD EPYC 9T24 96-Core Processor (16c) + 62GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.34.2 | Q8_0 | 466.9tok/s | 44.32tok/s | 2,821ms | r_b948ehgi8g4 |
| chat-long | ollama@0.34.2 | Q8_0 | 441.9tok/s | 39,673.0tok/s | 79.4ms | r_b948ehgi8g4 |
| concurrent-decode | ollama@0.34.2 | Q8_0 | 464.2tok/s | no data | no data | r_b948ehgi8g4 |
| agent-trace | ollama@0.34.2 | Q8_0 | 452.3tok/s | 91,077.2tok/s | 21.8ms | r_b948ehgi8g4 |
AMD64 Family 23 Model 104 Stepping 1, AuthenticAMD (6c) + 15GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.34.0 | Q8_0 | 23.88tok/s | 39.41tok/s | 3,172ms | r_b1rn5gi6b3b |
| chat-long | ollama@0.34.0 | Q8_0 | 14.69tok/s | 93.91tok/s | 33,563ms | r_b1rn5gi6b3b |
| concurrent-decode | ollama@0.34.0 | Q8_0 | 19.96tok/s | no data | no data | r_b1rn5gi6b3b |
| agent-trace | ollama@0.34.0 | Q8_0 | 16.80tok/s | 203.2tok/s | 7,977ms | r_b1rn5gi6b3b |
M1 Pro (16-core GPU) + 16GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | ollama@0.32.14 | Q8_0 | 103.1tok/s | 8.77tok/s | 14,247ms | r_w-p7v--eync |
| chat-long | ollama@0.32.14 | Q8_0 | 81.35tok/s | 1,246.5tok/s | 2,529ms | r_w-p7v--eync |
| concurrent-decode | ollama@0.32.14 | Q8_0 | 97.26tok/s | no data | no data | r_w-p7v--eync |
| agent-trace | ollama@0.32.14 | Q8_0 | 93.14tok/s | 2,975.3tok/s | 516ms | r_w-p7v--eync |