Qwen2.5-7B-Instruct-4bit
8 workload results across 3 hardware configurations.
Fastest local config
246.7 decode tok/s
on RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB via llama.cpp. see full run
Local runs (8 workload results)
Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.
RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | llama.cpp | - | 232.7tok/s | no data | 37.0ms | r_1shiviswt3d |
| chat-long | llama.cpp | - | 232.4tok/s | no data | 239ms | r_1shiviswt3d |
| concurrent-decode | llama.cpp | - | 240.7tok/s | no data | no data | r_1shiviswt3d |
| agent-trace | llama.cpp | - | 232.5tok/s | 38,007.7tok/s | 57.1ms | r_1shiviswt3d |
| chat-short | llama.cpp | - | 246.7tok/s | no data | 33.9ms | r_3yn-4321hp- |
M3 Pro (18-core GPU) + 36GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | mlx@0.31.3 | 4bit | 30.52tok/s | 161.9tok/s | 809ms | r_llzv_g-ymaf |
| chat-short | mlx@0.31.3 | 4bit | 15.67tok/s | 54.34tok/s | 2,411ms | r_e3t93rscswq |
M3 Ultra (60-core GPU) + 96GB unified
| Workload | Backend | Quant | decode tok/s | prefill tok/s | TTFT | Run |
|---|---|---|---|---|---|---|
| chat-short | mlx@0.31.3 | 4bit | 139.6tok/s | 190.0tok/s | 689ms | r_5r6rhiynenc |
Qwen2.5-7B-Instruct-4bit on hardware
Request a Qwen2.5-7B-Instruct-4bit run
No measured run yet for these pairs. Vote for the one you want; the most-requested runs go first. See all requests.
0 votes
We never sell or share your email. Privacy
We can rent this card · Submit your own