Qwen2.5-32B-Instruct-4bit

7 workload results across 3 hardware configurations.

Fastest local config

71.9 decode tok/s

on RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB via llama.cpp. see full run

For a standard 4-bit quantization using MLX, here are measured results for short-chat generation.

On MLX 4-bit, Qwen2.5-32B-Instruct is fastest on the M3 Ultra (60-core GPU) at 34.60 tok/s, 390% faster than the M3 Pro (18-core GPU) (7.05 tok/s).

Card	Decode tok/s	TTFT (131-token prompt)	Run
M3 Ultra (60-core GPU)	34.60 (single run)	923 ms	r_njgxtgyym1e
M3 Pro (18-core GPU)	7.05 (single run)	3921 ms	r_f4x3xaan2ay

Qwen2.5-32B-Instruct, 4-bit, MLX, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded.

Local runs (7 workload results)

Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.

RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	68.71tok/s	no data	77.0ms	r_bjy5a5izxjc
chat-long	llama.cpp	-	65.36tok/s	no data	968ms	r_bjy5a5izxjc
concurrent-decode	llama.cpp	-	68.60tok/s	no data	no data	r_bjy5a5izxjc
agent-trace	llama.cpp	-	66.45tok/s	10,407.9tok/s	214ms	r_bjy5a5izxjc
chat-short	llama.cpp	-	71.90tok/s	no data	115ms	r_twfs86tf_xf

M3 Pro (18-core GPU) + 36GB unified

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	mlx@0.31.3	-	7.05tok/s	33.41tok/s	3,921ms	r_f4x3xaan2ay

M3 Ultra (60-core GPU) + 96GB unified

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	mlx@0.31.3	4bit	34.60tok/s	141.9tok/s	923ms	r_njgxtgyym1e