Qwen2.5-14B-Instruct-4bit

7 workload results across 3 hardware configurations.

Fastest local config

133.3 decode tok/s

on RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB via llama.cpp. see full run

For a standard 4-bit quantization using MLX, here are measured results for short-chat generation.

On MLX 4-bit, Qwen2.5-14B-Instruct is fastest on the M3 Ultra (60-core GPU) at 70.85 tok/s, 392% faster than the M3 Pro (18-core GPU) (14.41 tok/s).

Card	Decode tok/s	TTFT (131-token prompt)	Run
M3 Ultra (60-core GPU)	70.85 (single run)	434 ms	r_v4bq1sviz4o
M3 Pro (18-core GPU)	14.41 (single run)	1111 ms	r_ia73dzeue0b

Qwen2.5-14B-Instruct, 4-bit, MLX, chat-short, batch 1. Only cards with their standard memory are listed; CPU-only hosts are excluded.

Local runs (7 workload results)

Runs from contributors' own machines via MLX, llama.cpp, vLLM, exllamav2, or ollama. Signed on the submitter's hardware.

RTX 5090 (32GB) + AMD Ryzen 7 9850X3D 8-Core Processor (8c) + 30GB

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	llama.cpp	-	130.3tok/s	no data	48.6ms	r_xr4qdv1hgf2
chat-long	llama.cpp	-	122.7tok/s	no data	489ms	r_xr4qdv1hgf2
concurrent-decode	llama.cpp	-	131.4tok/s	no data	no data	r_xr4qdv1hgf2
agent-trace	llama.cpp	-	122.8tok/s	20,315.4tok/s	111ms	r_xr4qdv1hgf2
chat-short	llama.cpp	-	133.3tok/s	no data	55.9ms	r_tj9bu7gvnvh

M3 Pro (18-core GPU) + 36GB unified

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	mlx@0.31.3	4bit	14.41tok/s	117.9tok/s	1,111ms	r_ia73dzeue0b

M3 Ultra (60-core GPU) + 96GB unified

Workload	Backend	Quant	decode tok/s	prefill tok/s	TTFT	Run
chat-short	mlx@0.31.3	4bit	70.85tok/s	301.7tok/s	434ms	r_v4bq1sviz4o