Local LLM speed, measured

Day-one test: NVIDIA's 2-bit Nemotron 3.5 Bolt 74B-A9B fits 24GB for single-user use. 107.3 tok/s on RTX 4090, 2.06x Qwen3.8-27B on a 5090. Ollama 0.40.4 can't load it yet.

Day-one: Muse Spark Open (Unsloth 1-bit 175.6 GB) on 3x A100. 25.2 tok/s chat-short, ~168 GiB peak. Roughly matches Apple's unverified dual-Studio claim. Single runs.

Measured on M5 Max: Qwen3.8-27B waited 8.17 s for the first token at 840 tokens and 113 s at 12K. Why Mac prefill is slow and what actually cuts the wait.

Measured on an RTX 4090: Qwen3.8-27B Q4_K_M decoded 84.51 tok/s with MTP on vs 43.06 off (llama.cpp). Where the speedup holds and what it costs.

On the same RTX 5090, Qwen3.6-35B-A3B decoded at 224 tok/s vs 74 tok/s for dense Qwen3.6-27B. Measured MoE and dense rows from signed runs on four rigs.

Original Qwen3.8-27B 4-bit MLX study on a 48GB M5 Max: 12 samples, first-token waits, prefix-cache costs, memory observations and downloadable reproduction files.

RTX 5090 test of Gemma 4 E2B QAT Q4_0 in llama.cpp: schema constraints fixed JSON parsing in nine small cases, but one routing decision remained wrong. Raw results and settings.

Slow local LLM? Check model loading, prompt processing, GPU offload, context memory and queued requests before you buy new hardware.

Measured on one RTX 5090: Gemma 4 12B QAT decoded 139.5 tok/s vs 65.7 for Qwen3.8-27B Q4_K_M (median of 3), with first-token latency and VRAM.

DeepSeek V4 local hardware: documented RTX 5090 plus system-RAM serving, runtime limits, and why the new V4.1-Flash checkpoint does not fit one 32GB GPU.

GLM-4.7-Flash on RTX 4090: inspect the 129.9 tok/s agent-trace result, missing chat timings, 24GB memory limits and a local benchmark setup.

Gemma 3 4B, 12B and 27B RTX 4090 source runs: short and longer prompt speeds, first-token waits, memory limits and how to choose a configuration.

Which LLMs fit a 512GB Mac Studio: Qwen3.8 and GLM-5.3 MLX downloads with exact sizes, memory checks and labelled measured M3 Ultra speeds.

Measured: Qwen3.6-27B Q4_K_M decoded 47.14 tok/s on a 24GB RTX 4090 and 73.96 on an RTX 5090 (llama.cpp, chat-short). Fit, context and what to check.

Signed runs on GPUs and Macs. On one M3 Ultra, MoE Qwen3-Coder-30B decoded 112.2 tok/s vs 34.5 for dense Qwen2.5-Coder-32B. What counts as fast.

Signed, reproducible decode tok/s for the top local coding models on a single RTX 5090, and why small-MoE coders decode ~4x faster than a dense 32B.

Measured on Ollama: Llama 3.1 8B ran 136.2 tok/s on an RTX 3090, within 1% of an RTX 4090 (137.3). On Qwen2.5-Coder 14.8B the 4090 was about 17% faster.

Prefill reads your prompt, decode writes the answer. Measured Gemma 4 and Qwen runs show why fast streaming can still start after a long wait.

Across 9 matched model pairs, the RTX 4090 decoded 0.8% to 22.5% faster than the RTX 3090 (median 14.9%, chat-short). What fits in 24GB, and when to upgrade.

Measured decode tok/s for Llama 3.1 8B on the RTX 5090, 4090, 3090, and Apple Silicon: which runs it fastest, and which is the best value.

Choose hardware for a local coding agent using task quality, memory and latency. Historical 3090/4090 source rows, 5090 studies and Mac configuration caveats.

gpt-oss-20b: 318.4 tok/s on an RTX 5090 (llama.cpp), 152.7 on a 96GB M3 Ultra (MLX, MXFP4); different runtimes, not matched. Only the Mac ran 70B (16.8 tok/s).

Measured Qwen3-Coder-30B-A3B decode: 259.9 tok/s on an RTX 5090, 145.2 on a 4090, 113.3 on an M4 Max and 112.2 on an M3 Ultra, single request.

Signed benchmarks: Qwen3-Coder-30B-A3B decodes about 2.5x faster than Codestral-22B on the same hardware, because it is a small mixture-of-experts. Which to run.

DeepSeek-R1 8B, 14B and 32B RTX 4090 source runs, memory-fit limits and a GPU-placement checklist. A slow 32B result does not prove you need a bigger card.