Does a 4-bit model use a 4-bit KV cache?

No. Weight quantization and KV-cache precision are separate settings. A 4-bit model can keep attention state at 16-bit precision. Longer prompts, generated tokens and parallel requests can increase that state; runtime buffers and retained prompt caches also consume memory. Check the exact backend and version before applying a cache setting. Ollama's MLX runner and the separate Python MLX-LM package do not share configuration flags.

What is a good tokens-per-second (tok/s) for a local LLM?

There is no universal responsive-rate threshold. Judge first-token wait and the time to finish your intended response. Tokens are not words, and reading speed, tokenizer, language and output length vary. tok/s here means decode tok/s — the streaming-output rate. llm-speed publishes measured decode tok/s per model and hardware so you can check a specific setup instead of guessing.

What's the difference between decode tok/s and prefill tok/s?

Prefill tok/s measures prompt processing before generation. Decode tok/s measures generated output. Prefill can process prompt positions in parallel, but there is no fixed ratio between these rates across models, runtimes and workloads. Assess both phases: long inputs can make prompt processing dominate first-token wait, while long outputs can make generation dominate completion time.

What is TTFT (time to first token) in an LLM benchmark?

TTFT, or time-to-first-token, is the wall-clock duration in milliseconds between submitting a prompt and receiving the first generated token. TTFT can include model loading, queueing, prompt processing and transport. Separate those costs and record whether the model and prefix cache were warm.

What is decode tok/s?

Decode tok/s measures generated tokens per second during the output phase. Memory bandwidth can limit performance, but bottlenecks depend on the model, hardware, context and serving strategy. llm-speed measures it as wall-clock output tokens divided by decode wall time, batch size 1 unless the workload says otherwise.

Glossary

LLM inference speed glossary

Every term llm-speed uses when reporting a benchmark, defined in one sentence first and elaborated below. Cross-reference /methodology for workload specs and /privacy for data-handling terms.

Common questions about LLM inference speed

Does a 4-bit model use a 4-bit KV cache?: No. Weight quantization and KV-cache precision are separate settings. A 4-bit model can keep attention state at 16-bit precision. Longer prompts, generated tokens and parallel requests can increase that state; runtime buffers and retained prompt caches also consume memory. Check the exact backend and version before applying a cache setting. Ollama's MLX runner and the separate Python MLX-LM package do not share configuration flags.
What is a good tokens-per-second (tok/s) for a local LLM?: There is no universal responsive-rate threshold. Judge first-token wait and the time to finish your intended response. Tokens are not words, and reading speed, tokenizer, language and output length vary. tok/s here means decode tok/s — the streaming-output rate. llm-speed publishes measured decode tok/s per model and hardware so you can check a specific setup instead of guessing.
What's the difference between decode tok/s and prefill tok/s?: Prefill tok/s measures prompt processing before generation. Decode tok/s measures generated output. Prefill can process prompt positions in parallel, but there is no fixed ratio between these rates across models, runtimes and workloads. Assess both phases: long inputs can make prompt processing dominate first-token wait, while long outputs can make generation dominate completion time.
What is TTFT (time to first token) in an LLM benchmark?: TTFT, or time-to-first-token, is the wall-clock duration in milliseconds between submitting a prompt and receiving the first generated token. TTFT can include model loading, queueing, prompt processing and transport. Separate those costs and record whether the model and prefix cache were warm.
What is decode tok/s?: Decode tok/s measures generated tokens per second during the output phase. Memory bandwidth can limit performance, but bottlenecks depend on the model, hardware, context and serving strategy. llm-speed measures it as wall-clock output tokens divided by decode wall time, batch size 1 unless the workload says otherwise.

Why a 4-bit model can still run out of memory

Before buying more memory, separate three costs: model weights, state for the current request, and state retained for later requests. Reducing one does not necessarily reduce the others. These are configuration and source checks, not a measured speed or quality comparison.

Weights: the downloaded model: A Q4 GGUF or 4-bit MLX label describes weight storage. File size helps planning but is not the complete resident-memory requirement.
KV cache: attention state for retained tokens: For equal key/value head dimensions in ordinary full attention, tensor bytes are approximately 2 × attention layers × KV heads × head dimension × retained tokens × bytes per value × independent sequences. Quantization metadata, allocation padding, sharing and non-attention state need separate accounting. Do not apply this formula unchanged to every hybrid model.
Prefix reuse: state kept between requests: A reusable prefix can avoid repeated prompt processing, but holding it costs memory. A new chat does not prove that the server has discarded earlier cached state. Record the cache policy when comparing runs.

Ollama on Mac and Python MLX-LM use different controls

Ollama’s FAQ documents OLLAMA_KV_CACHE_TYPE with f16, q8_0 and q4_0, alongside Flash Attention. Its approximate half/quarter memory savings concern the cache, not the whole process. Accuracy effects depend on the model and task. Apply environment settings to the process serving requests; a terminal export does not configure an already-running macOS application.

Do not assume those settings enable an 8-bit cache in Ollama’s MLX path. In the reviewed MLX KVCache implementation, new cache tensors retain the incoming keys’ and values’ data types; this path does not expose a q8_0 conversion. That source observation is scoped to the linked revision, not a test of every installed Ollama version.

The separate Python MLX-LM generation CLI exposes --kv-bits, --kv-group-size and --quantized-kv-start. Check your installed version’s help and model/cache compatibility. A 4-bit weight download does not activate these controls, and Python flags do not configure Ollama’s runner. Keep the quantization start threshold explicit in comparisons.

Check one cause at a time

Record the runtime/version, model file, actual prompt and output tokens, configured context, concurrent requests and memory pressure.
Repeat a fixed request before extending the conversation. Growth with longer history and growth across equally sized independent requests are different symptoms; neither alone establishes a memory leak.
Reduce the request budget or parallelism, then measure again. Only test cache quantization when the backend supports it, checking task correctness as well as memory and latency.

Source review: September 17, 2026. Use the memory-fit calculator with its stated assumptions, or inspect our Qwen3.8 MLX context study for measured first-token and prefix-reuse costs. That study did not test cache quantization and does not establish its quality or memory savings.

Metrics & timing

tok/s — tokens per second: tok/s means tokens per second. It describes token throughput, not words per second or a universal measure of human reading speed.; Technically, tok/s (tokens per second) is the rate at which a language model produces or consumes tokens during a benchmark. On llm-speed, tok/s without further qualification refers to decode tok/s — the streaming-output rate users care about during chat.
decode tok/s: decode tok/s measures generated tokens per second during the output phase. Memory bandwidth can limit performance, but context, batch size, compute and serving strategy also matter.; llm-speed measures decode tok/s as wall-clock output_tokens divided by decode wall time, batch size 1 unless the workload (concurrent-decode) says otherwise.
prefill tok/s: prefill tok/s measures the prompt tokens processed per second before generation. Runtimes can process prompt positions in parallel; the achieved rate depends on model, input length, hardware and implementation.; Do not assume a fixed prefill-to-decode ratio. Compare actual input and output lengths, cache conditions and both phase timings for the intended workload.
TTFT — time-to-first-token: TTFT, or time-to-first-token, is the wall-clock duration in milliseconds between submitting a prompt and receiving the first generated token.; Loading, queueing, prompt processing and transport can all contribute. A first request and a warm request with a reused prefix can have different first-token waits.
p50: p50, the 50th percentile, is the median per-token decode latency in milliseconds; half of decoded tokens take less than this and half take longer.
p95: p95, the 95th percentile, is the per-token decode latency in milliseconds below which 95% of decoded tokens fall; it captures tail latency that the median hides.
wall_ms: wall_ms is the end-to-end wall-clock duration of a workload in milliseconds, from request start to last token received.
agent-trace TTFT: agent-trace TTFT is the per-turn time-to-first-token captured during the agent-trace workload (W5); it is reported alongside per-turn decode tok/s, end-to-end wall time, and prefix-cache hit rate, so a backend whose first turn is fast but later turns regress (because the prefix cache is not hit) is visible from a single row.

Workloads & internals

suite-v1: suite-v1 is the current versioned llm-speed workload suite, comprising chat-short, chat-long, long-context-decay, concurrent-decode, and agent-trace; the suite_version field on every result identifies which protocol produced it.; Future suite versions (suite-v2, etc.) will not invalidate older results; runs stay accessible and comparable within their own version.
chat-short: chat-short (W1) is the baseline llm-speed workload: a 128-token natural-language prompt and a 256-token completion at batch size 1.
chat-long: chat-long (W2) is the long-prompt llm-speed workload: a 4,096-token prompt assembled from fixture paragraphs and a 1,024-token completion at batch size 1, designed to surface prefill scaling.
long-context-decay: long-context-decay (W3) is the opt-in llm-speed workload that re-runs the same model at 32k, 64k, and 128k input context (where supported), each producing a 256-token completion, to capture how prefill and decode degrade with context length.
concurrent-decode: concurrent-decode (W4) is the llm-speed workload that runs multiple simultaneous decode streams at batch sizes 1, 4, 8, and 16, reporting aggregate throughput, per-stream throughput, and p50/p95 latency at each batch.; Backends without true concurrent decoding (for example, MLX which runs serially in-process) are flagged on the result so the number is not misread as parallel throughput.
agent-trace: agent-trace (W5) is the llm-speed workload that replays a fixed multi-turn coding-agent trace with realistic tool calls (read-file, write-file, run-tests, fix), growing the context to about 16k tokens by the final turn, to measure inference speed under the workload that matters for daily-driver agentic LLM use.
batch size: batch size is the number of independent decode streams a backend runs in parallel; in the chat-short, chat-long, long-context-decay, and agent-trace workloads it is 1, while concurrent-decode sweeps batch 1, 4, 8, and 16.
context length: context length is the token budget available to a request, including its prompt and generated continuation. An agent prompt can include system instructions, tools and previous turns; configured capacity and the tokens actually processed are different measurements.
prefix cache: prefix cache, also known as prompt cache, is a backend feature that stores the KV-cache state for a prompt prefix so subsequent requests sharing that prefix skip prefill on the cached portion.; llm-speed records prefix_cache_hit_rate when the backend exposes it, so agent-trace numbers are interpretable on backends that cache between turns and on backends that do not.
prefill: prefill is the phase of LLM inference in which the model processes the input prompt to populate its KV cache before any output tokens are generated.
decode: decode is the phase of LLM inference in which the model autoregressively produces output tokens, one token per forward pass, after prefill has completed.
KV cache — key/value cache: the KV cache, short for key/value cache, is the per-layer attention state a transformer accumulates during prefill so that decode can produce each new token without recomputing attention over the full prompt.; For ordinary full-attention layers, stored key/value tensors grow with retained tokens. Sliding-window attention can cap that growth; recurrent and hybrid models require architecture-specific accounting. Cache precision is independent of weight quantization, and cache size alone does not explain every latency or memory problem.

Quantization & integrity

Q4_K_M: Q4_K_M is a llama.cpp / GGUF quantization scheme that stores most weights at 4 bits with k-quant blocks and selectively keeps important tensors at higher precision; it is a common balance of size and quality for local LLM inference.
Q4_K_M context: Q4_K_M describes a model-weight quantization format, not a context limit or KV-cache precision. Memory fit also depends on the exact weight file, attention architecture, cache type, retained tokens, concurrent sequences and runtime buffers; there is no universal KV-memory allowance for every 70B model.
model digest: the model digest is the SHA-256 prefix of the weights file used for a benchmark, recorded so that two 'Llama 3.3 70B Q4_K_M' results from different sources are not blended if one used a different quantization pipeline.
EdDSA — Edwards-curve Digital Signature Algorithm: EdDSA, the Edwards-curve Digital Signature Algorithm, is the digital-signature scheme llm-speed uses (specifically Ed25519) to sign every benchmark upload so the bytes cannot be tampered with after signing.
JWS — JSON Web Signature (RFC 7515): JWS, JSON Web Signature defined by RFC 7515, is the compact-token format llm-speed wraps each upload in; the public key rides in the protected header so any third party can verify the signature without the server.
fingerprint: the llm-speed hardware fingerprint is a SHA-256 hash over bucketed hardware fields (CPU model, core count, RAM rounded to 8 GB, GPU name, VRAM rounded to 8 GB, OS major version) that identifies a hardware class, not a device.; Two physically different machines with the same SoC, RAM bucket, and major OS produce the same fingerprint hash. The hash is omitted entirely in --strict-anon mode.
fingerprint hash: the fingerprint hash is the 16-hex-character prefix of the llm-speed hardware fingerprint, used server-side to group runs of the same hardware class for outlier detection.

Backends, hosts & CLI flags

MLX: MLX is Apple's Metal-native array framework for machine learning on Apple Silicon; llm-speed measures it via the mlx-lm package and reports it under the backend label 'mlx'.
llama.cpp: llama.cpp is a C/C++ runtime for transformer inference with first-class Metal, CUDA, ROCm, and CPU backends; llm-speed measures it via the llama-server / llama-cli binary on PATH and reports the build version.
vLLM: vLLM is a high-throughput inference server with PagedAttention and continuous batching, primarily targeting CUDA datacenter accelerators; llm-speed measures it via its OpenAI-compatible HTTP server or the vllm Python package.
ollama: Ollama is a model server with a registry and a local API, normally on port 11434. Its serving paths include llama.cpp and MLX; record the version and selected runner rather than assuming every Ollama result uses the same engine.
exllamav2: exllamav2 is a CUDA-only inference library specialized for fast decoding of GPTQ/EXL2-quantized weights on consumer NVIDIA GPUs; llm-speed measures it via the exllamav2 Python package.
hosted-api: hosted-api is the CLI backend label for supported remote inference endpoints. It requires explicit opt-in through LLMSPEED_ENABLE_HOSTED_API=1. The current public leaderboard rejects hosted-api uploads; a hosted result measures the remote serving stack, not the local client hardware.
--strict-anon: --strict-anon uses a fresh signing key, omits the fingerprint hash and suppresses upload authentication and the llm-speed-specific identification headers. This reduces application-level linkage; it does not hide the connection's IP address or guarantee that submissions cannot be correlated.
--anon: --anon is the llm-speed CLI mode that sends an X-LLM-Speed-Anon: 1 header instructing the server not to associate the upload with any account, while keeping the persistent keypair and fingerprint hash for outlier-cluster grouping.
--no-upload: --no-upload prevents benchmark-result submission and saves the result locally. It is not a network-isolation setting: model downloads or the selected backend can still use the network. A saved result can be submitted later with an explicit resume command.
outlier flag: an outlier flag is a server-side marker on a benchmark result whose decode tok/s falls more than 3 sigma from the mean of its (model x hardware x backend) cluster; flagged runs remain visible and may be challenged in a public dispute thread.
run badge: the run badge is an embeddable SVG image at llm-speed.com/badge/<run_id>.svg that renders a single run's headline decode tokens-per-second, model, and hardware in one image; intended for README files, blog posts, and hardware-review embeds. Added 2026-04-29.; Markdown embed: [![llm-speed](https://llm-speed.com/badge/r_y2_5y8oo97d.svg)](https://llm-speed.com/r/r_y2_5y8oo97d). The SVG is served with cache-control max-age=14400 and stale-while-revalidate, so the embed reflects the canonical run page without re-fetching on every view. Click-through resolves to the run permalink, which is the verifiable source of the number.

Models & hardware

Qwen3-Coder-Next: Qwen3-Coder-Next is Alibaba Cloud's coding-tuned large language model in the Qwen3 family, succeeding Qwen2.5-Coder; on llm-speed, Qwen3-Coder-Next runs are aggregated at llm-speed.com/m/qwen3-coder-next with hardware-side decode tok/s, prefill tok/s, and TTFT measured under suite-v1.; For Qwen3-Coder-Next coding-quality scores (HumanEval, MBPP, SWE-Bench Verified, LiveCodeBench), see the Qwen team's GitHub at github.com/QwenLM and the model card on Hugging Face. llm-speed publishes wall-clock speed only.
Llama 3.3 70B: Llama 3.3 70B is Meta's 70-billion-parameter instruction-tuned language model in the Llama 3 family; llm-speed aggregates Llama 3.3 70B runs at llm-speed.com/m/llama-3-3-70b-instruct, sorted by decode tokens per second across backends and hardware.; Check the exact weight file plus cache and runtime allocations. Shortening context reduces cache demand but cannot make weights larger than available VRAM fit entirely on that GPU; CPU offload or a different weight quantization changes the configuration being measured.
Qwen2.5-7B-Instruct: Qwen2.5-7B-Instruct is Alibaba Cloud's 7-billion-parameter instruction-tuned model in the Qwen2.5 family; llm-speed aggregates Qwen2.5-7B-Instruct runs at llm-speed.com/m/qwen2-5-7b-instruct.; At Q4_K_M the model is roughly 4 GB on disk, fits comfortably on Apple Silicon (M-series Pro / Max / Ultra) and any consumer GPU with 8 GB or more of VRAM, and is a common drop-in for local coding assistants alongside Qwen2.5-Coder-7B.
Llama-4-Scout: Llama-4-Scout is Meta's mixture-of-experts model in the Llama 4 family, roughly 109B total parameters with 17B active per token; on llm-speed, local-hardware Llama-4-Scout runs are aggregated at llm-speed.com/m/llama-4-scout.; Llama-4-Scout fits on M3 Ultra at 4-bit but overflows RTX 5090 32 GB VRAM at full context. To produce a measured number, run "llm-speed bench --models meta-llama/Llama-4-Scout --backends mlx,llama.cpp" on M3 Ultra.
Llama-4-Maverick: Llama-4-Maverick is Meta's larger mixture-of-experts model in the Llama 4 family; on llm-speed, local-hardware Llama-4-Maverick runs are aggregated at llm-speed.com/m/llama-4-maverick.
stable-code-instruct-3b: stable-code-instruct-3b is Stability AI's 3-billion-parameter coding-instruct model; aggregated on llm-speed at llm-speed.com/m/stable-code-instruct-3b.; Historical chat-short submissions received April 28, 2026 report 192.5 decode tok/s on M3 Ultra (r_y2_5y8oo97d) and 19.4 on M3 Pro (r_pqjsvd-cub4). These are separate submitted measurements, not a controlled hardware scaling study or a current model ranking.
Qwen3-Coder-30B-A3B-Instruct: Qwen3-Coder-30B-A3B-Instruct is Alibaba Cloud's 30-billion-parameter active-3B mixture-of-experts coding-tuned model in the Qwen3 family; aggregated on llm-speed at llm-speed.com/m/qwen3-coder-30b-a3b-instruct.; A historical chat-short submission received April 28, 2026 reports 112.2 decode tok/s, 204.0 prefill tok/s and 539.3 ms TTFT via MLX 0.31.3 on M3 Ultra (r_fpsca03u2o_). One throughput result does not establish coding quality or a current speed ranking.
gpt-oss-20b: gpt-oss-20b is OpenAI's open-weights 20-billion-parameter model in MXFP4-Q4 quantization; aggregated on llm-speed at llm-speed.com/m/gpt-oss-20b.; A historical chat-short submission received April 28, 2026 reports 152.7 decode tok/s, 239.9 prefill tok/s and 692.1 ms TTFT via MLX 0.31.3 (r_3ijun8ltjnb). The submitted model name includes MXFP4-Q4; that label alone does not verify the runtime kernel or a current ranking.
DeepSeek-Coder-V2-Lite-Instruct: DeepSeek-Coder-V2-Lite-Instruct is DeepSeek's coding-tuned mixture-of-experts model; aggregated on llm-speed at llm-speed.com/m/deepseek-coder-v2-lite-instruct.; A historical chat-short submission received April 28, 2026 reports 168.3 decode tok/s, 291.5 prefill tok/s and 449.5 ms TTFT via MLX 0.31.3 (r_l_v1-zq_qaz). This is a dated example, not a current leaderboard position.
Qwen2.5-Coder-32B-Instruct-4bit: Qwen2.5-Coder-32B-Instruct-4bit is Alibaba Cloud's 32-billion-parameter dense coding-tuned model at 4-bit; aggregated on llm-speed at llm-speed.com/m/qwen2-5-coder-32b-instruct.; A historical chat-short submission received April 28, 2026 reports 34.5 decode tok/s, 144.1 prefill tok/s and 909.4 ms TTFT via MLX 0.31.3 (r_721b4bls_oq). Comparing it with a different MoE model does not isolate architecture or establish equivalent output quality.
M3 Ultra: M3 Ultra is Apple's highest-end SoC in the M3 family, configured by fusing two M3 Max dies; it ships with up to 80 GPU cores, up to 512 GB of unified memory, and roughly 800-900 GB/s of memory bandwidth, with model capacity depending on installed memory, the weight artifact, context and runtime overhead.; On llm-speed, M3 Ultra runs are aggregated at llm-speed.com/hw/m3-ultra; both MLX and llama.cpp backends are supported and sorted side-by-side in the cheatsheet.
M3 Pro: M3 Pro is Apple's mid-range SoC in the M3 family, with 14- or 18-core GPU configurations, 18 GB or 36 GB of unified memory, and roughly 150 GB/s of memory bandwidth; it runs 7B-class models comfortably at 4-bit and is the most common machine in the current llm-speed dataset.; On llm-speed, M3 Pro runs are aggregated at llm-speed.com/hw/m3-pro; the 18-core / 36 GB SKU and the 14-core / 18 GB SKU are reported with the same accelerator_summary granularity that the cheatsheet uses for cross-rig comparison.
RTX 5090: RTX 5090 is NVIDIA's flagship consumer GPU in the Blackwell generation, with 32 GB of GDDR7 VRAM on a 512-bit bus delivering roughly 1,792 GB/s of memory bandwidth; on llm-speed, RTX 5090 runs are aggregated at llm-speed.com/hw/rtx-5090.; Peak hardware bandwidth is not a measured LLM speedup. Compare matched model bytes, runtime, context, GPU placement and workload before attributing a throughput difference to the GPU.
RTX 4090: RTX 4090 is NVIDIA's flagship consumer GPU in the Ada Lovelace generation, with 24 GB of GDDR6X VRAM and roughly 1,008 GB/s of memory bandwidth; on llm-speed, RTX 4090 runs are aggregated at llm-speed.com/hw/rtx-4090.

Cite a specific run as llm-speed.com/r/<id>, a per-model page as llm-speed.com/m/<slug>, and a per-hardware page as llm-speed.com/hw/<slug>.