Hardware FAQ
RTX 4090 — frequently asked questions
Direct answers to the questions local-LLM enthusiasts ask about RTX 4090, drawn from 31 signed runs on llm-speed. Every numerical claim links to a verifiable run permalink at /r/<id>.
What's the fastest LLM on RTX 4090?
The fastest measured LLM on RTX 4090 on llm-speed is gpt-oss-20b at 242.6 decode tok/s on llama.cpp 0.6.0-dev (build 1, commit df18bee) (workload chat-short, run r_t-a9v_ipdit). Cite as https://llm-speed.com/r/r_t-a9v_ipdit.
This is the headline decode tokens-per-second across every (model, backend) pairing submitted on RTX 4090; faster results may exist on hardware not yet benchmarked, but among signed runs this is the published top.
Can I run a 7B-class model on RTX 4090?
Yes — llm-speed has measured Nemotron-3.5-Bolt-74B-A9B-24GB at 107.3 decode tok/s on llama.cpp 0.7.0-dev (build 11669, commit 620d2225c) (workload chat-short, run r_vkjsinuhlyl) on RTX 4090, confirming the 7-9B parameter range fits and runs end-to-end on this hardware. Cite as https://llm-speed.com/r/r_vkjsinuhlyl.
7B-class models on RTX 4090 typically use the published quant scheme on the run page; check the run for the exact backend version and model digest.
Can I run a 30B-class model on RTX 4090?
Yes — llm-speed has measured Qwen3-Coder-30B-A3B-Instruct at 232.5 decode tok/s on llama.cpp 0.6.0-dev (build 1, commit df18bee) (workload chat-short, run r_1k8942tw3zf) on RTX 4090, confirming the 27-33B parameter range, including MoE 30B-A3B variants fits and runs end-to-end on this hardware. Cite as https://llm-speed.com/r/r_1k8942tw3zf.
30B-class models on RTX 4090 typically use Q4_K_M; check the run for the exact backend version and model digest.
Can I run a 70B-class model on RTX 4090?
Not at native precision: RTX 4090 has roughly 24 GB of VRAM/unified memory and a 70B-class model (70-72B parameter range) needs at least 48 GB even at 4-bit quantization, so a 70B-class model won't fit end-to-end on RTX 4090 as a single-GPU/SoC config.
For workloads that do fit, see the RTX 4090 leaderboard for the largest model successfully measured on this hardware.
How does RTX 4090 compare to RTX 5090 for local LLM inference?
For a head-to-head between RTX 4090 and RTX 5090, see the side-by-side comparison page at https://llm-speed.com/vs/rtx-4090-vs-rtx-5090, which lays out every (model, backend) pair where both rigs have a signed run.
As a single-rig anchor, RTX 4090 tops out at 242.6 decode tok/s on gpt-oss-20b via llama.cpp (run r_t-a9v_ipdit); the RTX 5090 top number is on its own /hw/<slug> page so the comparison stays grounded in measured numbers, not extrapolation.
RTX 4090 vs RTX 5090 (side-by-side) · RTX 5090 leaderboard · RTX 4090 leaderboard
Which backend is fastest on RTX 4090?
Of the backends with a signed local run on RTX 4090, llama.cpp leads at 242.6 decode tok/s on gpt-oss-20b (run r_t-a9v_ipdit), followed by ollama at 195.0 decode tok/s on gemma3 (run r_dlanfbgym0h).
These rows are not a same-model head-to-head — they're each backend's best result across any model on RTX 4090. For a true llama.cpp-vs-ollama comparison, run "llm-speed bench --backends llama.cpp,ollama --models <model>" on a RTX 4090 machine.
llama.cpp: run r_t-a9v_ipdit · ollama: run r_dlanfbgym0h · RTX 4090 leaderboard
How much VRAM do I need for local LLM inference on RTX 4090?
RTX 4090 has configurations with up to 24 GB of VRAM, which sets the ceiling on what fits: a 7B-class model at 4-bit needs roughly 6 GB, a 30B-class model at 4-bit needs roughly 18 GB, and a 70B-class model at 4-bit needs roughly 38 GB.
Long-context workloads add KV-cache pressure on top of the weights — a 32k-token context on a 70B-class model adds another 8-16 GB depending on attention layout, so the practical fit is tighter than the weight-only number.
For exact "did it actually run" evidence on RTX 4090, the leaderboard at /hw/rtx-4090 lists every model that has produced a signed result on this hardware.
Is RTX 4090 worth it for local LLM inference in 2026?
On signed data, RTX 4090 delivers up to 242.6 decode tok/s on gpt-oss-20b via llama.cpp (run r_t-a9v_ipdit), which puts it comfortably above interactive-coding thresholds (~80 tok/s) for the published top configuration.
"Worth it" depends on your model class: RTX 4090 is most useful for 7B-30B class local models with comfortable headroom.
For "what fits and how fast", the per-model rows on /hw/rtx-4090 are the honest answer; for cross-rig comparisons, see /vs/rtx-4090-vs-rtx-5090.
What quantization should I use on RTX 4090?
On RTX 4090, the top decode rates by quant are Q4_K_M at 242.6 tok/s (gpt-oss-20b, run r_t-a9v_ipdit) and MXFP4 at 141.7 tok/s (gpt-oss, run r_iu2sfa9ykvw); pick the smallest quant whose quality your task tolerates, since on memory-bandwidth-bound hardware decode speed scales roughly inversely with bytes-per-weight.
Quant choice is a quality-vs-speed tradeoff that this hardware FAQ does not arbitrate; llm-speed publishes hardware-side speed, not output quality. For quality scores, see the model card on Hugging Face and the LMSYS Chatbot Arena.
Q4_K_M: 242.6 tok/s (run r_t-a9v_ipdit) · MXFP4: 141.7 tok/s (run r_iu2sfa9ykvw) · Q4_0: 97.7 tok/s (run r_jfh0qkngw76)