What's the fastest GPU for running Llama 3.1 8B locally?

Llama 3.1 8B is one of the most-run local models: small enough for almost any GPU, capable enough for everyday chat and coding. So which hardware runs it fastest? We measured its decode speed on each, signed and reproducible; every number links to the run that produced it.

Llama 3.1 8B, ranked by decode tok/s

RTX 5090: 232 tok/s.
RTX 4090: 154 tok/s.
RTX 3090: 136 tok/s.
Apple M3 Pro: 29 tok/s.

The RTX 5090 is fastest by a wide margin at 232 tok/s, roughly 8x reading speed. Full standings across every rig are on the cheatsheet.

Why the spread is this wide

Decoding is memory-bandwidth-bound: each token streams the model's active weights out of VRAM, so throughput tracks bandwidth more than raw compute, and that is most of the story here. The RTX 5090 moves roughly 1.8 TB/s, the RTX 4090 about 1 TB/s, the RTX 3090 about 0.94 TB/s, and an M3 Pro only around 150 GB/s. Line those up against the measured 232, 154, 136, and 29 tok/s and the ranking falls out almost exactly. It also means a faster card mostly buys quicker decode, not the ability to run a bigger model. More on the two speed numbers in decode vs prefill, with definitions on the glossary.

Which should you pick?

All of these run Llama 3.1 8B comfortably at Q4 (the weights need only about 5 GB). For maximum speed, the RTX 5090. For the best value, the RTX 3090 already beats most hosted APIs at 136 tok/s for a fraction of a 5090's price; the RTX 4090 sits in between. A laptop-class Apple chip (M3 Pro) runs it too, just far slower; a desktop-class M-series (Max or Ultra) with more GPU cores is considerably quicker than the Pro shown here.

Reproduce it

Every number is one command from reproduction:

$ pipx install llm-speed && llm-speed bench

Numbers as of July 2026; the linked runs and the cheatsheet always reflect current data.