Llama 3.3 70B on a GPU: Speed and VRAM Fit
At 4-bit, Llama 3.3 70B needs ~48GB. Measured: two RTX 3090s ran it at 19.1 tok/s, an A40 at 13.0. Only IQ1/IQ2 quants fit 24GB. GGUF fit table for 24-80GB.
At 4-bit (Q4_K_M), Llama 3.3 70B needs about 48GB: two 24GB RTX 3090s decoded it at 19.1 tok/s (r_n93ukk0mkmm) and a single 48GB NVIDIA A40 at 13.0 tok/s (r_19f64yux97o), both measured with llama.cpp on chat-short. Each is one run on a rented host with a different llama.cpp build, so treat them as reference points, not a ranking. On one 24GB GPU, Q4_K_M doesn't fit; you'd need partial CPU offload. Only the smallest quants, IQ1_M, IQ2_XXS and IQ2_XS (15.6 to 19.7 GiB files), squeeze into 24GB at 8K context (estimate), at a large quality cost. The A40 run peaked at 42.1 of 45.0 GiB VRAM at 8K context; the dual-3090 run didn't record memory use. The table below shows which quants fit on 24, 32, 48 and 80GB cards. For rig options and the costs we've measured, see 70B rigs and costs.
Highest measured Llama-70B decode across 2 GPU setups: 2x RTX 3090 (24GB) at 19.14tok/s (llama.cpp, Q4_K_M; single runs on different hosts and builds). Apple Silicon runs use a different backend and 4-bit format, so they are not ranked here.
Meta's Llama 3.3 70B Instruct repository reports 70,553,706,496 BF16 parameters. At two bytes per parameter, the weights alone require about 141.1 GB (131.4 GiB), before KV cache, runtime buffers and other allocations. A single 80 GB H100 SXM cannot hold those BF16 weights entirely in GPU memory. Use a smaller quantized artifact, a supported multi-GPU split, or CPU offload; offload is a different performance configuration. A 192 GB MI300X has more memory than the raw weights, but usable capacity, runtime support, context length and concurrency still determine whether the workload fits. Two 24 GB GPUs provide 48 GB in aggregate only when the runtime can split the workload; this is not an automatic shared-memory pool or a guarantee that a particular 4-bit file plus context will fit. On a Mac, the installed unified-memory tier and memory available to the runtime matter too. Inspect the exact download and reserve headroom before buying or renting. Submitted Llama 3.1 and 3.3 70B speeds below are examples from different configurations, not a controlled hardware comparison.
Sources checked September 12, 2026: Meta model files · NVIDIA H100 specifications · AMD MI300X memory configuration.
Which Llama 3.3 70B quant fits your GPU?
Estimates, except the measured A40 row. Each total is the GGUF file size plus a context allowance for one 8K conversation: 2.5 GiB of FP16 KV cache (80 layers × 8 KV heads × 128 dimensions × 2 for key and value × 2 bytes × 8,192 tokens, from the model's config.json) and 1.0 GiB of runtime overhead. "Fits" means the total is at or under the card's assumed usable memory (24GB ≈ 23.5 GiB, 32GB ≈ 31.5 GiB, 48GB ≈ 45.0 GiB, 80GB ≈ 79.0 GiB), all layers on one GPU.
| Quant | GGUF file | Est. total at 8K | 24GB | 32GB | 48GB | 80GB |
|---|---|---|---|---|---|---|
| IQ1_M† | 15.6 GiB | 19.1 GiB | Fits (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| IQ2_XXS† | 17.8 GiB | 21.3 GiB | Fits (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| IQ2_XS† | 19.7 GiB | 23.2 GiB | Fits (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| IQ2_S† | 20.7 GiB | 24.2 GiB | Doesn't fit (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| IQ2_M† | 22.5 GiB | 26.0 GiB | Doesn't fit (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| Q2_K | 24.6 GiB | 28.1 GiB | Doesn't fit (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| Q2_K_L | 25.5 GiB | 29.0 GiB | Doesn't fit (est.) | Fits (est.) | Fits (est.) | Fits (est.) |
| Q3_K_M | 31.9 GiB | 35.4 GiB | Doesn't fit (est.) | Doesn't fit (est.) | Fits (est.) | Fits (est.) |
| Q4_K_M | 39.6 GiB | 43.1 GiB | Doesn't fit (est.) | Doesn't fit (est.) | Fits, measured: 42.1 of 45.0 GiB on an A40 (r_19f64yux97o) | Fits (est.) |
| Q5_K_M | 46.5 GiB (2 parts) | 50.0 GiB | Doesn't fit (est.) | Doesn't fit (est.) | Doesn't fit (est.) | Fits (est.) |
| Q6_K | 53.9 GiB (2 parts) | 57.4 GiB | Doesn't fit (est.) | Doesn't fit (est.) | Doesn't fit (est.) | Fits (est.) |
| Q8_0 | 69.8 GiB (2 parts) | 73.3 GiB | Doesn't fit (est.) | Doesn't fit (est.) | Doesn't fit (est.) | Fits (est.) |
† IQ1/IQ2-class quants: large quality loss, unmeasured. They squeeze the model into less memory by cutting precision well below 4-bit; we haven't measured their speed or answer quality. At 8K, IQ1_M, IQ2_XXS and IQ2_XS fit a 24GB card on this estimate (IQ2_XS with only about 0.3 GiB to spare).
Sources: GGUF file sizes from bartowski/Llama-3.3-70B-Instruct-GGUF (commit b6c5c9f, checked October 17, 2026); model shape from unsloth/Llama-3.3-70B-Instruct config.json (a public copy of Meta's gated repo, checked the same day). Longer context, parallel requests or other loaded models need more memory.
Two 24GB cards (measured once): two RTX 3090s ran Q4_K_M split across both cards at 19.1 tok/s (llama.cpp, chat-short, r_n93ukk0mkmm). A 2×24GB split gives about 47 GiB in total, but each card needs its own buffers; that run didn't record memory use, so the table above stays single-card.
CPU offload (unmeasured): with llama.cpp you can keep some layers in system RAM so a quant runs on a smaller GPU. It will be slower than GPU-only; we haven't measured how much.
Weighing cost and other machines? See 70B rigs and the one cost we've measured, or estimate memory for your own model and context.
Submitted benchmarks
| Hardware | Model | decode tok/s | Run |
|---|---|---|---|
| 2x RTX 3090 (24GB) | Llama-3.3-70B-Instruct | 19.14tok/s | r_n93ukk0mkmm |
| A40 (45GB) | Llama-3.3-70B-Instruct | 12.99tok/s | r_19f64yux97o |
See also: All hardware · All models · Methodology