Best GPU for local LLMs
A curated view of this site's own GPU and VRAM calculator data, not a new dataset. VRAM is the constraint that actually matters for local LLM inference — not raw TFLOPS — because a model that doesn't fit in VRAM simply won't load. Memory bandwidth is the second-order factor: it's usually the real ceiling on tokens/second once a model fits.
By VRAM budget
"Fits" below means the single largest current-generation model in this site's VRAM calculator collection that comes in under each card's VRAM at INT4 quantization and 4K context — the same formula the calculator itself uses, not a separate rule of thumb. Check a specific model + context length combination on that page directly; this table is a starting point, not a substitute.
The 48GB and 96GB workstation cards show the same "largest fit" above — that's a gap in this site's tracked model list, not a data error: the next model up from Qwen2.5 72B is Llama 3.1 405B, which needs well over 96GB even at INT4, so nothing in between currently separates those two cards here.
Picks by use case
Best value entry point: Radeon RX 9070 XT or Intel Arc B580
Under $600, both give real headroom over the 8GB cards at the bottom of the table above — the RX 9070 XT (16GB, $599) is the newer of the two and got unusually fast official ROCm support for a just-launched card; the Arc B580 (12GB, $249) is the cheapest way into double-digit VRAM at all, though Intel's software stack for local LLM tools is less mature than CUDA or ROCm — expect more setup friction.
The sweet spot: RTX 3090 (24GB)
RTX 3090's 24GB and full CUDA support (plus the biggest secondhand market of any card on this page, since NVIDIA no longer sells it new) make it the card most local-LLM hobbyists actually land on — MSRP shown above is the original 2020 launch price, not current secondhand pricing, which this site doesn't track. The RX 7900 XTX matches its 24GB at a similar or lower price if you're committed to ROCm; see ROCm on consumer GPUs for what that trade-off actually involves. The RTX 4090 has more compute and bandwidth than either at the same 24GB, if budget allows the premium.
No compromises: RTX PRO 6000 Blackwell (96GB) or RTX 6000 Ada (48GB)
Both are workstation cards, not consumer GPUs, and priced accordingly (unofficial street pricing for the RTX PRO 6000 Blackwell has swung well above its original launch price — see that page's notes). What they buy: enough VRAM to run this site's largest tracked models directly rather than working around VRAM limits with aggressive quantization or a multi-GPU split (which this table doesn't model at all — every row above assumes a single card).