Calculators · AI Rigs
Will it fit? VRAM calculator for local LLMs
On one of my 48GB-modded 4090s, Llama 3.3 70B loads with room to spare, then runs out of memory at about 21,000 tokens of context. The weights were never the problem. The KV cache was, and it's the number no spec sheet prints.
This VRAM calculator does the sum for you: pick your card, how many of them, a model and the context you want. The weights come from models I've measured on my own bench; the KV cache is worked out from the model's architecture.
Your setup
The model
The KV cache needs three numbers from the model's config.json. Read them rather than guessing.
The weights are this model's measured figure, but layers, KV heads and head dim are still the Llama 3.3 70B defaults. Swap in this model's own numbers from its config.json for a true KV figure.
Will it fit?
- Weights
- …
- KV cache at this context
- …
- Reserve (activations, CUDA graphs)
- …
- Total needed
- …
- Usable VRAM (at 0.92 utilisation)
- …
Longest context that fits on this setup: …
The 12 models I've measured, with the context each one served
| Model | Weights GB | Served context | Cards | KV |
|---|---|---|---|---|
| LFM2.5-1.2B-Instruct | 2.3 | 33k | 1 | fp8 |
| GLM-4.7-Flash-GPTQ | 16.6 | 33k | 1 | auto |
| Ternary-Bonsai-27B-AWQ | 18.7 | 33k | 1 | fp8 |
| Qwen3.8-27B-AWQ | 27.7 | 33k | 1 | fp8 |
| Qwen3.8-27B-FP8 | 30.9 | 33k | 1 | fp8 |
| Devstral-Small-2-24B-AWQ | 32.1 | 33k | 1 | fp8 |
| Nemotron-3.5-Lightning-30B-FP8 | 32.3 | 33k | 1 | fp8 |
| Llama-3.3-70B-AWQ | 39.8 | 21k | 1 | fp8 |
| Llama-4-Scout-INT4 | 64.9 | 33k | 2 | fp8 |
| gpt-oss-120b-official | 65.2 | 131k | 2 | fp8 |
| Llama-3.3-70B-FP8 | 72.7 | 33k | 2 | fp8 |
| NVIDIA-Nemotron-3-Super-120B-A12B-AWQ-4bit | 80.7 | 33k | 2 | fp8 |
Weights are measured on my own bench: two RTX 4090s modded to 48GB running vLLM 0.26.0. The KV cache is worked out from the architecture you enter, so it's only as right as those three numbers. A 1GB reserve and the 0.92 utilisation I serve at are both applied. Unified-memory machines (Mac Studio, DGX Spark, Strix Halo) share that memory with the system, so treat the answer as a rough guide there.
Questions people ask
How much VRAM does a model need?+
The weights, plus the KV cache for every token of context, plus a small reserve. The weights are fixed by the model and its quantisation; the KV cache grows with context, which is why a model that loads fine can still run out of memory once a conversation gets long.
Will a 70B model fit on a 24GB card?+
Not on its own. At 4-bit, Llama 3.3 70B is about 40GB of weights before any context, so it needs two 24GB cards, a 48GB card, or offloading to system RAM, which is much slower.
Why does context length matter so much?+
Every token of context needs its own slice of KV cache. On Llama 3.3 70B that is about 0.16MB per token at fp8, so 131k tokens of context needs around 21GB on top of the weights.
Choosing a card? The GPU comparison ranks them on VRAM and bandwidth. The full benchmark behind these numbers is on Houtini.