ToolSite LLM VRAM Calculator
Pick a model or load any one from Hugging Face, choose a quantization, context length and number of parallel requests, and see how much VRAM inference needs and which GPUs it fits on. Everything runs in your browser.
Advanced: the model numbers and overhead used
Editing any of these switches the model to Custom.
– estimated GPU memory
- Weights
- –
- KV cache
- –
- Runtime overhead
- –
- KV cache per token
- –
Which GPUs can run it
| GPU | Memory | Result |
|---|---|---|
| RTX 3060 | 12 GB | – |
| RTX 4060 Ti 16GB | 16 GB | – |
| RTX 3090 / 4090 | 24 GB | – |
| RTX 5090 | 32 GB | – |
| A100 40GB | 40 GB | – |
| Mac, 64 GB unified memory about 75% of it is usable by the GPU by default | 48 GB | – |
| L40S / RTX 6000 Ada | 48 GB | – |
| A100 / H100 80GB | 80 GB | – |
| Mac, 128 GB unified memory about 75% of it is usable by the GPU by default | 96 GB | – |
| H200 | 141 GB | – |
| B200 | 180 GB | – |
How to use
- Choose a model, or type a Hugging Face id such as Qwen/Qwen3-8B and press Load.
- Pick the weight precision you plan to run, for example Q4_K_M for llama.cpp or FP8 for vLLM.
- Set the context length and how many requests run at the same time.
- Read the total and the GPU table. Open Advanced to see or change every number behind the estimate.
FAQ
How is the estimate calculated?
Weights = parameters × bits per weight ÷ 8. KV cache = 2 × layers × KV heads × head dimension × bytes per value × tokens × requests. Runtime overhead = 0.5 GB for the CUDA context plus 10% of weights and cache for activations and fragmentation; you can change the 10% under Advanced.
Why does context length matter so much?
The KV cache grows with every token of every request. Llama 3.1 8B needs 128 KB of FP16 cache per token, so one 128K-token conversation adds 16 GB on top of the weights. An FP8 cache halves that.
Do MoE models like Qwen3-30B-A3B need less memory?
They compute with a few experts per token, but every expert has to be loaded, so weight memory follows the total parameter count: 30.5 billion for Qwen3-30B-A3B, not 3 billion.
Does quantizing the weights shrink the KV cache?
No. Weight precision and KV cache precision are separate settings, so Q4 weights with an FP16 cache still need the full-size cache.
How are DeepSeek and gpt-oss handled?
DeepSeek V3 uses multi-head latent attention and caches 576 values per token per layer instead of full keys and values. gpt-oss keeps only the last 128 tokens in half of its layers. Both come from the model’s config.json, as do other sliding-window and linear-attention layers when the config lists them.
What does "As published" mean?
It uses the precisions actually stored in the model files on Hugging Face, such as FP8 for DeepSeek V3 and MXFP4 for gpt-oss, so the weight size matches the download.
How accurate is it?
It is an estimate. Inference engines add their own buffers: vLLM reserves a fixed share of GPU memory up front, and llama.cpp allocates the cache for the full context when it starts. Leave some headroom, especially on a GPU that also drives your display.
Is anything I enter sent anywhere?
The calculation runs in your browser. The only network request is to Hugging Face, when you load a model.
Updated