How much VRAM do you need to run an LLM locally?

By

How to estimate the VRAM an LLM needs: parameter count times quantization bits, plus KV cache that grows with context length, plus overhead.

~~~

You want to run an LLM on your own machine. The first question is always the same: will it fit in memory?

The answer comes down to simple math. Parameter count, quantization, and context length. Once you know how these three combine, you can look at any model and know if your GPU can run it.

Let’s work through it.

The weights

An LLM is mostly a big pile of numbers, the weights. Each parameter is one number, and how much space it takes depends on the precision you store it at.

At full precision (FP16), each parameter takes 16 bits, which is 2 bytes. So the formula is:

weights in GB = parameters (in billions) × bits ÷ 8

An 8B model at FP16:

8 × 16 ÷ 8 = 16 GB

16 GB just for the weights. That doesn’t fit on most consumer GPUs. This is why almost nobody runs local models at full precision.

Quantization

Quantization stores each weight with fewer bits. Instead of 16, you use 8, or about 4. You lose some quality, but the model shrinks a lot.

Same 8B model at different precisions:

FP16: 8 × 16 ÷ 8 = 16 GB
Q8:   8 × 8 ÷ 8  = 8 GB
Q4:   8 × 4 ÷ 8  = 4 GB

Q8 often stays close to FP16 quality. Q4 variants such as Q4_K_M are a common sweet spot for local runs. Real quantized files also contain scales and metadata, so they use a little more than the theoretical number above.

My advice: start with Q4 and only go up if you see quality problems.

The KV cache

Weights are not the whole story. While the model generates text, it keeps a KV cache in memory: intermediate values for every token in the context, so it doesn’t recompute them on each new token.

The KV cache grows with context length. Longer conversations, bigger documents, more memory.

The exact size depends on the model architecture. Parameter count alone is not enough:

KV bytes =
  2 × layers × KV heads × head dimension
  × context tokens × bytes per KV value

The 2 accounts for keys and values. Grouped-query attention can give a large model relatively few KV heads.

For a Llama 3-family 8B model with 32 layers, 8 KV heads, a head dimension of 128, and FP16 KV values:

2 × 32 × 8 × 128 × 1000 × 2
  ≈ 131 MB per 1k tokens

At 8,000 tokens of context:

8 × 131 MB ≈ 1.05 GB

Weight quantization does not automatically quantize the KV cache. Some runtimes offer separate KV-cache quantization, but FP16 is a safer default for an estimate.

A Llama 3-family 70B model has 80 layers but still only 8 KV heads:

2 × 80 × 8 × 128 × 1000 × 2
  ≈ 328 MB per 1k tokens

At 8k context that is about 2.62 GB. Context still matters, but architecture matters more than a parameter-count shortcut suggests.

Overhead

The runtime (llama.cpp, Ollama, MLX) adds its own overhead on top: buffers, compute scratch space, framework bookkeeping. A 1.1× multiplier on the weights is a reasonable estimate.

Putting it all together for an 8B model at Q4 with 8k context:

weights:  8 × 4 ÷ 8 × 1.1 = 4.4 GB
KV cache: 8k × 131 MB/1k  ≈ 1.05 GB
total:                       5.45 GB

That fits comfortably on a 12 GB card.

I built a small calculator that does this math for any model size, quantization, and context length, and checks the result against common GPUs and Macs: the LLM VRAM calculator.

What fits where

One more thing before matching models to hardware: you never get 100% of the memory.

On a discrete GPU, the driver and display eat some VRAM. Count on about 90% usable. On Apple Silicon, the memory is unified: the GPU shares it with macOS and your apps, so about 70–75% is realistically available for the model.

With that in mind:

Let’s verify that 70B case with the formula. Weights at Q4:

70 × 4 ÷ 8 × 1.1 = 38.5 GB

Add 8k of context (about 2.62 GB of KV cache for this architecture) and you are at roughly 41.1 GB before any extra runtime allocations not covered by the weight multiplier. That can fit inside a 46 GB budget, but not with unlimited context or much room for other workloads. This is why “it fits” depends on both the architecture and the context you actually use.

The short version

Three numbers decide everything:

  1. Parameters × bits ÷ 8 gives you the weights.
  2. Layers, KV heads, head size, KV precision, and context length determine the KV cache.
  3. Multiply weights by ~1.1 for runtime overhead, and remember you only get 90% of a GPU (or ~72% of a Mac’s unified memory).

And if a model doesn’t fit and you fall back to an API, keep the provider swappable in your code. I wrote about how in an LLM adapter pattern for Cloudflare Workers. This calculator is part of a bigger set, too: I recently shipped 90 free browser tools for developers, all running client-side.

Tagged: AI · All topics
~~~

Related posts about ai: