# How much VRAM do you need to run an LLM locally?

> How to estimate the VRAM an LLM needs: parameter count times quantization bits, plus KV cache that grows with context length, plus overhead.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-07-26 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/llm-vram-requirements/

You want to run an LLM on your own machine. The first question is always the same: will it fit in memory?

The answer comes down to simple math. Parameter count, quantization, and context length. Once you know how these three combine, you can look at any model and know if your GPU can run it.

Let's work through it.

## The weights

An LLM is mostly a big pile of numbers, the **weights**. Each parameter is one number, and how much space it takes depends on the precision you store it at.

At full precision (FP16), each parameter takes 16 bits, which is 2 bytes. So the formula is:

```
weights in GB = parameters (in billions) × bits ÷ 8
```

An 8B model at FP16:

```
8 × 16 ÷ 8 = 16 GB
```

16 GB just for the weights. That doesn't fit on most consumer GPUs. This is why almost nobody runs local models at full precision.

## Quantization

**Quantization** stores each weight with fewer bits. Instead of 16, you use 8, or 4. You lose a little quality, but the model shrinks a lot.

Same 8B model at different precisions:

```
FP16: 8 × 16 ÷ 8 = 16 GB
Q8:   8 × 8 ÷ 8  = 8 GB
Q4:   8 × 4 ÷ 8  = 4 GB
```

Q8 is nearly indistinguishable from FP16 in practice. Q4 (the Q4_K_M variant you see everywhere) is the usual sweet spot for local runs: 4× smaller than FP16, with a quality drop most people don't notice. Below that, Q3 starts losing noticeably on hard tasks, and Q2 is often incoherent.

My advice: start with Q4 and only go up if you see quality problems.

## The KV cache

Weights are not the whole story. While the model generates text, it keeps a **KV cache** in memory: intermediate values for every token in the context, so it doesn't recompute them on each new token.

The KV cache grows with context length. Longer conversations, bigger documents, more memory.

The exact size depends on the model architecture, but a useful rule of thumb is about 75 MB per 1,000 context tokens per billion parameters (at FP16-equivalent precision, scaled down by quantization).

For an 8B model at Q4 (so scaled by 4/16), that's:

```
8 × 75 × (4/16) = 150 MB per 1k tokens
```

At 8,000 tokens of context:

```
8 × 150 MB = 1,200 MB ≈ 1.2 GB
```

Not huge for a small model. But watch what happens with a 70B model at Q4:

```
70 × 75 × (4/16) = 1,312 MB per 1k tokens
```

At 8k context that's over 10 GB, just for the cache. Context length is the hidden VRAM cost people forget about.

## Overhead

The runtime (llama.cpp, Ollama, MLX) adds its own overhead on top: buffers, compute scratch space, framework bookkeeping. A 1.1× multiplier on the weights is a reasonable estimate.

Putting it all together for an 8B model at Q4 with 8k context:

```
weights:  8 × 4 ÷ 8 × 1.1 = 4.4 GB
KV cache: 8k × 150 MB/1k  = 1.2 GB
total:                      5.6 GB
```

That fits comfortably on a 12 GB card.

I built a small calculator that does this math for any model size, quantization, and context length, and checks the result against common GPUs and Macs: [the LLM VRAM calculator](https://flaviocopes.com/tools/llm-vram/).

## What fits where

One more thing before matching models to hardware: you never get 100% of the memory.

On a discrete GPU, the driver and display eat some VRAM. Count on about 90% usable. On Apple Silicon, the memory is **unified**: the GPU shares it with macOS and your apps, so about 70–75% is realistically available for the model.

With that in mind:

- **RTX 3060 (12 GB)** — about 10.8 GB usable. Runs 7–8B at Q4 with room to spare.
- **RTX 4060 Ti (16 GB)** — about 14.4 GB usable. Comfortable for 13–14B at Q4.
- **RTX 4090 (24 GB)** — the enthusiast default. 32B at Q4 fits, 70B does not.
- **RTX 5090 (32 GB)** — more headroom, but a 70B Q4 still needs ~38.5 GB of weights alone. Not enough.
- **Mac with 16 GB unified** — about 11.5 GB usable. Same class as the 3060: 7–8B at Q4.
- **Mac with 64 GB unified** — about 46 GB usable. A 70B at Q4 fits at short context, but gets tight fast as the KV cache grows.
- **Mac with 96 GB or more** — 70B at Q4 with real context headroom.

Let's verify that 70B case with the formula. Weights at Q4:

```
70 × 4 ÷ 8 × 1.1 = 38.5 GB
```

Add 8k of context (10.3 GB of KV cache) and you're at ~48.8 GB. That's past the 46 GB usable on a 64 GB Mac. Drop to 4k context and you squeeze in at ~43.6 GB, tight. This is why "it fits" always depends on how much context you actually use.

## The short version

Three numbers decide everything:

1. **Parameters × bits ÷ 8** gives you the weights.
2. **Context length** grows the KV cache, and it's not negligible on big models.
3. Multiply weights by **~1.1** for runtime overhead, and remember you only get 90% of a GPU (or ~72% of a Mac's unified memory).

And if a model doesn't fit and you fall back to an API, keep the provider swappable in your code. I wrote about how in [an LLM adapter pattern for Cloudflare Workers](https://flaviocopes.com/llm-adapter-pattern/). This calculator is part of a bigger set, too: I recently shipped [90 free browser tools for developers](https://flaviocopes.com/introducing-tools/), all running client-side.