Education

How big AI models fit in your RAM

"How much RAM do I need to run that model?" — and what do Q4, FP16, parameters, experts and GGUF actually mean? This is the whole story, built from a real chat, told so anyone can follow it. Plug in a model, watch a weight get squeezed, and route a token through a mixture of experts.

  • How much RAM
  • Quantization
  • Parameters
  • Mixture of experts
  • GGUF & formats

Everything below is interactive — start with the calculator, then drag a weight down to 2 bits and light up some experts.

Start here

How much RAM do I need to run this model?

Short version: the memory a model needs is set almost entirely by two numbers — how many parameters it has, and how many bits you store each one in (its quantization). Pick a model and a quant level below and watch the numbers move. Every term here gets its own hands-on section underneath.

Pick a starting point, then tweak anything

744 B

All weights must sit in memory — even the ones that don't run on every word.

40 B

Only this slice computes per word — it sets the speed, not the memory.

Fewer bits = smaller & faster, a little less accurate. The Quantization section below lets you feel it.

32K

The conversation's memory (KV cache). Bigger window = more RAM on top of the weights.

Model file on disk —
RAM you'd want —
Rough speed (CPU) —

Approximate, for intuition — real GGUF quant files carry a little overhead and "dynamic" quants keep important layers at higher precision, so actual sizes drift a bit. The relationship is what's solid: size ≈ parameters × bits ÷ 8.

The first number

Parameters: a model is just a giant pile of numbers

When people say a model is "744B", they mean it's built from 744 billion numbers — the parameters (also called weights). Each one is a tiny dial the model tuned during training. That's literally all a model is: an enormous list of numbers plus the simple rules for multiplying your input through them. More parameters generally means more capability — and more memory to hold them.

Tap a model to feel the scale (at full 16-bit precision)

Each parameter is one number. Stored at full precision (16 bits = 2 bytes each), the count alone decides the raw size — which is exactly why we squeeze the bits next.

Numbers grouped into "tensors"

Those billions of numbers aren't a flat list — they're packed into grids called tensors (a fancy word for a multi-dimensional array of numbers). A model is hundreds of tensors stacked in layers. "Tensor type" just means what number format each grid uses.

Total vs active

In some models (mixture-of-experts) only a fraction of the parameters run on each word. The full count still has to live in memory; the active slice is what does the maths. Hold that thought — it's the whole trick behind big models that still run fast.

The big idea

Quantization: store each number in fewer bits

"Quant" is short for quantization — a compression trick that shrinks the model by storing every weight with fewer bits of precision. Trained models keep each number as a 16-bit value (2 bytes). Quantization rounds them to fewer bits — trading a little accuracy for a lot less memory. It's like saving a photo as a lower-quality JPEG: still clearly the photo, just lighter and slightly less crisp.

Drag the precision down and watch one row of real weights snap to a coarser grid. The file shrinks; the picture gets blockier. Notice how gracefully it degrades — that's why 4-bit is so popular.

4-bit
File size
100%
Detail kept
100%
Distinct values each weight can take

The bits-per-weight ladder

The names just tell you the bits. Fewer bits → smaller file, less memory, a bit more error.

"Dynamic" quants: smart about where to spend bits

A plain 2-bit quant crushes every weight to 2 bits. A dynamic quant (e.g. Unsloth's "UD" GGUFs) keeps the sensitive layers at higher precision and squeezes the rest harder. That's why a good 2-bit dynamic quant still works surprisingly well — it isn't uniformly 2-bit, it's clever about which bits matter. Which layers are sensitive? That's the next section.

The source format

Tensor types: BF16, F32 — the precision before you quantize

A model card often says something like "Tensor type: BF16 · F32". That's the original precision the weights were trained and released in — before anyone quantizes them. It's usually a mix: most weights in BF16, a few sensitive ones kept in F32. Every number format trades off two things — range (how big/small a number it can hold) and precision (how finely it splits the values in between). Click each to see how the bits are split.

Sign (+/−) Exponent — the range Mantissa — the precision

Why models ship in BF16, not FP16

BF16 and FP16 are both 16-bit, but they split the bits differently. FP16 spends more bits on precision but has a narrow range — big or tiny numbers overflow or vanish to zero. BF16 keeps the full F32 range with less precision. During training, not overflowing matters more than fine detail, so almost every modern model is released in BF16.

Why a few layers stay F32

A handful of layers are kept at full 32-bit because a rounding error there hurts the most — things like normalization and the MoE router. Quantization tools (and dynamic quants) protect exactly these. To see why the router is so touchy, route a token through the experts next.

Why a huge model can still run

Mixture of Experts: a router that wakes only a few specialists

In a normal ("dense") model, every weight is used for every word. A Mixture-of-Experts model splits part of each layer into many separate sub-networks — the experts. For any given word, a tiny router (a.k.a. the gating network) scores all the experts and sends the word to only the top few. That's how a model can have a huge total size but a small active slice. Send a token through it:

a word
router scores everyone
32
8
In memory (all experts)100%
Computed (only the chosen)25%

The library mental model

The model is a huge library of books (the experts); the router is the index card saying which few to grab; the inference program is the librarian who fetches just those. But the whole library still has to be built and stocked — which is why all the weights live in RAM even though only a few run per word.

Why the router is touchy

The router makes a discrete choice — which experts win. If quantization nudges a score and a different expert gets picked, the answer can change sharply, not gracefully. A small error in an ordinary weight just makes one number slightly off; a small error in the router can reroute the whole word. That's why it's kept at high precision.

What makes it fast or slow

Speed: the active slice and your memory bandwidth

The active parameters set the speed, not the total. Generating each word means reading those active weights out of RAM and doing the maths. On a CPU the bottleneck is almost never the maths — it's memory bandwidth: how fast weights can be shovelled from RAM into the chip. So a rough rule is speed ≈ bandwidth ÷ bytes read per word. Fewer bits (lower quant) means fewer bytes to read — which is why quantizing also makes it faster.

40 B
Read per word —
Rough speed —

What the chip actually does: multiply-add

Once weights arrive, the core operation is multiply-accumulate (MAC): multiply each weight by an input number, add it to a running total, repeat millions of times. Chips do many at once with SIMD instructions (AVX on x86, NEON on ARM). Weak SIMD support = much slower CPU inference.

Why it waits on the conveyor belt

The SIMD units finish their multiply-adds faster than RAM can deliver the next batch of weights — so the chip waits. The maths isn't the holdup; the conveyor belt feeding it is. GPUs flip this with far higher memory bandwidth (GDDR on consumer cards, HBM in datacentres), so they stay fed and run far faster.

A common myth

Languages: it's the training data, not the final step

It's tempting to think a model is "bad at Czech" because of some final step. It isn't. The last thing a model does is pick the next token from its vocabulary — the tokenizer's set of text chunks. As long as a language's characters are in the vocabulary (they almost always are for major languages), the model can emit it. How well it does comes from how much of that language it saw in training — and quantization erodes the weakest languages first. Type something in both languages:

For under-represented languages the tokenizer often chops text into more, smaller pieces — so the same sentence costs more tokens. That doesn't break the language, but it's slower, pricier, and eats the context window faster. (Simplified illustration.)

Vocabulary = can it emit the characters

The vocabulary mostly sets efficiency, not what's possible: modern tokenizers are byte-level, so a model can spell out any character byte by byte even with no dedicated token — it just costs more tokens. Czech ř č ž are representable everywhere; rare scripts simply cost more. So the vocabulary is rarely a hard limit.

Quantization hits weak languages first

A model's grip on a low-resource language is already thin, so squeezing weights to 2-bit degrades that before it dents its rock-solid English. If a heavily-quantized model's Czech feels shakier than its English, that's quantization eroding the thinnest representations — not the vocabulary refusing the language.

What you download

PyTorch, Safetensors, GGUF, MLX — who does what

On a model page you'll see a row of library tags. They split into three jobs: the engines that do the tensor maths, the high-level library that loads models conveniently, and the file formats that store the weights. For running a model locally on CPU/Ollama, the one that matters is GGUF. Tap a tag:

So what does GGUF stand for?

GGUF = GGML Universal File. It grew out of GGML — a lightweight C/C++ tensor library named after its creator, Georgi Gerganov (the developer behind llama.cpp). The old GGML files broke every time the format changed; GGUF replaced them in 2023 with a self-describing format — one file holds the quantized weights, the tokenizer, and metadata ("I'm this model, MoE, 4-bit, here's the chat template"). That's why Ollama or llama.cpp can run a single .gguf with no extra config files.

Curious how the model "thinks" in the first place?

This guide was about running a model. How one actually learns and predicts the next word is its own story. Open "How AI Works" →

In one breath

The whole thing, recapped

  1. A model is a pile of numbers — the parameters (weights). "744B" = 744 billion of them.
  2. RAM ≈ parameters × bits ÷ 8. All the weights must live in memory at once.
  3. Quantization stores each weight in fewer bits (Q4, Q2…) — smaller and faster, slightly less accurate; dynamic quants protect the important layers.
  4. Tensor types (BF16/F32) are the original precision; BF16 keeps F32's range, and touchy layers like the router stay F32.
  5. Mixture of experts: a router wakes only a few experts per word — huge total, small active slice.
  6. Active params set the speed; on CPU the real limit is memory bandwidth (bytes read per word).
  7. Languages come from training data (and quant erodes weak ones first), not the final vocabulary step.
  8. You download a GGUF (GGML Universal File) to run a quantized model locally with llama.cpp or Ollama.
About this guide

A RAM calculator and plain-language guide to running LLMs locally

This page answers the question people ask before downloading a model for llama.cpp or Ollama: how much RAM do I need to run it? The calculator at the top turns a model's size, quantization level and context window into a file size, the memory to plan for and a rough speed on a CPU.

The rest of the page explains every term behind those numbers, each with a small demo: what parameters are, what quantization does to a weight, why BF16 exists, how a mixture-of-experts router works, why memory bandwidth sets the speed, and what a GGUF file contains. It is written for developers, hobbyists and anyone choosing hardware for local AI, with no maths needed beyond multiplication.

Everything is calculated in your browser, needs no account, and every figure is an approximation meant to build intuition. The relationships behind it are the solid part: file size is about parameters times bits divided by eight.

What the calculator and demos show

  • RAM calculator: presets from 8B to about 744B, total and active parameters, eight quant levels and a context slider up to 256K.
  • Results you can act on: file size on disk, RAM to plan for, rough CPU tokens per second and which class of machine fits, with the formula spelled out.
  • Weight rounding: drag bits per weight from 16 down to 1 and watch an image and eight sample weights snap to fewer levels.
  • Bits-per-weight ladder from FP16 to Q1 with the effective bits real quant files use, plus the F32, BF16 and FP16 bit layouts.
  • Mixture-of-experts router: route a word, change the number of experts and the top-k, and compare memory with compute.
  • Memory bandwidth: pick DDR4, DDR5, a server, Apple unified memory or a GPU and see bytes read per word and tokens per second.
  • Tokenizer comparison of English and Czech, and a guide to PyTorch, Safetensors, GGUF and MLX.
Step by step

How to estimate the RAM a model needs

  1. Pick a starting model

    Tap a preset, from a small 8B dense model to a frontier-size mixture of experts, or drag the total parameters slider to the size of your model.

  2. Set the mixture of experts switch

    Turn it on for an MoE model and set the parameters active per token. This changes the speed estimate, while the file size still counts every parameter.

  3. Choose a quantization level

    Pick a quant from FP16 down to about 1 bit. Q4 is the common choice for running models locally.

  4. Set the context window

    Drag the context slider to the conversation length you need. A longer context adds KV cache memory on top of the weights.

  5. Read the result

    Check the file size on disk, the RAM you would want, the rough CPU speed and the machine tier that fits, then read the formula breakdown under it.

Good to know

Rules of thumb

  • File size in GB is about billions of parameters × bits per weight ÷ 8. A 70B model at Q4 (about 4.5 bits in practice) is roughly 39 GB, before any context.
  • Plan for more RAM than the file: the context (KV cache) and the operating system need room too, and a longer context needs more.
  • For a mixture-of-experts model, the total parameters decide whether it fits and the active ones decide how fast it runs.
  • Q4 is the usual sweet spot. Quality starts slipping at Q3, and Q2 or lower works best as a dynamic quant that keeps important layers at higher precision.
Questions

Frequently asked questions

How much RAM do I need to run an LLM locally?

Roughly the size of the model file plus room for the context and the system. The file is about parameters times bits per weight divided by 8, so a 70B model at Q4 is roughly 39 GB on disk and needs more RAM than that. The calculator adds an estimate for the context (KV cache) and a couple of GB for the operating system.

What does quantization do to a model?

It stores each weight in fewer bits, for example 4 instead of 16. The file shrinks to roughly a quarter, the model needs less memory and runs faster, and it loses a little accuracy. Q8 is practically lossless, Q4 is the popular compromise, and quality starts slipping at Q3.

What do Q4, Q8 and FP16 mean?

They give the bits stored per weight: FP16 is 16-bit floating point, the full precision most models are released in, Q8 uses 8 bits and Q4 uses 4 bits. Real quant files average a bit more than the nominal number because of overhead and layers kept at higher precision, so Q4 works out to about 4.5 bits per weight.

What is the difference between BF16 and FP16?

Both use 16 bits. BF16 keeps the 8 exponent bits of F32, so it has the same range, and only 7 bits of mantissa. FP16 has 5 exponent bits and 10 mantissa bits: more precision but a much smaller range. The wide range is why BF16 is the usual format for training.

What is a mixture-of-experts (MoE) model?

An MoE model has many expert sub-networks and a router that picks only a few of them for each token. All experts must sit in memory, but only the active slice is computed, so a model with hundreds of billions of parameters can generate text about as fast as a much smaller dense model.

Why is my local model slow on a CPU?

For every generated token the computer reads the active weights from memory, so speed is limited mainly by memory bandwidth, not by the processor. Roughly, tokens per second equal bandwidth divided by the bytes read per token. Dual-channel DDR5 gives about 80 GB/s, while GPU memory (HBM) reaches about 3,300 GB/s.

What is a GGUF file?

GGUF stands for GGML Universal File, the format used by llama.cpp and Ollama. It replaced the older GGML files in 2023. One self-describing file holds the quantized weights, the tokenizer and metadata, so you can run a single .gguf file without extra config files.

Does quantization make a model worse at Czech or other languages?

It can. A model's grip on a language it saw less of in training is already thin, so aggressive quantization tends to erode it before English. The vocabulary is rarely the limit: less common languages are split into more tokens, which costs speed and context but does not stop the model from writing them.

Are the calculator numbers exact?

No, they are approximations for intuition. Real quant files carry some overhead, the KV cache and the speed depend on the exact model and engine, and dynamic quants vary. The relationships are reliable: size is about parameters times bits divided by 8, and speed is about bandwidth divided by the bytes read per token.