Quantization

A model is mostly a giant table of numbers. Trained models usually store each one in 16 bits, which is why an 8-billion-parameter model needs roughly 16 GB just to load. Quantization re-encodes those weights with fewer bits, typically 8, 5, or 4, using schemes that work in blocks and keep the most sensitive parts at higher precision.

You meet quantization the moment you run a model yourself. Files in the GGUF format used by llama.cpp and Ollama carry names like Q4_K_M or Q8_0 that describe the bit width and the mix. Lower precision means a smaller download, a model that fits on a laptop or a single GPU, and faster generation, because producing tokens is mostly limited by how quickly weights can be read from memory.

The cost is quality, and it isn't spread evenly. Moderate quantization is often hard to notice, while very low bit counts degrade output visibly, and small models suffer more than large ones. A 4-bit k-quant is a common starting point; step up if your evals show problems, and rerun them after any change, because quantizing changes outputs. Remember that context needs memory too: the KV cache grows with the context length you configure, on top of the weights.