What is quantization, and does it hurt model quality?
Quantization stores model weights with fewer bits, for example 4 or 8 bits instead of 16. A 4-bit model needs about a quarter of the memory of a 16-bit one and usually runs faster. At 8 bits the quality loss is usually hard to notice, and at 4 bits it is small for most tasks.
A rough memory rule: parameters × bits ÷ 8 = bytes. A 70 billion parameter model needs about 140 GB at 16-bit, about 70 GB at 8-bit and about 35 to 40 GB at 4-bit, before context.
Below 4 bits, quality drops faster, especially on math, code and long reasoning. Larger models tolerate aggressive quantization better than small ones.
Some models ship already quantized. Kimi K2 Thinking is released natively in 4-bit (INT4), and OpenAI’s gpt-oss models ship with MXFP4 weights.
Sources
Updated