As artificial intelligence models grow larger and more complex, they frequently exceed the operational capabilities of standard hardware. To address mounting resource challenges involving memory usage, computational speed, and energy consumption, model quantization has become an essential technique. By reducing the numerical precision of parameters from high-precision formats like FP32 down to formats like FP8, developers can significantly shrink a model's memory footprint and boost inference efficiency without suffering major losses in accuracy.

Understanding Data Types and Numerical Formats

AI models rely on floating-point numbers to represent parameters, with data types directly dictating the computational resources required. A standard floating-point number stores numerical values across a specific number of bits divided into three components: a sign, an exponent, and a mantissa.

Common formats include FP32, FP16, BF16, and FP8. Specifications can also detail bit allocation; for instance, an FP8 format designated as E4M3 allocates 4 bits to the exponent and 3 bits to the mantissa. Shifting to lower bit widths decreases computational demands, though finding the right balance between hardware efficiency and model precision depends heavily on the target use case.

Key Elements Targeted for Quantization

In modern transformer-based models, three main components can be targeted for quantization to optimize performance:

  • Model Weights: The most straightforward target. Weights are static and data-independent, meaning they can be compressed easily. For example, storing the Llama2 7B model in FP16 or BF16 takes roughly 14 GB of memory. Reducing the weights to FP8 cuts that memory requirement in half to about 7 GB.
  • Model Activations: These are the dynamic, intermediate outputs generated by model layers during inference. While not explicitly part of the saved model file, quantizing compute activations allows specialized tensor core hardware to scale throughput.
  • KV Cache: Specific to decoder-based models that generate output tokens autoregressively, the Key-Value cache speeds up generation. With long context windows—such as 4096 tokens in Llama2 7B—the KV cache can consume several gigabytes of memory, making it a prime candidate for optimization.

Quantization Methods: PTQ and QAT

Converting high-precision floating-point values into low-precision representations involves specific mapping techniques, such as symmetric or affine (asymmetric) quantization governed by scale factors and zero-points. To implement these techniques across a model, developers typically choose between two main methodologies:

  • Post-Training Quantization (PTQ): Applied after a model is fully trained, PTQ can focus solely on weights or extend to weights and activations. Because activations change depending on input data distributions, PTQ relies on calibration steps using representative input data and observers to determine optimal scaling factors.
  • Quantization-Aware Training (QAT): Designed to counter quality degradation, QAT integrates quantization effects directly into the training or fine-tuning process. It simulates low-precision arithmetic using "fake quantization" modules during forward and backward passes. To handle non-differentiable operations like rounding, QAT applies the straight-through estimator (STE) to approximate gradients and allow the model to adapt to rounding errors.

Ultimately, balancing weights, activations, and KV caches through strategic quantization methods allows large language models to operate efficiently within constrained hardware environments.