Adaptive rounding techniques are one of the best ways to get more accuracy from quantization with the same bitwidth. In this series we’ll cover the evolution of adaptive rounding techniques starting from Adaround which delivers excellent quality with small models to more recent ones like GPTQ, OmniQuant, FlexRound which …
In my previous blogpost , we discussed in detail the outlier problem in transformers. We realized that outliers are not just random noise - they encode important directional information that lets the model selectively skip updating token representations. But the same property makes transformers notoriously difficult to quantize. In this post, we …
Modern Machine Learning models are trained with a large number of parameters, often too large, and this overparameterization is very useful during training as it creates a vast search space for the model to encode rich representations from data into its parameters. However, as it turns out, models do not …
A lot has happened in transformer quantization over the past few years, from barely being able to quantize a 7B model in INT8 without destroying accuracy, to routinely fitting a 70B model in 4-bits on a single GPU. But existing guides on the topic are fragmented: either focused on a …
Machine Learning (ML) has witnessed a surge in interest in recent years driven by the availability of large-scale datasets, advances in ML frameworks such as PyTorch and TensorFlow, rise of hardware accelerators (e.g., GPUs and TPUs) that enable large-scale training, and the development of increasingly powerful neural network architectures …
In this case study, we explore compressing neural networks for efficient deployment on edge devices with limited resources. We explore practical techniques like quantization, pruning, and tensorization using off-the-shelf open-source tools. Our aim is to illustrate a typical model compression workflow, highlighting the approaches and techniques used to analyse a …