I work on efficient deep learning, with a focus on quantization and low-rank methods for…

• LoQT: Low-Rank Adapters for Quantized Pretraining

  NeurIPS November 1, 2024

  LoQT enables efficient quantized pre-training of LLMs with results close to full-rank non-quantized models. It enables pre-training of a 13B LLM on a 24GB GPU without model parallel, checkpointing, or offloading strategies during training.

  See publication