Geometrical Insights for Implicit Generative Modeling Abstract: Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion. A careful look at the geometries induced by these distances on…
This short position paper argues that LLMs are best viewed as Fiction Machines, that machines able to write stories that might not be related to what is factual but are internally coherent. Considerable efforts are spent aligning these machines with our expectations, chasing hallucinations, and ignoring the fact that the ability to make up stories is key to intelligence.
Abstract: Abstract: “Many believe that Large Language Models (LLMs) open the era of Artificial Intelligence (AI). Some see opportunities while others see dangers. Yet both proponents and opponents grasp AI through the imagery popularised by science fiction. Will the machine become sentient and rebel against its creators? Will we experience a paperclip apocalypse? Before answering such questions,…
Publications Follow each publication link to access papers and supplemental data. Most papers are available in DjVu, PDF, and PS.GZ. * Download a DjVu viewer. * Link to my page on Google scholar 2026 ... in progress ... 2025 Jingtong Su, Jianyu Zhang, Karen Ullrich, Léon Bottou and Mark Ibrahim:
Memory Mosaics at Scale Abstract: Memory Mosaics, networks of associative memories, have demonstrated appealing compositional and in-context learning capabilities on medium-scale networks (GPT-2 scale) and synthetic small datasets. This work shows that these favorable properties remain when we scale memory mosaics to large language model sizes (llama-8B scale) and real-world datasets. To this end,…
MagicPIG: LSH Sampling for Efficient LLM Generation Abstract: Large language models (LLMs) with long context windows have gained significant attention. However, the KV cache, stored to avoid re-computation, becomes a bottleneck. Various dynamic sparse or TopK-based attention approximation methods have been proposed to leverage the common insight that attention is sparse. In this paper, we first…
Memory Mosaics Abstract: Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way (“predictive disentanglement”). We illustrate these…
Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need Abstract: Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. However, SSL requires to build samples that are known to be semantically akin, i.e. positive views. Requiring such knowledge is the main limitation of SSL and is often tackled by…
Birth of a Transformer: A Memory Viewpoint Abstract: Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their internal mechanisms in order to make them more reliable. These models appear to store vast amounts of knowledge from their training data, and to adapt quickly to new…