World Models will push the frontier for LLMs
Large Language Models are trained using the next-token prediction objective. This means that the vocabulary distribution of the model needs to be close to that of the training dataset. This objective is highly effective for language because it is already a discrete and compressed representation of world knowledge. Since the middle of 2024 1 , there has been a push for multimodality support in…