When the Babel Fish Dies: Replacing Legacy Language Detection

Facebook&rsquo;s fasttext and Google&rsquo;s cl3d ) were both deprecated mid-2024. While multilingual LLMs can accomplish language identification tasks, using even a 7B-parameter LLM to determine a text&rsquo;s language is overkill. I set out to find a replacement that is at least as fast and performant as the deprecated models. Code from these experiments is available here . TLDR Lingua is the…