The AI winter

Welcome to the December edition of the Token! The (AI) winter is coming, hopefully we are all better prepared for this one ⛄️

November was quiet, the exponential progress of the last two years seems to be slowing down 🐌 That said, Google managed to dethrone OpenAI from it’s top spot, Anthropic launched a feature that allows AI to control your computer 🖱️ and DeepSeek announced a model that is on par with GPT-4o 😮

In this issue, we also cover a case study on helping a scale up control a virtual simulation using their user’s voice 🗣️

• The latest Gemini model dethroned OpenAI’s GPT-4o from first place 🥇 on Chatbot Arena, something that has only happened once in the past - when Anthropic released Opus. Let’s see how long it stays there 🍿 https://venturebeat.com/ai/google-gemini-unexpectedly-surges-to-no-1-over-openai-but-benchmarks-dont-tell-the-whole-story/

• Scaling laws generalise to precision training or quantisation. It seems like 8-bit training is Pareto-optimal and training larger models with lower precision is at least equivalent and often superior to quantising post training 🔥 https://arxiv.org/pdf/2411.04330

• Anthropic introduced computer use, i.e. AI that controls your computer 💥 While "imperfect" (quoting Anthropic) it seems to outperform GPT-4o by a good margin. Could they made the right bet and outshine OpenAI by end of this year? https://www.anthropic.com/news/3-5-models-and-computer-use

• Mistral released Pixtral, its multimodal open model that outperforms the equivalent Llama to claim the title of best open multimodal model 🚀 https://mistral.ai/news/pixtral-large/

• DeepSeek announced R1, which seems to be on par or better than GPT4-o1 in tasks requiring more “thinking“ time 🤔 https://venturebeat.com/ai/deepseeks-first-reasoning-model-r1-lite-preview-turns-heads-beating-openai-o1-performance/

Controlling players in Virtual Reality can be somewhat cumbersome, something that breaks the immersion for the player. Instead of having to use menus or joysticks, it would be more realistic and natural to use your voice to control the simulation, especially in situations where a conversation with a character can drive the scenario forward. This is exactly what we helped a scale up to build for controlling a medical simulation environment.

Read more here https://mantisnlp.com/work/medical/

AI progress seems to be slowing down. The north star used to be OpenAI’s models but both closed and open source offerings seem to have caught up with GPT-4o, while GPT-4o1 or the rumoured Orion do not seem to be as transformational as GPT-4 was when it was released. We seem to have reached the limits of what scale alone can deliver and we might be in need of another breakthrough.

The good news is that we have only scratched the surface of the value that the current technology can unlock. So, even if this AI winter lasts longer than a few months or years, the impact AI will have on our lives over the coming years will be transformational. From better and conversational assistance in the applications and devices you use; to a different way to educate, learn, diagnose, write, and review; these are among the many areas that AI will touch.

During this (AI) winter, I don’t think we will see a drop in investment from investors nor a reduced appetite for integrating and leveraging AI from business. In that respect, it will be the best (AI) winter to date 🎄