How Qwen 3 Coder took over Kimi K2 Hype?

author: @himanshustwts

Welcome to the first-ever Spotlights by GroundZero! This week in AI was all about two Chinese heavy-weights. Qwen 3 Coder took the spotlight that Kimi K2 enjoyed for merely 13 days.

Qwen 3 Coder is not only half the size of Kimi K2, it scored higher in coding benchmarks. You might be wondering how’s it even possible that is significantly smaller in size outperforms other SOTA models that we use today?

After OpenAI dropped the scaling laws paper (link to paper) back in Jan 2020, industry focussed on expanding the existing models to get better performance. This is because scaling law proposed that we can predictively improve the model’s performance based on three key variables [SIZE, DATA, COMPUTE].

A lot of people confuse scaling law with with Moore’s law, saying that a LLM will just get better overtime. So they look at groundbreaking models like Qwen 3 Coder to confirm their bias.

Difference between Moore’s Law and Scaling Law!

People love to mix these up. Moore’s law is rather an observation of how fast technology advances, scaling law propose a power law between above three variables which gives this hot AI-industry the go-ahead in funding to take the existing architecture and scale them up to to get better performance, until it didn’t.
Although the industry now focusses on better architecture and better technique.

Qwen 3 Coder proves raw size is overrated. It’s not about who’s got the biggest; but who’s got the better recipe.

Highlight:

• Up to 1M context window

• 35B active parameters

• Same performance as Sonnet 4

Starting with MoE, Qwen 3 Coder is a 480B params in size with 35B active params and has 160 experts. In comparison Kimi K2 has 1T params in total size with 32B active params and has 384 experts.

MoE architectures (Sparse) make inference makes inference way faster and affordable compared to dense models that typically use entire model.

In Pre-Training stage, Qwen 3 Coder was trained on 7.5T tokens. Kimi K2? 15.5T tokens. That’s double the data for Kimi. Worth mentioning that Kimi K2 is not specifically a coding model like Qwen 3 Coder BUT the difference in data size is staggering. It outperforms Kimi K2 in coding benchmarks.

Though both models share MoE as key architecture but their pre-training stages looks totally different.

In Post-Training stage, Qwen focussed on two main strategies in Post-Training processes.

1. Code RL - optimized for coding tasks. (pass/fail, tight feedback loops)

2. Long-Horizon RL - give a model a lot of leash in figuring out how to plan and use tools

To top it off, Alibaba ran 20,000 coding simulation environments in parallel just to tune Qwen 3 for coding.

Though some users have been talking about speed and output is not up to the cost, which eventually happens with most of open sourced models.

Also, read a bunch of posts that Qwen 3 Coder is just bench-maxxed. Ah well!

Crazy times! Both of these models were released completely open sourced with Apache 2.0 license. As hardware improves in parallel, this will present a chance to run these models locally for a naive user. In Qwen 3 Coder there has been focus on training techniques that are used to improve the model rather than the sheer dependency on [SIZE, DATA, COMPUTE]

Qwen 3 Coder Blog: https://qwenlm.github.io/blog/qwen3-coder/

This is first newsletter-cum-post on Spotlights by GroundZero. Expect weekly drops on the latest in AI research, trends, and whatever’s shaking up the scene. Sub to the Substack, and catch us on X and YouTube for more.

Twitter/X: Ground Zero

YouTube: ML, Maths and Podcasts

Cheers!