I tried to understand Kimi K2.5 from First Principles.

Dude, I love Chinese Frontier AI Tech Reports and this paper was insane with so many nuances. I have tried to put down my raw thoughts from first principles here.

• See and understand the world (vision)

• Think and reason (text, code, math)

• Act autonomously (agents with tools, browsers, code exectuion)

The two core innovations (or if we say the big bets they’ve made):

1. Joint Text-Vision Optimization - making vision and text enhance each other instead of competing for capacity

2. Agent Swarm - parallel execution of agents instead of sequential (we are used to)

Native Multimodal Pre-Training

You might have heard about conventional wisdom for training VLMs:

Train your text model first, then bolt on vision late in training with high vision ratios (50%+). This lets the model quickly acquire multimodal capabilities.

• Early fusion with 10% vision → best results

• Mid fusion with 20% vision → worse

• Late fusion with 50% vision → worst

It proves that the model trained with less vision data earlier beats the one trained with more vision data later.

Let’s think about why this has to happen?

Think about how humans learn languages. If you want a kid to be bilingual, you don’t wait until they’re 18 and then cram. You expose them early, consistently and at moderate levels. The neural pathways form together.

Same principle here. When you inject vision late at high ratios, something ugly happens:

See that dip-and-recover pattern in the 50:50 curves? When vision gets injected late, text performance crashes temporarily. The model has to rewire itself. It eventually recovers, but never quite catches up.

Early fusion avoids this trauma entirely. The representations grow together from the start.

The takeaway: VLMs doesn’t need vision-heavy training. It needs vision-early training.

Zero-Vision SFT - A Hack!

After pre-training, we need to teach the model to use tools. For a VLM, that means teaching it to use tools with visual inputs like crop images, analyze charts, manipulate pixels etc.

The obvious approach: annotate thousands of visual tool-use trajectories. Show the model examples of “look at image → decide to crop → execute crop → analyze result.”

Kimi’s approach: don’t do any of that.

They call it Zero-Vision SFT. During supervised fine-tuning, they use only text data. No visual examples. No annotated vision trajectories.

And it works better. Interesting, right.

This is the training curve after zero-vision SFT, when they start RL on vision tasks:

The starting point is from zero-vision SFT. The model already knows how to reason visually. (it just learned it through text)

How is this possible?

The joint pre-training did the heavy lifting. By training text and vision together from the start, the model learned aligned representations. Tool-use capabilities learned from text automatically generalize to vision.

It’s like teaching someone to cook by describing recipes. If they already know what ingredients look like (joint pre-training), they can figure out visual cooking tasks without you showing them videos of cooking.

When they did add human-designed visual trajectories during SFT, performance got worse. The hand-crafted examples were too narrow. They hurt generalization.

We can generalize here: if our pre-training already aligned modalities well, low-quality SFT data for the secondary modality can actually hurt by overfitting to narrow patterns. Better to let capabilities transfer naturally from high-quality text SFT.

Vision RL Improves Text - This was Mind-Bending!

They trained RL on pure vision tasks — grounding, counting objects, reading charts. Standard visual reasoning stuff.

Then they checked the text benchmarks:

Their hypothesis is visual RL improves “calibration” for structured information extraction.

Basically, when you get better at counting pixels - you also develop better counting skills generally. When you get better at extracting data from charts - you get better at extracting data from text tables. Likely this is how underlying cognitive machinery is shared.

This is the strongest evidence that joint training actually works. The modalities aren’t competing rather they’re enhancing each other bidirectionally.

Agent Swarm: Parallel Agents Are Hard!

The problem with current AI agents is:

Everything is sequential. Think → call tool → wait → think → call tool → wait. Repeat “n” times.

Even if your model can do 100 reasoning steps (like Kimi K2 Thinking), the latency scales linearly and complex tasks become unbearably slow.

The obvious fix is parallelize them. Spawn multiple agents working simultaneously.

The less obvious problem: training parallel agents is a nightmare.

Two fundamental challenges:

1. Credit assignment ambiguity: If the final answer is wrong, which subagent screwed up? If it’s right, who deserves credit?

2. Training instability: When multiple agents learn simultaneously, gradients become noisy. Everyone’s changing while everyone else is changing.

Kimi came up with PARL (Parallel-Agent Reinforcement Learning)

The insight here is: don’t train everything end-to-end.

• Train only the orchestrator.

• Freeze all subagents (use old model checkpoints).

• Treat subagent outputs as “environment observations” not as differentiable decision points.

This is a clean separation. The orchestrator learns coordination intelligence like when to spawn agents + how to divide tasks + how to aggregate results. The subagents just execute with fixed capabilities.

Look at that training curve. Smooth accuracy increase. Parallelism naturally emerges as training progresses and the model learns when parallelization helps.

The Infrastructure - Hack!

The problem: In pipeline parallelism, the vision encoder sits in Stage-0 of the pipeline. But vision inputs have wildly variable sizes (different image counts, different resolutions). This creates massive load imbalance. Stage-0 becomes a bottleneck.

Standard fix here: Manually tune the pipeline configuration for multimodal workloads but its painful and fragile.

Kimi came up with Decoupled Encoder Process (DEP).

1. Balanced Vision Forward: It replicates the (small) vision encoder on ALL GPUs. Distribute vision compute evenly by image count, regardless of pipeline stage. Throw away intermediate activations.

2. Backbone Training: Normal forward/backward for the LLM. Inherits all the optimizations you’ve built for text-only training.

3. Vision Recompute & Backward: Re-run vision forward (it’s small + cheap), then backward pass.

This is how the vision encoder processing is completely decoupled from the LLM pipeline strategy.

Result: 90% training efficiency relative to text-only. The multimodal overhead is almost eliminated.

The Actual Numbers!

Okay, enough of theory. It’s a pretty good model.

Strong on agentic tasks (BrowseComp, WideSearch). Competitive on reasoning (AIME, SWE-Bench). Solid on vision (Video-MME, OSWorld).

The model checkpoints are open source: huggingface.co/moonshotai/Kimi-K2.5

This is all for this blog post.

For queries/collaborations, say hi - @ himanshustwts