Hi everyone,
In this edition of The Weekly Kaitchup:
DeepSeek V4 Flash: Now Better than DeepSeek V4 Pro (Preview)
Inkling-Small: Better than Inkling (?)
Escha-W2: Qwen3.6 35B Compressed to 12.3 GB and Faster
DeepSeek released an update for its V4 Flash model. As usual, DeepSeek released the weights for this update immediately, rather than imposing the waiting period we have seen from companies such as MiniMax and Qwen.
It now outperforms the much larger V4 Pro (Preview), with particularly strong gains in agentic coding:
They did not publish results for other task categories. I checked Artificial Analysis’ additional results and found that it also improved on other benchmarks, including GPQA Diamond, which is a very different task from agentic coding. In fact, none of the reported benchmarks appear to show a regression. This is not merely a more specialized model, it is a substantial improvement over the preview and a very promising sign for the next V4 Pro update.
On a related note, Qwen also released Qwen3.7 Flash this week. It is currently available only through APIs and is very inexpensive. The community has speculated that it may be a Qwen3.7 35B-A3B model and that Qwen will eventually release the weights. I am not certain this is correct, but it would make sense.
Despite the name, Inkling-Small is not a lightweight model in the usual sense. It is still an open-weight, decoder-only multimodal model that accepts:
Text
Images
Audio
It follows the same general architecture as the larger Inkling. Both models use a sparse mixture-of-experts design, routing each token through six of 256 experts, along with two shared experts. Both also support context windows of up to one million tokens.
So Inkling-Small is mostly the same design, scaled down.
The larger Inkling has:
66 transformer layers
975 billion total parameters
About 41 billion active parameters per token
Inkling-Small reduces that to:
42 transformer layers
276 billion total parameters
About 12 billion active parameters per token
That makes Inkling-Small a little over one-quarter the size of the larger model by total parameter count. Its active parameter count is reduced by a similar amount.
Inkling-Small requires around 600 GB of aggregate VRAM in BF16.
The NVFP4 checkpoint brings that down to roughly 180 GB.
For comparison, the larger Inkling’s NVFP4 checkpoint is around 592–600 GB. In other words, the quantized version of the large model takes roughly as much memory as Inkling-Small does in BF16.
The word “small” is doing some work here, but 180 GB is at least in the range of a compact multi-GPU server rather than a large GPU cluster.
The surprising part is that Inkling-Small beats the larger Inkling across most published head-to-head benchmarks.
A few examples:
SWE-bench Verified: 80.2% vs. 77.6%
Humanity’s Last Exam: 31.6% vs. 29.7%
IFBench: 82.2% vs. 79.8%
The smaller model appears particularly strong in coding, tool use, instruction following, and several reasoning evaluations.
Thinking Machines attributes the improvement to a revised data mix, an updated training recipe, distillation from the larger Inkling model, and additional reinforcement learning focused on agentic coding.
The larger model still performs better in some areas, especially factual recall, broad knowledge coverage, and most audio evaluations.
Still, it is an unusual result: the smaller model uses far fewer parameters, needs much less memory, and yet comes out ahead on a majority of the reported benchmarks.
Another unexpected result is that it significantly outperforms Qwen3.5 397B despite being much smaller, especially on benchmarks where Qwen3.5 has always been exceptionally good, like GPQA Diamond.
Is it benchmaxxed?? I’m waiting for the community feedback!
If you primarily use GGUF models locally, a 12.3 GB version of Qwen3.6 35B may not sound especially impressive. After all, that is roughly the size of a heavily quantized GGUF model, like a Q2_K_XL, and we already know those can perform well, as shown by the results I published a few months ago.
The key difference with Escha-W2 is that it uses (presumably) a more advanced quantization method and comes with an optimized inference runtime. As a result, it should run faster than the typical GGUF model, while already supporting SGLang for significantly better throughput under concurrent workloads. The developers have also said that vLLM support is coming later.
We don’t really know yet.
The original model is a mixture-of-experts model with:
35 billion total parameters
Around 3 billion active parameters per token
256 experts
Escha Labs keeps that underlying architecture but compresses most of the expert weights to roughly two bits. More precisely, it uses a mixture of two- and three-bit quantization for the expert projections, while keeping the dense layers in INT8.
The resulting checkpoint takes up 12.3 GB on disk.
It can run on:
A single 24 GB GPU under the recommended configuration
A 16 GB GPU with reduced context length or fewer concurrent requests
On an RTX 4090, Escha reports around 225 tokens per second for a single stream. The same setup reaches about 1,321 tokens per second when serving 32 requests at once.
The more interesting part is how little the compression appears to affect most of the reported evaluations.
Across six benchmark categories, Escha-W2 averages 100.2% of the FP8 model’s score. That does not mean the quantized model is genuinely better, the small gains are mostly within normal evaluation variance, but it does suggest that the overall quality loss is limited.
A few examples:
MMLU-Pro: 80.9 versus 82.3 for FP8
MATH-500:93.8 versus 91.2(ignore this one; this is too old and too easy for Qwen3.6)GPQA-Diamond: 77.8 versus 74.7
BFCL tool use: 88.9 versus 88.2
RULER long-context retrieval: 89.9 versus 89.4
The clearest regression is on longer coding tasks, where quantization always does more damage.
On LiveCodeBench v6, Escha-W2 scores 62.6, compared with 67.0 for the FP8 baseline. Note: I don’t know which framework they used to get a 67.0, but these scores seem very low. It should be around 85.0. This suggests that they ran it with a limited context length, like 32K max tokens. Not great. This means that the accuracy gap could be greater at longer context length, as quantization tends to be worse as sequence length increases.
There is one practical catch: Escha-W2 needs a custom runtime.
Its weights use a custom packed format, so the model does not simply load into standard inference software as a regular GPTQ, AWQ, or GGUF checkpoint. Escha currently provides two options:
An SGLang-based runtime for concurrency, tool calling and structured output
A standalone ZML runtime aimed at single-user inference
The problem with specialized runtimes is that it’s very hard work to maintain them.
The checkpoint is also text-only, even though the original Qwen architecture includes vision components.
When they release the vLLM runtime, I’ll double-check the accuracy on longer context and compare it with quantized versions of similar size.
A $50 Coupon to Try Verda’s GPUs
In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).
Coupon code:
KAITCHUP-50Follow these instructions to redeem it.
Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.
That’s all for this week.
If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):
Share The Kaitchup – AI on a Budget
Have a nice weekend!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.