Qwen3.8 Is Almost Here — and Agent Benchmarks Are More Fragile Than They Look

Hi everyone,

We have a countdown for the Qwen3.8 releases:

Next week is going to be a busy one.

Here’s my plan:

• Start my own evaluation of Qwen3.8 27B as soon as the model is released. This time, I’ll also cover agentic coding. I plan to publish the full analysis the following week.

• Run and report on evaluations of the GGUF versions by the end of the week.

• Publish a deeper analysis of the quantized models within 14 days of release.

This will be similar to what I have done for Qwen3.6.

I’ll also release my own quantized versions using my new quantization pipeline that brings together AutoRound (for high-quality quantization), LLM Compressor (for packing with a vLLM-friendly format), and a custom repacker (to correct bugs with some layers). The goal is to finally make mixed-precision models, with different layers quantized anywhere from 2-bit to 8-bit, work well with vLLM.

Conceptually, it brings some of the flexibility that GGUF provides for llama.cpp to the vLLM ecosystem, while targeting much higher throughput under heavy concurrency, for example, when running multiple sub-agents in parallel, by leveraging the Humming kernel.

I’ve already published Qwen3.6 27B variants built with this pipeline.

• Qwen3.6 WNA16

The 3.8 and 4.3 bit versions work well. I’m still evaluating the 3.5 and 3.0 bit versions. I’ll update the model cards once I’m sure they all work well. I’ll also add versions with a quantized language modeling head later this week. I thought about quantizing the token embeddings, but this isn't well supported by vLLM for the Qwen3.5 architecture (it doesn’t crash, but only generates “!!!!!”; that’s a bug, not a quantization quality issue).

Now I’m ready to do the same for Qwen3.8.

A very interesting test from Composio showed how much the agent harness can change the results, even when the model stays the same.

Composio@composio

We ran DeepSeek V4 Flash through 4 agent harnesses (Claude Code, Codex, OpenCode, Oh My Pi) on 30 agentic tasks. A different harness won on each metric: success rate, cost, and speed. 🧵🧵🧵

11:43 AM · Aug 6, 2026 · 75.1K Views

81 Replies · 51 Reposts · 740 Likes

Yes, it sounds obvious when stated like this, but I increasingly come across model evaluations claiming superiority over other models while relying on numbers produced with suboptimal agent harnesses for the evaluated models, or with harnesses configured in completely different ways (more turns, higher timeouts, …).

At best, these comparisons are useless. More often, they are actively misleading.

Composio ran DeepSeek V4 Flash through four different harnesses, Claude Code, Codex, OpenCode, and Oh My Pi, across 30 multi-step agentic tasks. The tasks involved live tools such as Gmail, Google Sheets, GitHub, Slack, Notion, Calendar, Airtable, and PagerDuty. A task was considered successful only if it passed every predefined check.

Observations

• Oh My Pi achieved the highest success rate, completing 17 out of 30 tasks. Claude Code and Codex each completed 16, while OpenCode completed 14.

• OpenCode had the lowest estimated cost per successful task at $0.073, followed by Codex at $0.081. Oh My Pi came in at $0.103, while Claude Code was the most expensive at $0.195.

• Speed produced yet another ranking. Claude Code had the lowest median completion time at 122.7 seconds, with OpenCode close behind at 129.7 seconds. Codex took 245 seconds, while Oh My Pi was the slowest at 272.4 seconds.

So, it seems we have different trade-offs:

• Claude Code was the fastest, but also the most expensive.

• Oh My Pi completed the most tasks, but was the slowest.

• OpenCode was the cheapest, but also had the lowest success rate.

But we also have another angle not tackled here: inference-time sampling. Rerun the same experiments, and you may get a very different picture. I agree with Composio’s general conclusion, but we need many more runs to conclude which is the cheapest, fastest, and most accurate harness.

This is also why evaluating models for agentic coding is extremely difficult, and extremely expensive to do properly.

I’m currently preparing an article focused on Laguna S2.1 and how I benchmarked it for agentic coding. I ran into far more issues than I would have liked to admit, but the process made one thing very clear to me: a large number of the agentic coding benchmark tables I see online should be treated with extreme caution.

Without controlling for the harness, its configuration, tools, prompts, retry behavior, and execution environment, comparing model scores can quickly become meaningless.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Share The Kaitchup – AI on a Budget

Have a nice weekend!