I Talked with Arcee AI for 100 Minutes and Everyone Needs to Know This

I sat down with @latkins (CTO) and @stochasticchasm (Pre-training Lead) from Arcee AI for about 100 minutes, and honestly this ended up being one of the most technical conversations I’ve had on GroundZero.

Here are the things that stayed with me.

This came up early and kept coming back throughout the conversation. Lucas has this philosophy — pick extremely difficult things you have no business trying to accomplish, and then bang your head against the wall until you figure it out. He was describing how Varun went from a software engineering job to leading pre-training on a 400B param model in roughly a year.

Varun’s story is actually quite grounded. He’d been interested in AI since 2017, took ML and distributed systems courses during his masters, and happened to be in a deep learning for NLP class when ChatGPT launched. His prof was apparently losing his mind showing it off every lecture. His first real pre-training run was AFM-4.5B, a dense model on 8T tokens. “Trial by fire”, as Lucas mentions. He was basically soloing the infrastructure work on AWS with maybe three other people helping.

watch the full pod here:

himanshu@himanshustwts

The Arcee AI Podcast is here! In this episode, @latkins and @stochasticchasm join us to discuss the story of Trinity models and everything frontier. I can say, this talk has been one of the most amazing and technical conversations we've had on Ground Zero. 0:00:00 - Intro

1:53 PM · Apr 8, 2026 · 27.3K Views

11 Replies · 32 Reposts · 190 Likes

The popular version of this story is that Arcee hit a “ceiling” in post-training and realized they needed to go deeper. Lucas actually pushed back on that framing a bit. He said if he could rewrite the manifesto, he wouldn’t phrase it exactly that way.

The real reasons were more structural than technical. Three things converged.

First, the geopolitical shift. Before DeepSeek-R1 came out, enterprise customers never asked what base model Arcee was starting from. They didn’t care. After R1, compliance teams and legal teams suddenly had very strong opinions about not using Chinese base models. Which meant if a customer wanted something under 14B parameters and wouldn’t use Qwen, Arcee’s only option was a Llama that hadn’t been updated in nearly a year. That’s a bad position to be in.

Second, the business model problem. Fine-tuning as a service is inherently hands-on. Every customer is different. To grow revenue 30x, you’d need to grow your field engineering team 30x. Head count scaling one-to-one with revenue is not what venture-backed companies want to hear.

Third, the capability ceiling. This one is more subtle. Lucas framed it as — if some new frontier coding model comes out and you want to match it at 10B, you don't want the limiting factor to be that your base model just hasn't seen enough code tokens. When you own pre-training, the question changes from "can we do it" to "how long will it take and how much will it cost." It becomes a when-not-if problem, which is a much better place to operate from.

I asked about Ilya’s comments on pre-training saturation. Varun’s answer was sharp. The old framing was that we’re running out of web text. And sure, if you’re just doing vanilla web crawl pre-training, there’s a wall. But pre-training itself has evolved.

Synthetic data is the obvious breakthrough — Datology has a paper on rephrasing, which Varun described as a way to upsample high-quality information. The analogy he used was from diffusion models: a masked diffusion objective lets the model see data in different ways, kind of like data augmentation. Rephrasing does the same thing for text. You’re presenting the same information about the world but through different expressions. The model sees it from multiple angles rather than memorizing a single form.

The other shift is that the boundaries between training phases are blurring. More instruct data and agentic traces are showing up earlier in the pre-training mix. Varun mentioned a paper from Datology showing that integrating domain-specific data earlier in training led to better downstream performance and less catastrophic forgetting during fine-tuning. His take is that eventually all of this just gets absorbed by pre-training. The pure web text model is slowly going away.

Lucas added something I found interesting — they have never finished a single run where they didn’t think “if we immediately went back and did that again, we could make it 30-40% better.” There’s always something to improve. After four models, they feel like they’re just getting started. Qwen has shipped upwards of 120 or 130 models across generations. The sky is still the limit.

Trinity Large is a 400B sparse MoE with 13B active parameters per token, routing 4-of-256 experts. That's a 1.56% routing fraction — extremely aggressive compared to DeepSeek-V3 at 3.13% or Qwen3 at 6.25%.

What I found interesting is that almost every architectural decision was driven by stability concerns, not by trying to be novel. Lucas was explicit about this: it does not have to be new, it does not have to be novel, but it does have to be good. He described "usefulness per token" as a metric he's trying to get better at quantifying. If they pre-trained a crazy new non-transformer architecture and got decent results, it would be cool but not useful. As they're a venture-backed startup, they need to generate revenue from what they build.

Lucas told two debugging stories that I think anyone doing training at scale should hear.

The first: during Trinity Mini training, the run would crash deterministically at the same step. Restart from checkpoint, hit the same step, crash again. Skip that step by fast-forwarding the data loader, no crash. So it was data-related. But why would specific data crash the training? They ablated everything. Nothing worked. Lucas said his rule of thumb is now — if you’ve exhausted all other options and have no idea, look at the tokenizer. Their early tokenizer was choking on extremely long sequences of numbers in the pre-training data, causing it to hang and then crash.

The second: during AFM-4.5B, loss wasn’t going as low as expected and evals were off. They swapped their custom tokenizer for Llama’s tokenizer and things worked. Tokenizer issue.

The meta-lesson: lower your search space when debugging. Don’t try to solve it across the entire training pipeline. Isolate variables. If swapping data fixes it, it’s data. If swapping the tokenizer fixes it, it’s the tokenizer. It might be simple in theory but hard when each experiment at 400B scale costs $30-50k.

They used the Muon optimizer instead of AdamW. The Moonshot paper claimed 2x compute efficiency. I asked if they saw that at 400B scale and 17T tokens. Varun’s answer was refreshingly honest — it’s impossible to say because they couldn’t run a full AdamW vs Muon comparison at that scale. He cited one paper that found only a 10% improvement and the Moonshot paper that found 2x. His conclusion: it’s unsettled.

Lucas added something broader that I think applies to a lot of decisions in this space. He said there’s an impulse when you’re struggling with performance to reach for a new optimizer or a new training regime as a fix. And while it might work, if you’re using it to band-aid over a lack of fundamentals — stable training, good data, not burning the model with your learning rate, maximizing throughput — you’re going to have to go back and fix those fundamentals eventually anyway. So you might as well do them upfront.

On the Muon variant explosion (Muon-NSR, Muon-VS, MuonEq, AdaMuon, NorMuon), Varun said they’ve only tested on small experiments so far. NorMuon looks promising. But the philosophy is pragmatic — don’t switch paradigms to solve a problem that better fundamentals would solve.

I asked about mid-training — there’s a lot of noise about it right now. Lucas’s definition was blunt: “it’s the training you do in the middle.”

More specifically, for Arcee it’s two things. First, the last phase before post-training where they do their learning rate anneal — warmup-stable-decay schedule, so most pre-training runs on a flat constant schedule, then they decay whenever they want. This phase uses their highest quality data blend. Second, after context extension, they go back and do an additional training phase. For Trinity Large specifically, this was about rebalancing experts — after extending context, you can measure how unbalanced the 256 experts are and then do further training to fix that before handing off to post-training.

Varun added that Chinese lab reports often include a phase after long context extension where they expose the model to more agentic and long-form workflow data. His prediction: this will formalize over time, with more and more gradual distribution shifts across multiple phases leading up to the final model capability.

Lucas made a point that I think more people need to hear. Models are getting extremely good at extremely hard benchmarks — frontier math problems that 0.001% of the population can understand. Great indicator of how smart the models are getting. But the usefulness to an average person is actually quite low.

His example: a model might get 20% better at frontier math, but only a couple percent better at the thing you actually care about, like generating TypeScript for slide decks. The improvements in day-to-day work are getting smaller because the models are already quite good at those tasks.

His take is that individual users need their own eval sets — even just 20-30 questions that represent how they actually use these models. Because the reported evals from labs are going to keep measuring intelligence at levels that are increasingly irrelevant to what most people do with these tools.

When I asked about RL environments (the whole thesis from Scale AI and Wing VC that the bottleneck has shifted from data to environment design) Lucas was honest. He said he’s excited to actually get to focus on it. For the last eight months, there’s always been a model training that was consequential to the company’s future. They had to make decisions that worked for the next day (has to be early!)

For Trinity-Large-Thinking, they used a mix of environments from providers, pre-existing ones from places like PI, and some they built themselves. Not all the provider environments made it into the final run. And the harness bias question, whether using OpenCode for dataset construction means the model works best with OpenCode, Lucas said no. They ended up using much less of the OpenCode traces than originally intended, and they found you can rewrite prompts from one harness format to another to generalize trajectories.

The ceiling on RL environments is unbelievably high, Lucas said. And the ceiling on the model itself is high too. This is not the best possible Trinity Large. It’s the best they could do after pivoting from pre-training to post-training in two months.

Lucas was direct — they’re trailing Chinese competitors. Even though Trinity’s benchmarks were competitive, there are areas where they need to improve drastically. They can’t let up.

But he pushed back on the framing. The closed providers are about to get massive amounts of new compute, so if you measure the gap purely as “how many months behind on evals are open models,” that gap might actually get larger. The question is whether that matters.

His argument: the models are getting good at things 99.99% of humanity has no need for. What most people need is fast, reliable, cheap, and proficient for their specific use case. And if you look at OpenAI, every major product they’ve shipped in the last year uses a custom variant. If OpenAI has concluded that the best way to make sticky products is merging customization of a strong base model with a powerful UX and harness, everyone else should pay attention. And to do that, you need strong open models that are good at being customized for downstream domains.

The 3.37 trillion tokens served on OpenRouter in two months? It helped that Preview was free, Lucas admitted. But there are lots of free models on OpenRouter that nobody uses. The timing mattered. An instruct model that answers immediately, right when OpenClaw was exploding, turned out to be exactly what people wanted. A breath of fresh air compared to waiting three minutes for chain-of-thought reasoning.

Varun’s biggest blocker right now: GPUs. He said he wouldn’t have expected H200s to suddenly go out of existence, but they have. His most exciting thing for the next year: more data centers and compute capacity coming online. Lucas joked about Varun becoming a “flop head” (GPUs, GPUs, GPUs.)

For Arcee specifically, 2026 is the year of infrastructure. They need a consistent compute cluster for persistent experimentation and ablation. They need to scale their team. They need to get their multimodal people actually working on multimodal instead of being borrowed for text work.

A few things keep bouncing around in my head since this conversation.

The first is that the gap between “understanding what to do” and “actually having done it” is enormous in pre-training. Varun and Lucas can cite specific papers and paragraphs about load balancing techniques, but when those techniques didn’t work at their sparsity level, they had to figure it out themselves in ten days with the clock running and $50k experiments. Reading papers gets you to the starting line. Actually training models is where the learning happens.

The second is that usefulness and intelligence are genuinely different things, and the field hasn’t internalized this yet. A model that thinks for 50,000 tokens before telling you its name is intelligent by any reasonable measure. It’s also useless. The models that people actually love using are the ones that answer fast and do what you ask reliably. The benchmark race and the usefulness race are diverging.

The third is about constraint. Arcee did this for $20 million. Four models, six months. No massive compute commits, no ability to ablate at scale, time-locked GPU reservations. That constraint shaped everything like the architecture choices were about stability not novelty, the debugging had to be fast, and they couldn’t afford to chase every new paper. The constraint didn’t make the models worse. In some ways it made the decisions sharper.

And the last thing is something Lucas said early on that I keep coming back to: you can just do things. Pick something impossibly hard, try it, fail, try again. Pretty much this is how Varun went from software engineering to leading pre-training on a frontier model.