<div style=...
Intro If you're active at all when it comes to language model research these days, you almost certainly have an implicit mental model for what kinds of training have been useful for capability improvement in the post-o1 era. The paradigmatic shortlist is essentially as follows: Pretraining , which targets diverse conditional prediction on natural webtext (or its synthetic derivatives).…
I think there is a serious anti-pattern that sometimes shows up in papers doing online reinforcement learning at scale. DAPO is a paper from ByteDance that proposes something that, on the surface, seems fine and practical; overly long samples during RL are undesirable, so we should mask them out from contributing to the loss. This is secretly introducing something that you never want to introduce;…
Hi! This is kalomaze. In this post, I'll be covering LoRA training and its recent incorporation into prime-rl for both SFT and RL finetuning, including practical implementation details & experimental training results for some of our RL environments. When to use LoRA? A recent blogpost from John Schulman caught the attention of many prominent reinforcement learning practitioners; that is, "LoRA…
Author's Note This is an experimental piece - I recorded an hour long voice memo while on a walk, talking to myself out loud, stream of consciousness style. Some things are probably worded quite messily, but it's 3 times longer compared to what I've written before, and I've found that cleaning up the transcriptions is a significantly faster process compared to writing from scratch (what I've done…
Introduction Many GRPO reproductions for LLM reinforcement learning available online lack useful intuitions or recommendations regarding hyperparameters and reward shaping. Most examples focus on GSM8k or similar math problems - tasks with clear right/wrong answers. Additionally, demonstrations seen so far primarily use models smaller than ~7B, and don't demonstrate realistic failure modes when it…
DeepSeek. Everyone in the LLM space knows that name now, and for good reason; after a little over a year of their team quietly iterating on architecture, engineering, & data work, they've finally caught up to [and in some aspects surpassed] the frontier with the release of DeepSeek r1, fresh off the heels of their latest pretraining run (DeepSeek-V3). When OpenAI released the first checkpoint that…
Note: This is a repost of an older rentry page: https://rentry.org/corruption_model_writeup I've been building a custom model specifically tailored to take input text that has been "corrupted" in specific sections. The actual "corruption" looks like UTF8 character substitution, which my finetuned Qwen7b has learned to reconstruct the original from (as best as it can, given the surrounding…