Purusottam Mupunu

Cloudanix • 6K followers

LLMs are amazing at generating code. However, one of the biggest limitation of this code isn’t intelligence - it’s lack of context. Things get interesting (and messy) when it comes to large codebases. It falls apart and it's painfully obvious why. LLMs can generate working implementations from a high-quality prompt. At their best, they can produce a correct feature in one shot (“one-shotting”). This helps dramatically speed up development for clearly scoped tasks. This helps with: ✅ Productivity boost for implementation-heavy tasks - For repetitive CRUD operations or isolated scripts, LLMs can collapse hours of work to minutes. ✅ Enhanced prototyping — They help quickly explore solutions, enabling engineers to iterate design ideas faster. ✅ Lower entry barrier — Less context switching (e.g., from editor to docs), faster feedback loops. But these benefits fade when tasks require deep architectural knowledge, modular context, or long dependency chains, especially in large codebases. In this blog post Kieran Gill from Blueberry Pediatrics highlights some of the reasons it fails and how it can be improved. Here are a few reasons why LLMs fail with large codebases: ❌ Lack of global context: LLMs don’t “understand” the whole system - they only see what’s in the prompt. ❌ Hallucinations in unfamiliar parts: When missing architectural knowledge, LLMs may generate incorrect code, leading to bugs or regressions. ❌ Rework becomes more expensive than doing it manually: Failing to one-shot means repeated rounds — which erodes the time savings. Here's how you can solve the challenges: ▶️ Guide LLM using Prompt libraries that show Architecture, best practices, domain knowledge, etc. ▶️ Improve the codebase by making code modular, consistency in naming, and keep code clean. ▶️ Invest in review & automation for verification of LLM's design choices and end product (code, unit tests, etc.) In large codebases, the real productivity is gained by guidance + oversight + modular architecture. #ai #llm #aiengineering #engineering https://lnkd.in/gFzFunEV

Mario Larcher

Canva • 5K followers

It had been on my list for a while to read the SID-1 technical report after seeing a post about how using OpenAI-style messages in RL can be surprisingly dangerous in multi-turn settings with many tool calls. The first insight is about the messages abstraction. Converting a raw token stream into messages and then back into tokens is lossy because it changes how the exact byte sequence is tokenized. A concrete example is when the model generates a sequence of bytes that the tokenizer splits into two very common tokens with normal probabilities. After parsing and reformatting through the chat template, those same bytes can be re-serialized in a way that the tokenizer now maps to a single, very rare token that the model almost never produces. The reward is computed on the original tokens the model actually generated, but the log-probs used for the update correspond to this new token that was never sampled. Since this token has an extremely low probability, its log-prob has a large magnitude, and in policy gradient this translates into a disproportionately large gradient. That single artificial token can end up dominating the update, creating a feedback loop that gradually destabilizes training and eventually leads to collapse. There is a second, distinct effect happening at the same time. Malformed tool calls or slightly wrong formatting can get “repaired” by the parser and chat template, so bad rollouts end up looking syntactically valid before they ever reach the trainer. The environment sees a correct tool call, the reward is good, and the trainer also sees a correct sequence. The model never receives signal that it actually produced something malformed. Stability is preserved, but the model does not learn tool correctness because the fixing layer steals the learning signal. Their fix is conceptually simple. Use a strict Tokens-In Tokens-Out pipeline, where the trainer sees exactly the token sequence the model generated. No parsing, no chat templates, no message abstraction in between. Another very interesting part of the report is their analysis of the “length debiasing” proposed in Dr. GRPO and similar works, where the per-token advantage is no longer normalized by the rollout length. This assumes that rollout length and quality are unrelated. In tool-use and reasoning-heavy environments this is not true. Bad rollouts tend to be longer than good ones. In that case, removing the length bias makes the average per-token advantage negative. Over long runs, this slowly pushes down the logits of all sampled tokens while pushing up the logits of tokens the model never uses, until the model starts emitting garbage or out-of-vocabulary tokens and collapses.