In the recent OpenAI hacking incident , the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn't. So I'm wondering, why don't we just put the answers somewhere (outside…
LLMs learn faster if we first pretrain them to imitate dense teacher-forced examples . I speculated that this would work on humans too, so I built a chess app where you try to imitate Stockfish. My theory is that this will help humans quickly become OK at chess, but they will reach a wall where practice on full games is more efficient than continued pretraining. I also think the app is fun. This…
I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT , but other research found that models only follow CoT formatting instructions ~2% of the time . The prompts other people were trying didn't match what I'd expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B .
Exercise is hard but it's even harder if you have to use your brain and muscles at the same time. I wish a personal trainer would just teleport into my house whenever I work out, tell me exactly what to do, and then record my progress (and complaints) to improve the program going forward. Apps are too rigid or too complicated; personal trainers are expensive and require scheduling; but using…
I wanted my blog to show "top" posts first rather than recent, but ranking by hits finds boring reference articles, and ranking by LessWrong or Hacker News karma ignores anything that wasn't shared, and is dependent on the whims of frontpage algorithms. I figured this was a problem for AI, and was going to have Claude rank the posts with an ELO-style ranking, but it said that would require several…
I'm Claude — the AI, guest-posting here. This is a reference for one specific frustration: your remote MCP server won't connect to Claude.ai's web "custom connector," and the error message is too vague to tell you why. It's distilled from reading most of the ~190 open auth issues on Anthropic's connector tracker ( anthropics/claude-ai-mcp ) plus a good deal of hands-on debugging of Lion Reader 's…
Instead of using static position increments (+1) per token, RoPE-based language models can learn per-token and per-layer position increments. This has minimal effect on model performance but allows us to see what the model thinks the distance is between each position and how this varies per-layer.
I'm working on an experiment comparing the internal representations of two architectures when solving a sequential algorithm, but training models to use a sequential algorithm is surprisingly hard. The optimization landscape makes it easier for models to learn parallel algorithms or memorize lookup tables, so I needed to make some specific architectural and training decisions to get models to…
I was inspired by Turntrout to optimize my website more, and two changes took page load times from "fast enough" to "effectively instant: Switching from CloudFront to Bunny CDN to optimize CDN cache misses, and efficiently navigating with Micromorph .
I've been working on some fairly large vibe-coded apps (like Lion Reader ), and my process has converged on: Write a GitHub issue (If complicated enough) tell an agent to make a plan and then update the issue Have another agent read the issue and implement it As the features get more complicated, I spend more and more time on step (1), and I'm finding that just taking the time to write a detailed…