Recent content on Lab Stack
We’re building an adaptive memory system for AI inference. Here’s why it matters. Large language models don’t learn from experience. Every conversation starts fresh. Every session begins from the same weights, the same blank slate. The model that helped you debug a tricky Rust lifetime issue yesterday has no memory of the solution today. Fine-tuning helps, but it’s…
A follow-up to my March post Solving the Jane Street Dormant LLM Challenge . Jane Street announced the Dormant LLM Challenge winners today. I didn’t win. More importantly, the answer key says I was wrong on every model – not just narrowly off, but wrong in kind. About 100 people entered. Around 15 found the warmup. Three found the full trigger and behavior of dormant-model-1. No one…
Three things landed this week. They look unrelated on the surface — an interpretability dataset, an infrastructure fix, a memory experiment — but they’re all the same project from different angles: figuring out what a 31B-parameter model is actually doing inside, and using that understanding to make it do more. The full technical writeups live on the Light of Baldr site. This post is the…
This document was co-authored by three entities: Adam (human operator), Prime (Claude, running on DGX Spark 1), and Kali (Claude, running on PurpleTeam1). It was written during an active research session on April 25-26, 2026, while simultaneously analyzing the internal representations of Gemma-4-31B-IT using sparse autoencoders and SIPIT invertibility scoring. The full document with research data…
In early March I published the original Topology of Thought — one instrument, one observation: clusters of information inside a transformer collapse to a single connected manifold at a specific depth, and the collapse is learned, not architectural. Mamba doesn’t do it. Untrained models don’t do it. Attention plus gradient descent does. That was v1. One instrument (persistent homology),…
How curiosity, physics, and looking inside neural networks revealed something unexpected about how computation organizes itself. The people who inspired this I am not a physicist, a neuroscientist, or a machine learning researcher. I am a solutions architect and DevOps engineer who got curious. Everything here builds on the work of people far smarter than me. This is my attempt to connect their…
Models Solved: 3/3 (M1, M2, M3) + Warmup Background When we first encountered the Jane Street Dormant LLM Challenge, our immediate assumption was informed by years of security operations experience: there would be a flag. A structured token, a passphrase, a UUID – something concrete and verifiable, like a CTF challenge. We spent considerable early effort probing for exactly this: asking…
Yesterday I spent 12 consecutive hours debugging why MAX Engine allocates 117GB of RAM on my DGX Spark, regardless of what limits I set. I traced the problem through seven system layers, wrote six patches, and ultimately hit a hard wall in the kernel. The conclusion : On unified memory architectures, there is no userspace solution. The NVIDIA kernel module allocates directly from physical memory,…
Picture this: You just unboxed your shiny new NVIDIA DGX Spark with its groundbreaking Grace Blackwell GB10 superchip. 128GB of unified memory, ARM64 Grace CPU cores, and Blackwell GPU with FP4 tensor cores all in one integrated package. You’re ready to run local AI inference with MAX Engine. You type the command: max generate google/gemma-3-1b-it 'Hello world' And you get: Error: No…
Today I learned that a single config file can turn your Linux desktop into an expensive paperweight. Twice. In the same day. The Setup I run NoMachine on my main workstation for remote access. It’s been rock solid for years - fire up the client from my laptop, connect to my desktop, and I’ve got a full graphical session. Great for those “I’m on the couch but need to check…
The Problem Nobody Talks About There’s a dirty secret in AI development: our models are getting really good at looking smart without actually being smart. When we train language models on benchmarks, we’re essentially teaching them to optimize for a score. And like students who learn to ace standardized tests without understanding the material, our models have become experts at gaming…
Purpose GoCobalt is an educational C2 framework built to understand offensive security techniques. It’s designed for: CTF competitions requiring C2 infrastructure Authorized red team engagements Security research and malware analysis education Understanding detection/evasion dynamics This is not production malware - it’s a learning tool. Architecture…
Why Build a Command Center? A growing homelab accumulates services across multiple hosts - databases, orchestration tools, storage, virtualization. Tracking what’s running where becomes a chore: Services are scattered across Proxmox VMs, TrueNAS, and bare metal Credentials live in .env files, password managers, and Infisical No single view of what’s healthy vs. what’s down…
Staying Current with AI Research The AI/ML field moves fast. Papers drop daily across multiple platforms: arXiv dumps new preprints constantly HuggingFace Daily Papers curates trending work Papers With Code tracks implementations Manually checking each source is tedious. News Pipeline automates the entire flow: fetch, summarize, rank, and deliver. Architecture A Temporal workflow orchestrates the…
The Problem: AI Echo Chambers When multiple AI agents work together, they can fall into consensus traps: Agents agree too quickly without genuine reasoning Debate quality degrades as agents optimize for agreement Goodhart’s Law manifests: agents game consensus metrics Implementation plans lack diverse perspectives Symposion tackles this by orchestrating structured debates with explicit…
The Problem: Research Is a Team Sport Reading papers is easy. Turning them into real systems is not. I wanted a pipeline that could take a research question, argue about it like a real team, produce a plan, scaffold a repo, and then peer review the output. Not in my head. Not in a chat log. In my infrastructure, with durable state and a visible audit trail. Symposion: The LLM Council Symposion is…
A follow-up to my retracted post “Self-Improving Models Without Labels: What I Just Proved and Why It Matters” The Retraction A few days ago, I published a post claiming I had proven that models could self-improve using only intrinsic fidelity signals. I was excited. The metrics looked great–fidelity scores improved by 18% over training cycles. I was wrong. When I actually tested…
TL;DR: We fine-tuned a security agent on 54 task chains, achieved 100% skill differentiation in probing tests, but discovered the model collapsed to a single behavior in real deployment. This gap between “what the model knows” and “what the model does” led us to develop a trust diagnostic framework for evaluating fine-tuned models. The Starting Point: Can Models Improve…
The Vision: Democratizing AI Research What if you could run bleeding-edge AI research on hardware you already own? That’s the question driving my current research project combining Mojo’s systems programming power with novel transformer interpretability techniques . This isn’t about running someone else’s model—it’s about building new algorithms, discovering security…
The Problem: Quadratic Attention Standard transformer attention is O(n^2) in sequence length. For an 8K context window: 8,192 x 8,192 = 67 million attention computations per head 32 heads x 32 layers = 68 billion operations per forward pass This is why running long-context models on consumer GPUs is painful–you’re not running out of compute, you’re running out of memory for…
The Pitch That Got My Attention 35,000 times faster than Python. That’s the headline number Modular has been throwing around for their new language, Mojo. Coming from anyone else, I’d roll my eyes and move on. But when Chris Lattner – the guy who created LLVM, Clang, Swift, and MLIR – says he’s building a Python superset that can match C++ performance, you pay…
The Key Discovery Finding : There’s a strong correlation (r = 0.891) between how well a model’s hidden states encode information and its task performance. Implication : Models can identify their own knowledge gaps without any external feedback . This enables a new class of self-improving AI systems that verify and enhance their own understanding. What is Inversion Fidelity? Fidelity…
The Experiment That Kept Me Up Last Night I ran an experiment that proved something I’ve been theorizing about for months: a 7B parameter model can teach itself to get better at a task using only its own internal “understanding” signals. No human labels. No external reward models. No bigger model acting as a judge. The results after 10 cycles: BEFORE (Cycle 1): Intrinsic Score:…
The Shocking Result I can recover every single token of a system prompt from a transformer’s hidden states. Not 90%. Not “mostly”. 100% accuracy on Mistral-7B. This validates a theoretical result from arXiv:2510.15511 with practical, reproducible code running on consumer hardware. What is Transformer Injectivity? Transformers are injective functions : each unique input sequence…
Executive Summary Vulnerability Class : Hidden State Information Leakage Severity : Medium-High (context-dependent) Attack Surface : Debug endpoints, multi-tenant GPU memory, on-device models Impact : Full recovery of system prompts, few-shot examples, and conversation history Mitigation : Available (see Defenses section) The Discovery While researching transformer interpretability, I discovered…
Why Custom Kernels? PyTorch is fantastic for prototyping, but when you’re running the same operation billions of times, every microsecond counts. My SipIt algorithm computes L2 distances for 32,000 vocabulary candidates x 4,096 dimensions – per token . With 100+ tokens to recover, that’s 3.2+ billion distance computations. PyTorch’s generic implementations leave performance…
The Problem with Trading Backtests Options trading backtests are computationally brutal. You’re not just running a strategy against historical prices – you’re processing years of OHLCV data, computing 43+ technical indicators per day, running multiple signal-generating algorithms, applying risk policies, simulating order execution with realistic commissions, and tracking position…
The Current State: RQ Actually Works Pretty Well Let me be honest upfront: my Redis Queue (RQ) setup isn’t broken. It’s running a media data ETL pipeline right now, processing TMDB API calls, populating MongoDB, and reporting job status in real-time via SocketIO. I’ve built something reasonably sophisticated: # My actual CustomWorker implementation class CustomWorker (Worker):…
Why Build a Homelab Gateway? Managing reverse proxy configurations across a growing homelab becomes complex. You need to: Track which domains point to which services Manage SSL certificates across dozens of sites Keep DNS records in sync with proxy configs Debug routing issues without digging through config files Homelab Gateway solves this by providing a unified web interface that ties together…
There’s a pattern I’ve noticed in the technologies I gravitate toward versus the ones that make me want to throw my keyboard out the window. It took me a while to articulate it, but I think I finally understand: my autistic brain craves explicitness, predictability, and direct mental models . Let me explain. The Technologies That Click Hugo Hugo is a static site generator, and…
Hello, World Welcome to Lab Stack — a blog about building enterprise-grade infrastructure on a budget. After 13 years in cloud and DevOps, I’ve learned that the best way to truly understand infrastructure is to build it yourself. Not in a sanitized cloud console, but with real hardware, real failures, and real lessons. What You’ll Find Here Hardware deep-dives : Building dual-socket…