Software architecture, LLMs, algorithms, and engineering craft.
When you type a sentence into an LLM, the model never sees your letters. It sees a sequence of integers. Before any attention head fires or any logit is computed, a tokenizer chops your text into pieces called tokens and looks up each one’s ID in a fixed vocabulary. The word “tokenization” might become ["token", "ization"] → [3323, 2065]. Everything the model knows about language, it knows in…
In 1964, a Canadian scientific expedition landed on Easter Island — Rapa Nui — a volcanic speck in the South Pacific, 2,300 miles from the nearest continent. The island is famous for its enormous stone statues, the moai, which stand with their backs to the sea and look inward across a treeless landscape. The scientists weren’t there for the statues. They were looking for something stranger:…
In October 2016, the Nobel Committee awarded the Prize in Physiology or Medicine to a 71-year-old Japanese cell biologist named Yoshinori Ohsumi. The prize was for discoveries that, on the surface, sound almost counterintuitive: he figured out how cells eat themselves.
Every token you’ve ever received from an LLM was generated one at a time. No matter how capable the model, no matter how fast the hardware: one forward pass, one token, repeat. This constraint is so fundamental that it has a name — autoregressive decoding — and it is the dominant factor in LLM inference latency.
Every time you run a serverless function on AWS Lambda, execute a container on Google Cloud Run, or spin up a GitHub Actions workflow, your code runs on a physical machine shared with hundreds of other tenants. The only thing standing between your workload and theirs is a sandbox – an isolation boundary that determines what your code can see, touch, and break.
Tool call hallucination comes in three flavors. Your model calls search_orders when your tool is named get_orders. It passes user_id when your parameter is customer_id. Or it invokes web_search with full confidence even though you never registered that tool. Each failure looks different in the logs, but they share a root cause: the model is doing exactly what it was trained to do, and your tool…
When AI agents first started multiplying, every vendor shipped their own integration approach. Anthropic’s Claude connected to tools one way. OpenAI’s assistants connected another way. If you wanted two agents to collaborate, you wrote glue code. If you wanted your agent to talk to a database, you wrote more glue code. The ecosystem was useful but incoherent.
There’s a famous chart that shows a near-perfect correlation between the per capita consumption of mozzarella cheese in the United States and the number of civil engineering doctorates awarded each year. The line traces are almost identical. If you didn’t know better, you might think that eating more cheese somehow causes more people to get engineering PhDs, or vice versa.
There’s a tension at the heart of software that functional programming makes explicit: side-effects are what make programs useful, yet they’re also what make programs hard to reason about.
Software supply chain attacks don’t exploit your code—they exploit your trust. Rather than breaking through your defenses directly, attackers compromise the tools, registries, and packages you pull in and rely on. The attack surface is everything upstream of your own code, and it’s much larger than most teams realize.
Give a language model a few examples of a pattern — say, foo → FOO, bar → BAR, baz → — and it completes the sequence correctly without retraining. No weights change. Somehow the model reads the pattern and applies it. This is in-context learning: the ability to adapt to a new task using only examples in the prompt.
The container runtime is what you interact with. But the image — the artifact that gets built, pushed, and pulled — is governed by a specification most developers never read. Understanding it demystifies a surprising amount of the plumbing: why image pulls are fast, how multi-platform images work, why digests are immutable, and how tools like crane can copy images between registries without ever…
In a previous post, we looked at post-training as a category — SFT, RLHF, DPO — and contrasted it with in-context learning. But we glossed over the mechanics: when fine-tuning actually runs, what happens to the model’s weights? Which weights change? How much do they change? And why does it matter?
If you’ve spent any time working with large language models, you’ve probably encountered two very different ways of getting them to do what you want: post-training and in-context learning. They solve the same fundamental problem — adapting a general-purpose model to a specific task — but they do it in completely different ways.
In the first post of this series, we looked at how special tokens like <|im_start|> and <|im_end|> form the structural grammar of LLM conversations, and how the newer Harmony format extends this with tokens like <|call|> and <|return|> for tool invocations. In the second post, we established that built-in tools outperform function tools because they’re in-distribution – the model was trained on…
The AI tooling landscape has fractured into a bewildering number of SDKs, frameworks, and agents – each claiming to be the right way to build with large language models. OpenAI has an API SDK and an Agents SDK. Anthropic has a Claude SDK and Claude Code. Google has a GenAI SDK and an Agent Development Kit. Microsoft merged Semantic Kernel and AutoGen into a single Agent Framework. Then there’s…
Andrej Karpathy recently published microGPT – a complete GPT implementation in 200 lines of pure Python with zero dependencies. No PyTorch, no TensorFlow, no NumPy. Just math, random, and the raw algorithm. He also wrote an excellent companion blog post explaining the motivation and design.
In the previous posts in this series, we established that when you give an LLM a list of function tools, the model must interpret JSON schemas at inference time and produce structured output that conforms to them. We showed that built-in tools outperform function tools because they’re in-distribution, and we traced the token-level mechanics of how tool calls actually fire. But we glossed over…
When you connect multiple MCP servers to a coding agent like Claude Code, something specific happens to the model’s context window at every step of the reasoning loop. All tool schemas, from every server, are presented simultaneously as a flat list. The model must parse them at inference time, weigh them for relevance, and decide which – if any – to invoke. Add enough servers and the tool list…
Every AI agent framework — LangChain, LlamaIndex, Microsoft Agent Framework, CrewAI — wraps the same idea. Strip them down and you find the same beating heart: a while loop.
I was in my late thirties when I started paying attention to my health – not in a vague, someday kind of way, but urgently. I looked around and saw men in their forties and fifties whose bodies were failing them. Stiff joints, expanding waistlines, low energy, lives getting smaller. Some of them were younger than me.
In the previous post, we established that built-in tools like code_interpreter and web_search outperform custom function tools because they’re in-distribution – the model was trained on their exact invocation patterns during post-training. Custom function tools, by contrast, are out-of-distribution: the model encounters them for the first time at inference and must rely on in-context learning to…
When you give an LLM a list of tools, two things can happen. Either the model recognizes the tool because it was part of its training data, or it encounters the tool for the first time and must figure out what to do from the name and schema alone. This distinction – whether a tool is in-distribution or out-of-distribution – has a measurable impact on how well the model uses it, and understanding…
If you’ve ever looked at the raw token stream behind a ChatGPT conversation, you’ve seen things like <|im_start|>, <|im_end|>, and <|im_sep|>. These aren’t markup that gets rendered somewhere — they’re special tokens, atomic units in the model’s vocabulary that act as structural delimiters. They tell the model where one message ends and another begins, who’s speaking, and when to stop generating.…
Large language models are great at calling tools — but when a task requires chaining several tools together, the model ends up shuttling data back and forth, burning tokens on intermediate results it doesn’t need to see. mcp-compose fixes this by letting models write TypeScript that declares the composition, while the runtime handles the data flow.
It’s been a relief to move away from the bloat of WordPress. I decided to use Hugo and host all of the content in a private Github repository. I am even able to use Github Actions to manage the continuous integration/delivery of the source to Github Pages all entirely for free. While Github Pages, and Hugo both generate static html pages, I was able to enable comments via giscus which stores the…
As most of you probably already know WebSocket provides full-duplex communication over a single TCP connection. .NET 4.5 added support for WebSockets as part of the BCL. In this article I am going to talk about few of the subtleties that you need to think about.
Most software built today have a notion of a client and a service. This is even more true with mobile/web applications because you want your client apps to do as little as possible, and your service to do most of the heavy lifting. This allows you to improve your service without requiring constant client updates. Also since you have a single service that serves potentially various native clients…
Sebastian Junger (pronounced Younger) was on Bill Maher last night. Junger is an award winning Afghanistan war correspondent, and director, and his latest documentary Which way is the front line from here? has been proclaimed a success at the Sundance Film Festival this year. During the interview Bill asked why war felt like an addiction to some soldiers. In particular, what psychology drove this…
In most computer science programs, machine learning is usually a graduate level course. It’s a specialization within the field of artificial intelligence, which is often thought of as a theoretical study than practical applications. But yet, machine learning today is used heavily to solve problems. Our team for instance, uses it to build acoustic models for speech recognition. It’s no longer a…
Update (3/31/2013 - 11:41 PM PST):
One of the first things I did when I started this blog was to change the way static files are cached by IIS to improve site speed. Static files are things that aren’t going to change, things like CSS, JavaScript, images, documents, and so on. With IIS this is actually quite easy to achieve, you can start by making the following change to your web.config:
Came across this wonderful quote from Jeff Hammerbacher: “The best minds of my generation are thinking about how to make people click ads”. I won’t get into the details of what the implications of this are - but I think you can figure it out.
There are several ways to extend the behavior of a class. One of my preferred approach to extending behaviors is composition, and that is what we are going to talk about today. Let’s start with a simple example of inheritance:
Monitoring your Azure services couldn’t get easier. Windows Azure recently released a capability to test connectivity of your service (REST or website) from over three continents. It’s super simple to setup. Login to your azure management console, add your website URL to monitor and select up to three locations to monitor from: With that you can start getting response time, and uptime reports (my…
Here’s a cool technology you might not have heard of: scriptcs. Gives you the ability to use C# as a scripting language. It does this using the Rosyln compiler. So here’s an example below, taken directly from the scriptcs website:
We are at the finale of this three part series. In part I we discussed the MVVM design pattern, in part II we looked at the overall architecture and how the server side fits into the picture. In this article we are going to look at the client side of this and close with some final thoughts. The best place to start with the client is the HTML - so here’s the part of the HTML that’s interesting:
I’ve always been a junkie when it comes to markup validation. It’s important to me that my sites are 100% XHTML/strict compatible. But recently I came across a situation that I thought I simply couldn’t get around. In this article I am going to show you how you can add the X-UA-Compatible attribute and at the same time validate your pages through W3C validation. My website had the following tag:
Last time we looked at what the MVVM pattern was, and how it is used today in XAML-based applications. Today we are going to take a step further and build a mechanism to use the MVVM pattern on traditional web applications. Imagine how powerful it would be if your UI could evolve independently. We are going to build a simple web application that will display top 5 “memory intensive” processes on…
There seems to be two types of developers out there. One that believes in integration tests, and the other that believes in both unit and integration tests. Integration tests are tests that execute everything as if they were real “transactions” only to verify the expected results. For instance, if you were building a web application that managed a list of employees and their supervisors. Your…
In this series I will be discussing how the MVVM pattern can be used for almost any type of web application. MVVM (a variant on the MVP pattern) - is a really powerful pattern that can be used in UI applications. Traditionally these UI applications has been WPF and Silverlight. In this article I will show you how you can use the same principles to build web applications. With MVVM you have the…
If you or your kids want to learn how to program - you can start here: https://code.org/. For instance, just see this example. You have a great editor, where you can type and see results immediately on the right. On top of that, you have a video, with someone talking you through basic ideas like functions, and parameters. The site has support from top names like Bill Gates and Mark Zuckerberg. And…
So a cool new feature in .NET 4.5 is the ability to create anonymous methods that are asynchronous (async).
When dealing with .NET tasks – we often want to timeout the operation if it does not complete within a certain period of time. For instance, if you make an asynchronous WebRequest call – you may want to timeout this asynchronous operation within 3 seconds if you don’t get a response back. In the past, we had to create Timers to monitor the Task (or the Thread) for completion in the callback of the…
Before I go into the technical details of this article. I want to provide some context for this discussion. So we are using PRISM to develop a user-interface (UI) platform in the context of a Silverlight (but they apply to WPF also). The idea behind the UI platform is we will provide the basic functional look and feel and various plumbing (such as security, service encapsulation, messaging and…
If you’ve ever worked on an intranet site, you often want to know, programmatically, if a client that’s accessing your website has the ability to automatically login to your site (Integrated Windows Authentication). With Integration Windows Authentication, Windows can basically send (as a HTTP header) the currently “logged on” username. So your application can access this LOGON_USER HTTP header…
You can drastically improve your PowerPoint presentations by simply leveraging pptPlex! It’s a add-on for PowerPoint to build dynamic, contextual presentations. Contextual presentations allows your viewers to see the global context of your discussion, which in turn helps them to follow your line of thought.
The MultiView control, Wizard control or even a custom panel-based control is very useful to collect large sets of data in wizard form; and it’s often useful to give visual ques to the user as to which step he/she is in. Something similar to:
I presented a topic on jQuery at our campus web publishers meeting. Here are the sample pages and the PowerPoint presentation. I will talk more discussions on jQuery and how it has helped us to build a powerful web-application next week.
Generally all programmers are lazy–and specially when it comes to documentation because it takes so much time and doesn’t necessarily produce any material benefit. GhostDoc1, designed by Roland Weigelt, is a plugin for Visual Studio 2005 that automates the generation of XML comments. For example if you have the following C#1 function: