Recent content on Amit Bahree's (useless?) insight!
The Stack Below the Stack , a 3-part series on how modern LLM inference actually works, told through a single DeepSeek V4 dtype bug. Part 1 · Physics of a request : why the first token is a different problem from every token after it, and why batching exists. Part 2 (this post) · Below Python : what actually runs under vllm serve , and why the escape hatches failed. Part 3 · Serving at scale :…
The Stack Below the Stack , a 3-part series on how modern LLM inference actually works, told through a single DeepSeek V4 dtype bug that crashed a two-node deployment at startup. Part 1 (this post) · Physics of a request : why waiting for the first token is a different problem from streaming the rest, and why batching exists. Part 2 · Below Python : what actually runs under vllm serve , and why…
I am excited to announce that my new book, LLM Customization and Fine-Tuning: Adaptation, Distillation, and Alignment , written with Weehyong Tok , is now available in Early Access (MEAP) from Manning Publications . 📚 A general-purpose model API is a fine place to start. Sooner or later, though, most teams need something an API cannot provide: a model that speaks their house terminology and calls…
The earlier cluster benchmark write-up asked which frontier models were fastest. This post asks the next question: once hardware, serving stack, and benchmark shape are held constant, what do deployment choices - weight quantization and KV-cache compression - actually do to real coding-agent behavior? The model family stays fixed; the things operators actually touch get varied. This grew out of a…
Over the last several benchmark cycles, I kept coming back to the same practical question: once you hold the hardware and methodology constant, which large open models are actually pleasant to serve, which ones merely load, and which ones become operationally awkward the moment you move beyond a demo? This post is a technical deep dive into that question. Instead of presenting a generic…
5-Part Series : Part 0: Why build an OS from scratch? Part 1: Foundations Part 2: Communication Part 3: Concurrency Part 4 (this): Memory and beyond GitHub Repository : bahree/rust-microkernel - full source code and build scripts Docker Image : amitbahree/rust-microkernel - prebuilt dev environment with Rust, QEMU, and source code Recap from Part 3 : we added timer interrupts and preemptive…
5-Part Series : Part 0: Why build an OS from scratch? Part 1: Foundations Part 2: Communication Part 3 (this): Concurrency Part 4: Memory and beyond GitHub Repository : bahree/rust-microkernel — full source code and build scripts Docker Image : amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code Recap from Part 2 : we built message-passing IPC with a mailbox…
GitHub Repository : bahree/nanoclaw - full source code NanoClaw is a headless AI assistant running on my personal server. It processes messages from WhatsApp, Telegram, and Slack, runs scheduled tasks, and manages conversations with Claude agents in isolated containers. It’s been incredibly useful, but it had one major pain point: no visibility into what it was doing or why. If something…
5-Part Series : Part 0: Why build an OS from scratch? Part 1: Foundations Part 2 (this): Communication Part 3: Concurrency Part 4: Memory and beyond GitHub Repository : bahree/rust-microkernel - full source code and build scripts Docker Image : amitbahree/rust-microkernel - prebuilt dev environment with Rust, QEMU, and source code Recap from Part 1 : we have a bare-metal kernel that boots on…
5-Part Series : Part 0: Why build an OS from scratch? Part 1 (this): Foundations Part 2: Communication Part 3: Concurrency Part 4: Memory and beyond GitHub Repository : bahree/rust-microkernel — full source code and build scripts Docker Image : amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code You’re about to write code that runs with nothing underneath…
5-Part Series : Part 0 (this): Why build an OS from scratch? Part 1: Foundations Part 2: Communication Part 3: Concurrency Part 4: Memory and beyond GitHub Repository : bahree/rust-microkernel — full source code and build scripts Docker Image : amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code Why this, why now? I recently wrapped up an incredible chapter at…
TL;DR In this final part of our 4-part series on building language models from scratch, we explore the evaluation, testing, and deployment pipeline that transforms our trained historical language models into working systems. Part 1 showed you how to use the published models, Part 2 covered data collection and custom tokenization, and Part 3 detailed the model architecture and training…
Deadlock by design: two vibes, two locks, zero unlocks. In vibe coding, that’s not a bug - it is a feature. 😎 Here’s what happens when agreement-first engineering meets C++ and mutexes: #include <iostream> #include <mutex> #include <stdexcept> std :: mutex mVibes, mProd; void shipToProd ( bool agree) { // We lock the vibes and production—because feelings // and facts both need exclusive access.…
TL;DR In this third part of our 4-part series on building language models from scratch, I explore the complete training infrastructure that transforms our clean historical data and custom tokenizer into working language models. Part 1 How to build a Large Language Model from Scratch - covered using the published model Part 2 Building LLMs from Scratch - Part 2: Data Collection & Custom Tokenizers…
I 💖 Diwali. For those celebrating, from my family to yours!
TL;DR In this second part of our 4-part series on building language models from scratch, I explore the two foundational areas of LLM development: data collection and custom tokenizer creation. Part 1 - Building LLM from Scratch covered using the published model; here, we build the complete pipeline from raw historical documents to a custom tokenizer that understands archaic English, London…
TL;DR In this post, I show how to build a working LLM from scratch and show a complete end-to-end pipeline from data gathering to training to deployment of a language model. For this project I concentrate on Old English and only related to London, using historical London texts (1500-1850). To show the flexibility, I built two language models which are identical in architecture and the only differs…
TL;DR As part of my role at Microsoft’s AI Foundry Applied AI engineering team in CoreAI, I have participated in numerous detailed discussions about the evolving landscape of AI models. In conversations with many customers, from CxOs to engineers, one recurring topic is the rise of reasoning AI models . These models are designed to perform complex tasks by explicitly breaking down problems…
TL;DR: KV cache is a memory optimization central to efficient LLM inference. It enables faster, longer, and more cost-effective generation by caching previously computed attention keys and values—unlocking the practical deployment of models like GPT-4o, Llama 3, etc. 1. Introduction Generative AI, powered largely today by Large language models (LLMs) such as GPT-4o, Llama 3, etc., is transforming…
1. Overview Rust has been gaining attention recently due to its unique combination of performance, safety, and modern programming features. Its strict ownership model eliminates common memory issues like null pointer dereferencing and data races, providing a secure environment for developers. At the same time, its expressive syntax and focus on developer productivity make it a strong contender for…
AI is advancing at an unprecedented pace, with Mixture of Experts (MoE) models being one set of model architectures at the forefront of this revolution. These architectures enable breakthroughs in efficiency and scalability by leveraging a modular design where only a subset of specialized “expert” networks are activated for each input. MoE architectures have become a cornerstone in…
1. A little background I have been meaning to automate the deployment of my blog post to a dev server (running locally) for a while, but I haven’t had the time to get around to it until now. In addition to deploying this, the dev server also had several constraints. It is one of the machines at home and is not exposed directly to the internet. I also don’t have any ports opened on the…
The one thing I wanted to do after my book Generative AI in Action was complete was to create a summary in natural speech and possibly use TTS (Text-to-speech) to create an audio summary—think of it as a podcast that is easier for people to consume and get a quick sense of what the book is about. TTS (Text to Speech) or not to TTS? Initially, I was inclined towards using TTS (Text to Speech) for…
In today’s rapidly evolving tech world, mastering Generative AI isn’t just an advantage—it’s a necessity. Are you ready to harness its power to transform your business and solve real-world challenges? I’m excited to announce that my new book, Generative AI in Action , is now available in print and ebook formats from Manning Publications . 📖 Special Launch Offer 🌟 As a…
I have been running a couple of instances of Teslamate - one locally on a server at home and another in Azure in a Ubuntu VM (see 👉 this blog post for details). I have been backing up the data to a NAS and then an offsite backup for the local instance. For the Azure instance, I have been running various backups during the day and backing up the data to OneDrive. This allows me to have a data…
We released Phi-3 recently, which builds on Phi-2 ( read more on that here ) and it is a great model to use for various tasks. In this post, we will show how to run Phi-3 locally including a demo of it running on a phone. There should have been a video here but your browser does not seem to support it. 1. What are Small Language Models (SLMs)? Before diving into running Phi-2 locally, let’s…
1. What are Small Language Models (SLMs)? Before diving into running Phi-2 locally, let’s take a moment to understand the concept of small language models (SLMs) and their significance in natural language processing (NLP). A SLM is a type of AI model that has been trained on a massive dataset of text but is limited in terms of its size and capabilities compared to a Large Language Model…
How many prompt engineers does it take to change a light bulb? Just one, but first, they need to fine-tune the model to make sure the AI doesn’t end up writing a poem about darkness instead. DALLE generated image of How many engineers it take to change a light bulb
🌐 As software continues to revolutionize the world, the advent of Generative AI is transforming the very fabric of software itself. My latest book, Generative AI in Action delves into this transformative journey. I am thrilled to announce the early release of my latest book, Generative AI in Action now available through Manning Early Access Program (MEAP) . This publication is a deep dive into…
What does AI think of humans and our humor? I asked #AI - #CodeInterpreter plugin for #ChatGPT to express its experience as an AI working with humans as a meme. It wrote this code to create the meme below. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import matplotlib.pyplot as plt import numpy as np # Create a new figure with a white background fig, ax = plt . subplots(figsize = ( 6 , 6 ),…
What is Whisper from OpenAI? Whisper is a speech recognition model (ASR – automatic speech recognition) from OpenAI. The model itself is multi-task model and as a result in addition to speech recognition, can also do language identification and speech translation across a number of languages. The model is open sourced and it comes in 5 sizes. Of these, 4 have a english-only variant which…
Bing is getting a new look and feel, powered by Microsoft AI and OpenAI (ChatGPT) and was announced yesterday. There is a lot of buzz around this, and I thought I would share my thoughts on this as I got access today. What is the new Bing? Well, it is the thing that is making the 800-pound gorilla in the room, Google, come out and dance on its toes. 🦍 The new Bing is an overhauled version of the…
API Introduction Polite Fork Off As A Service (or PFOaaS) - https://pfoaas.desigeek.com/ is a modern REST API that solves the problem of one telling people to politely fork off. 😇 There are days when we all need such a service for various reasons, and I think it is a great way to release some pent-up frustration. 🖤 It is also a great way to get some laughs too. This is of course meant for…
In the last week or so, all the range online has been #OpenAI’s new chatbot called #ChatGPT (you can read more details on ChatGPT here ). This also got me thinking, about how can we use #CoPilot more than just code. GitHub CoPilot as you might recall is your #AI powered pair-programmer. And as we can see below, it indeed is possible to use Codex as sort of a more general purpose usage. I…
OpenAI recently released #ChatGPT , a GPT-3 based chatbot that can be used to chat with. ChatGPT is a fine-tuned model of GPT3.5 , using #RL (specifically a PPO algorithm) similar to the Instruct series. This post is my experience in using it. Blog post with ChatGPT What better place to start with, than asking it about itself? 😃 Prompt: write me a blog post, about writing a blog post using a ai…
I had been thinking for a while to move away from WordPress for the blog to something simpler and cleaner. WordPress has been great for me when I first moved to it from another engine. However, over time, I found that things have gotten slower, as I added themes and add-ins. Some of these have been great, and others are not really needed. I also wanted to dogfood some of the things we built at…
Thank you; Amit will get the details, and depending on how he feels, he may or maynot get back to you! 🙄 -->
You can contact me here! Name: E-mail address Message Submit
Here is an example of how one can use a text prompt to generate a series of frames, that then are stitched together into a video. The prompt I used was: “a man walking in the parking lot with a miniature poodle”. the final video generated is shown below. There should have been a video here but your browser does not seem to support it. --> There should have been a video here but your…
I have said this before - with the advent of large AI models, Prompt Engineering is critical and is the next challenge for us to master. What is Prompt engineering? Prompt engineering is the process of fine-tuning large models and often is written in natural language, outlining the intention of the user. Prompt engineering is a key element that allows the output to be accurate and reflect the…
I recently needed to install Visual Studio 2022 on one my existing machines to debug a new zeroshot model that has a dependency on our Speech SDK. The Speech SDK is one of our key #AI services in Cognitive Services (as part of #AzureAI). I already had VSCode running, but in this case I need the bigger brother. After installing Visual Studio, I could not get any nuget packages to install; I could…
I recently installed Ubuntu on one of the Pi’s are home and installed Podman - which I hadn’t heard of until recently and is a container engine, similar to docker but doesn’t have a daemon. When trying to get a basic alpine test image running I got this error: Error: error creating build container: short-name "python:3.7-alpine" did not resolve to an alias and no…
The mysterious life of developers has evaded many of us, until now … There should have been a video here but your browser does not seem to support it. The mysterious life of a developer (courtesy Spoon Norge)
If you have a Tesla, then you should absolutely check out TeslaMate which is data logger for your car(s) that one self-hosts. This uses the car’s API and gets all different kinds of telemetry of your drives, charging, batter conditions, acceleration, braking, parking, etc. I personally prefer this, over other online services (of which there are a few) - as it is giving away the keys to the…
It is 2021. And we have #AI writing #AI code. 🤪 It is quite interesting, but also can be quite boring once you get beyond the initial technology, and just think of it as one of the tools in your arsenal. And getting to that point is a good think. As part of a think at work I recently started playing with GitHub Copilot , which is using GPT3 to be your pair programmer – helping write code.…
Reinforcement Learning is teaching by example – it is how most of us learn. Reinforcement Learning (#RL) is a different approach to ML – it is a set of techniques that allows AI algorithms to experiment and learn from experience. RL falls in between supervised and unsupervised learning – there isn’t any labeled data, but at the same time it isn’t unsupervised either. At its most simple form, RL is…
I have been kicking the tires with Open AI’s #GPT-3 . Based on the screenshot below, it might be easy to think “oh boy does the model think highly of itself”, but as with most things in life - the devil is in the details.😃 The screenshot below was a forked version of davinci engine and follows the Q&A structure. GPT-3 vs other AI assistants Using OpenAI’s API is quite…
A #ML algorithm cheat sheet - helping narrow down to a certain set of #algorithm grouping depending on the problem at hand and what we are trying to solve from a business perspective. ML algorithm cheat sheet Figure 2 shows what additional characteristics we need to consider when choosing the right ML algorithm for your situation at hand. This is something that cannot be generic and is very…
#GeekyJokes
If you are like me and get annoyed with the big PowerShell upgrade ’nag’ ‘reminder’ (see screenshot below); instead of trying to figure out what to download and install the update, there is a simpler way to get the latest update and address the nag. :) You can just run the code below in an elevated prompt to get the latest release of PowerShell - it is easy-peasy. :) 1 iex…