I am a research engineer based in San Francisco and New York City. My work has ranged from building infrastructure tools for machine learning teams, to no...
I’ve been using Sora for a week now ( some of my creations ), and I think feeds are the correct form factor for learning how to use generative media models at scale. Early generative media products followed a pattern where I’d open up a standalone interface, and I'd have to figure out what generations were possible on my own. Eventually, I’d run out of ideas because I didn’t understand how to use…
TLDR: I trained a Flappy Bird world model to run in your web browser, try out the demo here! Flappy Bird world model running in Safari on iPhone 14 Pro Recently, I’ve been pretty interested in world models, which build on video generation research to create real-time simulations based on a user's input. I’ve seen them being applied everywhere from creating environments for autonomous vehicles and…
Last year, the Character.AI team released a blog post that detailed their approach in building a highly efficient inference system that serves over 20,000 inference queries per second, roughly 20% of Google Search's traffic. They focused on reducing the KV Cache size, which plays a significant role in increasing the decoding speed in transformers, but is the key bottleneck in inference.…
This past week, I came across the DiffusionDB dataset curated by the Polo Club of Data Science at Georgia Tech. They scraped over 14 million image-prompt pairs collected from users generating images in the Stable Diffusion Discord. Each entry includes the image and the text prompt used to create the image, along with detailed metadata such as the sampler settings, image properties, and usernames.…
I was recently re-reading Finbarr Timber’s post on transformer inference optimizations, and I wanted to try to implement each of these techniques 1 in nanoGPT to see how much we could practically speed up inference with GPT-2’s architecture and reduce the computational bottlenecks in the model. For my benchmark model, I'm going to use the GPT2-XL model weights and load them into nanoGPT. This will…
This post is going to consist of my ongoing review of various sampling methods I’ve found through papers and online implementations. While samplers are relatively straightforward to create and combine, there is a challenge in trying to sample both high quality and diverse outputs. I'm particularly interested in finding samplers that are context-aware, resist hallucination, and can explore multiple…
Disclaimer: I wrote this article before OpenAI released o1, which now hides its "reasoning" tokens making them inaccessible through the tiktoken library. I think o1 uses the same base tokenizer as GPT-o, but unfortunately we can no longer observe this directly. GPT-o introduces a new tokenizer o200k_base , which both doubles the model's vocabulary size to 200k (previously 100k with GPT-4) and…