I write my thoughts to get stuff out there. [1] My calling is to create new moments of understanding for people, which I manifest through writing, cooking, a...
Llama from scratch I want to provide some tips from my experience implementing a paper. I'm going to cover my tips so far from implementing a dramatically scaled-down version of Llama for training TinyShakespeare . This post is heavily inspired by Karpathy's Makemore series , which I highly recommend. I'm only going to loosely follow the layout of their paper; while the formatting and order of…
A brief exploration of what makes LLMs hard to deploy in production for builders and customers, a framework for understanding the problem, and some strategies for making it better. So, you want to add a chatbot I’ve been working with LLMs for the past few months across every part of the technology stack: from low-level (pytorch and ggml) to abstraction and orchestration layers (LangChain); from…
As someone chronically privileged, being an overpaid, underachieving skilled "laborer" with no debt, dependents or real responsibilities, I basically led a life where I could afford everything I ever wanted, and still had plenty of days free, each day filled to the brim with hours to be bored. Free time and objects are so relatively cheap for privileged people like me today, that many have stopped…
Trying to do a startup is hard, relative to regular work. Relative to regular work, I've noticed three poverties: poverty of validation, poverty of presence, and poverty of money. Validation The first is the poverty of validation. In regular employment, and also in school, there is a constant cycle of validation. After you complete an assignment (assignments which are scoped to be believably…
There are a few old sayings going around on the relationship between who your friends are and who you are. "Show me your friends and I’ll show you your future," or "you are the sum of the five people you spend the most time with." To me, these broadly ring true. Having good friends is pretty fucking great. Good friends want the best for you. Good friends respect you. Good friends enjoy spending…
This evening, I watched Frances Ha (2012). It's a film about a not-so-young aspiring dancer who finds herself a witness to her own life, rather than the choreographer of it. It's fitting that the film takes place in New York, a claustrophobic petri dish of rapidly evolving cells who crowd both my physical and mental space. To be a cell in a cramped dish where everyone else around you appears to be…