The personal blog of Alex Merose.
I write this from the veterinary hospital. My one-year-old cat Twiggy ate a dozen hair ties last night while we were at a friend’s house celebrating NYE. Alas, what an omen for 2026. I am almost certain that she’ll be ok. She’s always been our little diva, and now I have the vet bill to prove it. Act 1: A stomach of rubber The waiting room of a vet hospital just so happens to be an excellent place…
DataLoader vs Dataset: * Dataset: stores samples + labels * DataLoader: like an iterable Dataset, provides easy access to samples. I think it’s worthwhile to look at built-in datasets to see how they are structured. A custom Dataset must have a constructor, __len__ and __getitem__ . Constructors don’t need to call super() . __getitem__ returns a sample/label pair, i.e X,y . A custom DataLoader is…
I took the notes in a meeting today 1 . Sometimes I get complements on how I take notes. Today, though, I got an unusual one: In the middle of a discussion that I was capturing, someone paused and asked, “Is this an AI note-taking services? I don’t think so because it’s making spelling mistakes.” I looked up from my keyboard a bit surprised, bearing a smile. It made me think a bit about the…
Scientists think they’ve found evidence to add a new law of thermodynamics. Darwin’s Theory of Evolution seems to apply to any system where there is reproduction, variation, and selection. The theory might be a new universal law of information. References https://www.space.com/scientists-propose-missing-law-evolution-of-everything-in-the-universe
Paper compares 4 intracranial neuroelectrophysiology “data repositories” (read: data formats): - DABI: Data Archive for the BRAIN Initiative - DANDI: Distributed Archives for Neurophysiological Data Integration - OpenNeuro - Brain CODE These two efforts also utilize NWB (Neurodata Without Borders) and BIDS (Brain Imaging Data Structure). The paper calls this field “neuroinformatics.” I really like…
I’ve just following some instructions 1 to get Obsidian set up on my mobile and — get this — synced through git (!!!). If this works, it means I can write zettels on the go. If I can write these paper slips on the go, then… [Takes a long, pensive look out the window] then… I think I’ll just exist in cyberspace now. References…
Notes from the OpenXLA talk: JAX: Low-level control with shard_map and Pallas. See references for the full video. Shard-Map “A souped-up parallel map” that parallelizes functions on multiple devices, with each device receiving a different shard from the input data. “shmapped” ## Custom Kernels triton_call Call triton code from Jax pallas_call Recommended. Expresses Kernes in Jax. Custom kernels…
AKA: “General and Scalable Parallelization for ML Computation Graphs” References https://arxiv.org/abs/2105.04663 https://youtu.be/5ilr4gcenaA?si=jp19aHvAnyQ_0-_x&t=89
Embedded in the ML modeling process is a huge search problem : References S. Zayd Enam’s Why is machine learning ‘hard’?
Common things I take away from Saul’s talk: * OSS Python could substantially benefit from frontier programming language (PL) research * Talks about the birth of NumPy in 1995 * NumPy is “pointers, strides, dtypes” between Python and C arrays. * History of OSS Python is an ecosystem of collaborative, multi-stakeholder, interdependent yet individual projects. * Terrific overview of the PyData…
The future of PyTorch is to become a more complicated, more technical-debt ridden Jax. References PyTorch is Dead. Long Live Jax.
A product of the hazy-research-and-flash-attention lab is ThunderKittens . In their README, they write (emphasis mine): ThunderKittens is built from the hardware up – we do what the silicon tells us. And modern GPUs tell us that they want to work with fairly small tiles of data. A GPU is not really a 1000x1000 matrix multiply machine (even if it is often used as such); it’s a manycore processor…
References https://hazyresearch.stanford.edu/blog/2024-05-12-tk https://github.com/HazyResearch/ThunderKittens Community worth joining: https://discord.gg/gpumode
Another classic joke about Google’s engineering culture: Everything is either deprecated or not ready yet. – Googlers to Nooglers during orientation. Goomic: Everything is either deprecated or not ready yet References https://goomics.net/50/
There’s an old joke about Google’s engineering culture that goes like this: Come work at Google! Where the impossible is merely difficult , and the easy is merely difficult . – The L7 TL in Boulder, CO who first told it to me. References
The Hazy Research Lab, creators of Flash Attention, combine both systems software engineering expertise with ML algorithms development. In the algebra of a ML algorithm, there are a lot of ways to follow equivalent equations. The best way to forge a path, then, is to choose what best takes advantage of the underlying hardware. This, in essence, is how FlashAttention was invented. Since so much of…
Originally sketched on September 29, 2023. I’m incredibly excited about Cubed . When I first learned about the project via this Xarray blog post , I knew it was worth betting on. To be able to perform array computation serverlessly – without having to worry about managing memory (!!) – seems like the future of data science in the cloud. This slide deck from Tom White, the primary author, does an…
In your career as a technologist, whenever possible, invent on principle. Find principles to guide everything you build. Live by them. “No Modes”: The guy who invented copy and paste invented on the principle that software should not contain modes, or different environments to do different things. That directly lead to the invention of copy and paste. Bret’s principle: when creating an interface,…
References Marcel Duchamp and the Refusal of Work
Just because a system or process becomes more efficient, doesn’t mean it consumes less energy. In fact, it’s possible that when it becomes more efficient, it leads to an increase in demand. Notable Examples: - Google SERP webpage optimization (pages were more lightweight) lead to an increase of server load. Suddenly, people from NBU countries were able to access the pages that they weren’t able to…
All productivity, as measured by GDP, leads to hydrocarbon consumption. Economics as a field of study has a big error by not properly accounting for energy. We pay the interest on energy (oil), but not the principle (10 million years of heat and pressure). We will soon run out of fossil fuels altogether. We hit peak oil in the 70s. Since then, we’ve been in the era of “tight oil”. The costs for…
anndata i.e. Annotated data is a NumFOCUS-supported Python package that handles annotated data matricies on memory and disk, and is between Pandas and Xarray. Without going into every fancy Python-based data structure, we think that still today no other alternative really exists that: Handles sparsity Handles unstructured data Handles observation- and feature-level metadata Is user-friendly –…
metamorphic-testing can be used for Machine Learning, since ML systems are really hard to test. This method reminds me a lot of common [[data augmentation]] strategies for model pretraining, such as MAE, or more traditional algorithms like [[jitter]]. References https://www.hillelwayne.com/post/metamorphic-testing/ https://www.cs.hku.hk/data/techreps/document/TR-2017-04.pdf…
Property based testing is difficult because it’s hard to find good properties. Metamorphic testing is a type of PBT under a “metamorphic relation”. Here, we see if the outputs all match when many of them have undergone automated mutation (that shouldn’t but could affect the output). For example, if a given soundclip transcibes to output out , then we should still get output out if we: Double the…
References https://blog.lamin.ai/arrayloader-benchmarks
A lattice is complete if all subsets have both a join and a meet. A complete lattice is more restrictive than a normal lattice. Every non-empty finite lattice is complete. Examples : The power set of a given set, ordered by inclusion. The supremum is given by the union and the infimum by the intersection of subsets. … The convex subsets of a real or complex vector space, ordered by inclusion. The…
Apparently, this term doesn’t mean much. Why? Well, if a semilattice is complete (all subsets hold the meet / join property), then it will be complete in the other direction, making it a complete-lattice [A] complete meet-semilattice has all non-empty meets (which is equivalent to being bounded complete) and all directed joins. If such a structure has also a greatest element (the meet of the empty…
A semilattice is like a directed lattice in either the meet (LCM) or join (GCD) direction (“for any nonempty finite subset”). Meet -semilattices are “lower semilattices” whereas join -semilattices are “upper semilattices.” A set S partially ordered by the binary relation ≤ is a meet-semilattice if For all elements x and y of S, the greatest lower bound of the set {x, y} exists. — # References…
It’s the eve of the start of my new job. I start on Monday the 13 – fitting, since I quit on Friday the 13th. It’s been quite a busy few weeks that I’ve been home. These previous two days were especially exhausting. Between too many errands and driving hours to see an old friend, I’m really happy to have a down day at home. Besides a notable walk and phone call, I’ve nearly entirely spent the…
It’s the best advice for creativity I have yet received. I was a bit lost when I first heard it, having just quit my job about to go travel for who-knew-how-long. I thought I might use my time off to figure out new ways to be creative. Before I left, I had a phone call with a close friend of mine who is a musician . Man, I love Zach. Drone love me by Wyrm Rilye When I told him my intentions,…
My wife and I absolutely love our wedding video . I honestly can’t imagine how it could have been any better. In addition to their excellent work, we chose Amari Productions as our videographers because they offered us the ability to get all our wedding footage in raw. This really appealed to us since we think it would make a really great anniversary present down the line to eventually produce a…
Breaking: Elon Musk has just reinstated Donald Trump’s Twitter account. — No Lie with Brian Tyler Cohen ( @NoLieWithBTC ) November 20, 2022 I’m reading Cal Newport’s Digital Minimalism right now. What a perfect time to do this, just as Twitter is on the Musk train to hell. The book is making me notice how little I really get out of my media consumption habits. Screen Time shows that I spend four…
I got married a few weeks ago, and the strangest thing happened the Friday before my big day . I headed to the barber shop to get my beard cleaned up, as one does. I got there right on time. “Alex for a 2:30 appointment?” I asked as I entered the shop. This question seemed to disgruntle the person in the waiting area next to me. “ I’m Alex who has a 2:30 appointment” he said as if I was cutting in…
I see why people say that their wedding is the best day of their life. Mine certainly was.
WIP? I’ve been thinking a bunch about how to tackle this problem . It seems like my trip to the barber shop seems like a weird variant of the Birthday Paradox : What is the probability, in a group of n randomly chosen people, at least three will have the same name and wedding day? I’ve poked around stats exchange and found some general solutions to the birthday paradox, including an R…
created: 2021-02 updated: 2021-08 While building a Beam pipeline for work, I found myself surprised to find there was no documentation on how to structure the pipeline as a CLI 1 . I wanted to make running my pipeline as easy as possible. What’s easier than installing a python package via pip, and invoking a well-documented command? With features like pip search and --help messages, one never…
Prototypes and thinking about electric origami I was reading hacker news aimlessly on the couch, as one does when home alone on a Sunday afternoon, when I came across this post . Printable circuits with a cheap inkjet printer? How exciting! Think of the possibilities! I shared the post with a few of my friends. Jason got back to me, and we started thinking about what we could do with this kind of…
Me just after I’ve proposed I like surprises, but Camille doesn’t. She’d rather be able to look forward to things. For her, half the pleasure of enjoying, say, a vacation comes purely from its anticipation. I had a challenge before me: How could I give Cam the biggest surprise of her life while also letting her look forward to it? I originally had the idea to propose to Camille around the end of…
When I first started looking for a ring, I was looking at how much bang for my buck I could get in diamond sizes… which was totally the wrong way to go about it. Cam wouldn’t want a big rock on an ugly ring – it would get in the way and be a liability! Besides, the ring had to compliment her style and personality. The trick that worked for me to better figure out what would best suit her was to…
The student org I used to be a part of (the Cognitive Science Student Association ) asked me to lead a workshop on machine learning at their annual conference . The following are the slides and tutorial I presented. Most of my talk consisted of live coding a simple ML application in the browser using tensorflowjs and glitch. Together, we made a mock version of “Rainforest Connection” , an app…
A lattice is a partially ordered set, or poset. Here is a lattice ordered by divisibility: Lattice of 120 ordered by divisibility See other examples . Every two elements of a lattice has: - a supremum , or least upper bound, or join . e.g. LCM - an infimum , or greatest lower bound, or meet . e.g. GCD — # References https://en.wikipedia.org/wiki/Lattice_(order)
A new meditation: Let ice melt on your tongue. Feel the cold on your palate. Watch the pain fade into an electric buzz, as you inhale a dragon’s breath. Fight the nervous impulse to chew. Notice when you are chewing, when you became lost in a thought. Let the fragments return you to the moment. Taste the present.
A week ago (late 2020), I started a new habit : I now try to find 15 minutes of boredom a day. These are uninterrupted periods of no stimulation. I have my phone off or in another room. I avoid screens in general. No music plays in the background. I just sit there alone with my thoughts and perceptions – preferably outside. I find my 15 minutes to be exhilarating. They start out… boring. Soon…
Tip: For better work-life balance, consider sending emails to your self. Separate your work and personal apps across devices. Don’t always be available to receive work messages – when you close your laptop, work should be done. The hardest part of this separation, for me, is that shutting a laptop doesn’t stop the spontaneous flow of ideas. Maybe when your brain is trying to relax , you think of a…
In 2016, the year I graduated from college, my partner, Camille, gave me a Passion Planner . This planner is a journal / ideation system that helps you figure out what you want in life, and then helps you realize it. A guide designed by someone who felt adrift after her own graduation, it was exactly what I needed. The system works something like this: First, you brainstorm ways how you would like…
Ideally, I don’t want my website to be just a blog. I don’t think in terms of sequences of posts. I am not trying to create a newsletter – while this seems fun, the upkeep feels like more of a burden than empowered expression. Why not, instead, aim to express systems of ideas free of time? I have a lot of ideas , but most of them are small and only make sense in context with other thoughts. While…
My partner’s food intuition – her ability to weave haphazard ingredients into a coherent, delicious meal – is nothing short of magic. Here are a few meals we’ve made together that I’d like to remember. farmers market mushroom pasta 2020-Oct-17 behind the recipe We had just picked up fresh porcini mushroom pasta from the local farmers market. Such a fine ingredient required a worthy recipe to go…
I make a smoothie for myself every morning. One half cut of frozen batteries AA, AAA, 9-Volt, You can have just one variety, too. Add three-fourths cup of bleach to make it easier to blend, less if you like a thinker gulp. Fresh plastic adds easy petroleum that you can’t taste, and a good green color, to boot. Some say local spider pollen helps with your allergies, but I can’t say it’s worked for…
I had a conversation a few weeks ago with my project manager, Bernhard Seefeld , about strategies to combat writer’s block. At the time, I had been taking longer than I would like to write documentation. I asked him how he deals with writer’s block. He gave me some great advice, which I thought was worthy of sharing. Can’t start a draft? Send an email A crafty use of psychology: sitting down and…
Well, here’s a very meta story about this website and how I’ve grown since I started it. Let’s talk about an issue that has bugged me for quite some time and the annoying maze I had to get out of to fix it. Check out the URL of this domain. Is it secure? How do you know? Let me show you how I know it’s not, as of writing: My site does not use HTTPS, and therefore is not secure “How embarrassing. A…