I was on a podcast! Check out the latest episode of the Data Is Plural postcast to hear me and fellow crossword archivist Saul Pwanson talk about our respective crossword datasets. I also opine a bit on milk carton data collection, and I’m only now realizing how prescient those comments were! 1 For more, check out my previous blog post introducing the dataset, the dataset itself or the…
My team at Flatiron Health (but especially the inimitable Blythe Adamson ) recently wrote an excellent overview of how we use machine learning to build research-ready oncology datasets from patient medical records — in a paper called Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records. This is a refreshingly simple and concise articulation…
This blog post contains spoilers for Hanukkah of Data 2022. This holiday season I’ve been doing the Hanukkah of Data , which is a puzzle suite by a group of hackers called the Devottys . It’s a sequence of programming puzzles, with one puzzle dropping for every day of Hanukkah. If you’re familiar with Advent of Code , it’s very similar to that, except (a) it only lasts 8…
Suppose you want your computer to take some action (whether it’s showing you information about something, navigating to a particular file, etc.). You’re not reading through a menu and thinking through what to do: you already know what you want to do, and you just have to execute. In these instances, the slowest thing you could possibly do is read. 1 The more you can make your computer…
I recently finished reading Word by Word: The Secret Life of Dictionaries by Kory Stamper , which was an unexpected page-turner. What intrigued me most was (perhaps unsurprisingly) Stamper’s description of how Merriam-Webster gets written, and what a striking resemblance that process has to many successful unstructured data projects in the wild. I want to use this blog post to ruminate on…
This is an at-most-monthly link bulletin, where I compile and post a handful of links that I’ve read and thought about with minimal explanation and commentary, on the theory that the links I find interesting might also be interesting to others. Reduce Friction “Some MLOps principles I think every ML platform should have” “It’s not uncommon for tech companies to have…
Lately, I’ve become worryingly knowledgeable in datasets for crosswords… so I’ve written up basically everything I know that might be helpful to crossword constructors (and makers of other word puzzles, too). However, in writing this, I realized that this may be helpful to just about anybody who works with words — lyricists, poets, marketers, scholars, etc. Hopefully…
This is the first of a new kind of blog post that I’m trying, on the theory that the links I find interesting might also be interesting to others: at most once a month, I’ll compile and post a handful of links that I’ve read and thought about, with minimal explanation or commentary. Here we go! Ninth Circuit Court Opinion, hiQ Labs, Inc. v. Linkedin Corporation California…
You’ve read that web design is 95% typography . You have a static website. You’ve wanted to improve its typography but have never had the time or patience. You’ve might’ve even heard of Butterick’s Practical Typography . If this sounds like you, you’re in luck! A foreword: you can achieve almost everything I describe here by adding CSS in a <style> tag at the…
This weekend I migrated my blog to Hugo. My website is now based on the Hugo Bear Blog theme , generated with Hugo , hosted by GitHub Pages and served with Cloudflare . I’ve also migrated from the eigenfoo.xyz domain to the more creditable-sounding georgeho.org (sadly, georgeho.com and georgeho.net were already taken). In terms of typography, the header typeface is Nicholson Gothic , the…
For people who make careers out of data, data scientists don’t have nearly enough experience in data collection — and many data scientists don’t seem to feel much cognitive dissonance from this fact, despite (very persuasive!) overtures by a few valiant data professionals 1 . With this blog post I want to give a defense of data collection — not as an activity that’s…
A lot of data science and machine learning practice assumes a static dataset, maybe with some MLOps tooling for rerunning a model pipeline with the freshest version of the dataset. Working with streaming data is an entirely different ball game, and it wasn’t clear to me what tools a data scientist might reach for when dealing with streaming data 1 . I recently came across a pretty…
An exciting professional update: today is my last day at Point72 and next month I’ll be joining Flatiron Health as a data scientist on their machine learning team! I’ll be working out of their SoHo offices, and will continue to be based in New York. Flatiron Health is a technology company in the healthcare space, trying to accelerate oncology research and improve quality of cancer care…
cryptics.georgeho.org is a dataset of cryptic crossword clues 1 , collected from various blogs and publicly available digital archives. I originally started this project to practice my web scraping and data engineering skills, but as it’s evolved I hope it can be a resource to solvers and constructors of cryptic crosswords. The project scrapes several blogs and digital archives for cryptic…
Counting the number of valid American-style crossword grids is more or less a solved problem. For example, see this FiveThirtyEight Riddler and Michael Kleber’s answer in a Twitter thread . However, the same doesn’t seem to be true for British-style cryptic crosswords. Hence this blog post! Now, counting the number of valid grids is a different task from enumerating them, and…
“Bayesian modeling is harder than deep learning” is a sentiment I’ve been hearing a lot lately. While I’m skeptical of sweeping statements like that, I agree when it comes to the central inference algorithm — how MCMC samplers work (especially the de facto standard samplers, NUTS and HMC) is one of the most difficult concepts I’ve tried to learn, and is…
I get confused with tensor computation libraries (or computational graph libraries, or symbolic algebra libraries, or whatever they’re marketing themselves as these days). I was first introduced to PyTorch and TensorFlow and, having no other reference, thought they were prototypical examples of tensor computation libraries. Then I learnt about Theano — an older and less popular…
Sometimes I get sniped by Hacker News posts , and this one plunged me down a rabbit hole for coding fonts. Many coding fonts are lightly stressed, monospaced sans serifs: in other words, each glyph takes the same width, and each glyph looks like a stick figure, with constant stroke width (a.k.a. stress ) throughout the glyph. But as the Internet stranger uncanneyvalley pointed out, there’s…
Recently there has been a modularization (or, if you’re hip with tech-lingo, an unbundling ) of Bayesian modelling libraries. Whereas before, probability distributions, model specification, inference and diagnostics were more or less rolled into one library, it’s becoming more and more realistic to specify a model in one library, accelerate it using another, perform inference with a…
I’ve picked up a new hobby (or perhaps just another fleeting fascination) — digital publishing. The catalyst was the book Practical Typography by Matthew Butterick . There were so many interesting things about it: it is gorgeous, it expounds well-argued (if slightly controversial) views on how the average writer should think about typography, it has a little widget that would change…
Floating-point formats are not the most glamorous or (frankly) the important consideration when working with deep learning models: if your model isn’t working well, then your floating-point format certainly isn’t going to save you! However, past a certain point of model complexity/model size/training time, your choice of floating-point format can have a significant impact on your model…
I’ve recently had to learn a lot about natural language processing (NLP), specifically Transformer-based NLP models. Similar to my previous blog post on deep autoregressive models , this blog post is a write-up of my reading and research: I assume basic familiarity with deep learning, and aim to highlight general trends in deep NLP, instead of commenting on individual architectures or…
A while back, I explored the possibility of simplifying 1 PyMC4’s model specification API by manipulating the Python abstract syntax tree (AST) of the model code. The PyMC developers didn’t end up pursuing those API changes any further, but not until I had the chance to learn a lot about Python ASTs. Enough curious people have asked me about my experience tinkering with ASTs that I…
I was lucky enough to be invited to attend the Gradient Retreat earlier this month. It was an entire week on a beautiful island with some amazingly intelligent Bayesians, and no demands on my time other than the self-set (and admittedly vague) goal of contributing to probabilistic programming in some way. I initially tried to implement mass matrix adaptation in Tensorflow Probability, but I…
I made a Vim plugin for Stan! I’ve been reading and writing a lot of Stan lately, but mainly in barebones text editors (or even just by cat ing out the file), so I had to make do with none of the creature comforts of my favorite text editor, Vim. But I also wasn’t happy with the syntax highlighting provided by existing Vim plugins (and they also looked out of date and thinly…
Recently, the PyMC4 developers submitted an abstract to the Program Transformations for Machine Learning NeurIPS workshop . I realized that despite knowing a thing or two about Bayesian modelling, I don’t understand how probabilistic programming frameworks are structured, and therefore couldn’t appreciate the sophisticated design work going into PyMC4. So I trawled through papers,…
Some exciting personal news: I’ve (finally) graduated from The Cooper Union , and I’m joining Point72 Asset Management as a data scientist/research analyst! Point72 is an American hedge fund, headquartered in Connecticut. I’ll be based in New York, working out of their Hudson Yards offices. In this next chapter of my life, my professional focuses are: Keep learning. Bayesian…
I ported Jonas Lindeløv ’s essay, Common Statistical Tests are Linear Models from R to Python. Check it out on my blog , GitHub , or Binder !
This is the second of a two-part series about Bayesian bandit algorithms. Check out the first post here . Previously , I introduced the multi-armed bandit problem, and a Bayesian approach to solving/modelling it (Thompson sampling). We saw that conjugate models made it possible to run the bandit algorithm online: the same is even true for non-conjugate models, so long as the rewards are bounded.…
My current project involves working with deep autoregressive models: a class of remarkable neural networks that aren’t usually seen on a first pass through deep learning. These notes are a quick write-up of my reading and research: I assume basic familiarity with deep learning, and aim to highlight general trends and similarities across autoregressive models, instead of commenting on…
Lately I’ve been troubled by how little I actually knew about how Bayesian inference really worked . I could explain to you many other machine learning techniques , but with Bayesian modelling… well, there’s a model (which is basically the likelihood, I think?), and then there’s a prior, and then, um… What actually happens when you run a sampler? What makes…
Natural language processing is in a curious place right now. It was always a late bloomer (as far as machine learning subfields go), and it’s not immediately obvious how close the field is to viable, large-scale, production-ready techniques (in the same way that, say, computer vision is ). For example, Sebastian Ruder predicted that the field is close to a watershed moment , and that soon…
This is the first of a two-part series about Bayesian bandit algorithms. Check out the second post here . Let’s talk about Bayesianism. It’s developed a reputation (not entirely justified, but not entirely unjustified either) for being too mathematically sophisticated or too computationally intensive to work at scale. For instance, inferring from a Gaussian mixture model is fraught…
Recently I’ve started using PyMC3 for Bayesian modelling, and it’s an amazing piece of software! The API only exposes as much of heavy machinery of MCMC as you need — by which I mean, just the pm.sample() method (a.k.a., as Thomas Wiecki puts it, the Magic Inference Button™ ). This really frees up your mind to think about your data and model, which is really the heart and soul of data…
Note: the following article contains several examples of hate speech (including but not limited to racist, misogynistic and homophobic views). Have you heard of /r/TheRedPill ? It’s an online forum (a subreddit, but I’ll explain that later) where people (usually men) espouse an ideology predicated entirely on gender. “Swallowers of the red pill”, as they call themselves, maintain that it is men ,…
As I learn more and more about data science and machine learning, I’ve noticed that a lot of resources out there go something like this: Check out this thing! It’s great at this task! The important task! The one that was impossible/hard to do before! Look how well it does! So good! So fast! Take this! It’s our algorithm/code/paper! We used it to do the thing! And now you can do…
Here’s a short story you might know: you have a black box, whose name is Machine Learning Algorithm . It’s got two modes: training mode and testing mode. You set it to training mode, and throw in a lot (sometimes a lot a lot) of ordered pairs $(x_i, y_i), 1 \leq i \leq l$. Here, the $x_i$ are called the examples and the $y_i$ are called the targets . Then, you set it to testing mode…
Linear discriminant analysis (commonly abbreviated to LDA, and not to be confused with the other LDA ) is a very common dimensionality reduction technique for classification problems. However, that’s something of an understatement: it does so much more than “just” dimensionality reduction. In plain English, if you have high-dimensional data (i.e. a large number of features) from which you…
I was lucky enough to have the chance to intern at Quantopian this summer. During that time I contributed some exciting stuff to their open-source portfolio analytics engine, pyfolio , and learnt a truckload of stuff while doing it! In this blog post, I’ll describe and walk through two of the new features that I authored: the risk and performance attribution tear sheets. Risk Analytics A…
This is a follow-up post to my first post on a recent project to model hate speech on Reddit . If you haven’t taken a look at my first post, please do! I recently gave a talk on the technical, data science side of the project, describing not just the final result, but also the trajectory of the whole project: stumbling blocks, dead ends and all. Below is the slide deck: enjoy! Abstract…
The Utah teapot! (Basically the “hello world” of computer graphics). This is the first post of what will (hopefully) be a cool and interesting blog. Hope you like it! For those who are interested, this website is based off the Minimal Mistakes theme by Michael Rose, generated with Jekyll , hosted by GitHub Pages and served using Cloudflare . I’ve had no complaints with this blogging stack:…