Recent content on Emir's blog
Abstract This article is about how to use LLMs as an approximate joint probability distribution over tokens rather than as an expert system. I show how multinomial/ordinal queries with grammar constraints avoid errors related to greedy recursive generation, allow for uncertainty quantification via logits, and enable robust inference via invariant query reformulations which expose logical…
Introduction Before deep learning was a thing, if you had \(N\) vectors, and you wanted to match the closest one to some noisy version you would use kNN (\(k\) nearest neighbour). This implies comparing the input to all candidates, and usually weighting the final answer by the distance \(d_i\) to the top \(k\) matches. If \(k=N\) and the weighting has the form \(exp(-d / \rho)\), this becomes the…
Implementation code is available here. Its just North of 200 lines of Python code, so after getting the gist from this post, its straightforward to read the code directly for details. Introduction Query understanding is a set of methods used to better interpret the intention of a search query prior to its execution: things like “query segmentation”, “entity recognition”, “rewriting”, “spelling…
Snakes & ladders is a classic board game, originally imported into the United Kingdom from India circa 1890 according to Wikipedia. Its a square grid numbered row wise from 1-100, with some of the cells connected together either by a snake or a ladder. Players roll a dice and then advance that many steps. If a player lands on the head of a snake or the foot of a ladder they are transported to the…
Introduction The pyevidence repository, installation and usage examples are available here. I had properly discovered evidence theory – also known as Dempster-Shafer theory – recently, and wrote about it a bit here. Its a simple theory to get going with, and the Wikipedia article does a good job of introducing it. Briefly, the subject of evidence theory is the powerset of some set \(X\), to which…
Here is an interesting scheme I encountered in the wild, generalised and made abstract for you, my intrepid reader. Let \(X\) be a set of binary variables. We are given information about subsets of \(X\), where each update is a probability ranging over a concrete set \(A \subset X\), the state of which is described by an arbitrary quantified logic formula. For example, \[P\bigg\{\exists_{x_i, x_j…
Here are a few practical questions of my own invention which are easy to comprehend but very difficult to solve without statistical reasoning competence. They are provided in order of difficulty. The answers are at the end. If you find errors or have elegant alternative solutions, please email me (address in bio)! QUESTIONS 1. Sorting fractions under uncertainty You are given the number of trials…
Summary I present a weak supervision paradigm called “data programming” which uses maximum likelihood estimation to produce soft labels from heuristics. These soft labels can then be used to train other models, without true labels being required at any stage. I’ve included a simple example from first principles to show that the methods work. The original authors have a fully featured package…
SUMMARY I use 500 Hacker News (HN) titles and an LLM to derive an article ranking model from a user supplied preference description. The LLM supplies the labelled data, whilst Ridge regression and cheap sentence transformer embedding provides the features. The surrogate has 0.74 Spearman correlation with the LLM labels, which is remarkable given that the experiment is entirely unoptimised.…
INTRODUCTION This post is about sizing independent simultaneous bets through methods related to the Kelly criterion. I’ll start by explaining what the Kelly criterion is and how to derive it. I’ll then discuss a simple way to extend it to simultaneous independent binary bets. KELLY CRITERION The Kelly criterion imagines a single bet made sequentially an infinite number of times. It aims to…
A basic application of linear methods are linear regression models. However, in some settings, they can be limiting at least because: (1) they may have few degrees of freedom and therefore saturate quickly, and (2) they may imply a rigid geometry (a hyper-plane) which is often unrealistic. It turns out that if we can express similarity between data points as a Gram matrix, we can do linear…
I read this paper a while ago, which sets out the problem of linear metric learning nicely. I wanted to see whether metric learning was possible to carry out in closed form. It turned out to be relatively straightforward. Say we have some feature vectors \(x_i \in \mathbb{R}^p\) and some responses \(y_i \in \mathbb{R}^k\), We want: \[(Ax_i - Ax_j)^\top (Ax_i - Ax_j) \approx (y_i -y_j)^\top (y_i…
SUMMARY I tackle 1BRC in Fortran which requires processing 1B rows of weather station data (~15GB) to obtain min/max/mean for each station as quickly, as you can muster. I started out with a time of 2m8s and reduced it to a best run time of <6s on a 4 i7 laptop with 16GB RAM. I herein document how. INTRODUCTION The 1BRC data looks like this: Hamburg;12.0 Bulawayo;8.9 Palembang;38.8 St.
Here are some Advent of Code solutions: 2023 (Prolog) 2022 (Haskell) 2021 (Python & Scala) (in progress at the time of writing). Here are some comparative notes: My Haskell solutions were mostly < 27 LoC. The Prolog solutions where considerably longer. The Prolog solutions were, on average, much harder to code for me. My Prolog solutions ended up looking rather functional for the most part.
INTRODUCTION [If you want to have a go straight away, jump to the examples at the bottom of this post.] Making a novel logic puzzle has been a bucket list item for me since yesteryear and I was finally handy enough with Prolog to endeavour for something elegant without having to write reams of code. I arbitrarily decided that I wanted the puzzle to be expressed in terms of Dominoe tiles.
What follows are some notes about a minimal proof-of-concept for a stochastic simulator in Prolog via a meta-interpreter. META-INTERPRETER Here is a Prolog meta-interpreter which supports probabilistic head clauses through the use of the p/2 predicate: prove(true) :- !. prove((A,B)) :- !,prove(A),prove(B). prove(Head) :- clause(Head,Body), (p(Head,P)->(random(X),1-P<X);true), prove(Body).…
INTRODUCTION Gentle reader, permit me to try and convince you that data analysis is better with logic programming. In this post I’ll analyse a staple dataset – the ggplot2 diamond prices – using a symbolic approach which, I will demonstrate, is able to establish a robust model, otherwise difficult to recover. DATA I’ll use the diamond prices data which comes with the R ggplot2 package. It consists…
Summary Some notes about the construction of a Block Dominoe playing algorithm for a hidden information variant of the game. I build a game simulator, learn from a heuristic algorithm and then develop some play-out based algorithms which seem fairly good. I conjecture the final algorithm approximates optimal play. The final SWI Prolog implementation is available here. I am selling an optimised…
SUMMARY I parse HackerNews (HN) “Ask HN: Who is hiring?” posts from 2013 to time of writing and analyse them to better understand the trends in the data job market with a focus on the fate of data science. Here are my main conclusions: It is likely that the Data Scientist role is in a long term decline and that skills such as data mining and visualisation are also out of favour.
Introduction I happened upon a post by Gwern discussing, in some detail, various solutions to riddle #14 from Nigel Coldwell’s list of quant riddles. I initially got as far as the problem description in Gwern’s article and avoided reading further so I could first solve it for myself. The problem is stated as follows: You have 52 playing cards (26 red, 26 black). You draw cards one by one. A red…
Introduction I had recently started going to a new specialist gym that runs 3 classes per day during the working week and is closed the rest of the time. I’ve been at a few different times on a few different days, and already I was seeing many of the same people from the first class. It occurred to me that the chance of seeing the same faces should somehow scale with the number of people going to…
Summary I explain blocking, optimal design and covariate adjustment as methods to improve power in design of experiments. I try to motivate this as something data scientists working with online experiments ought to be doing since it can drastically improve the power of an experiment and make design of experiments tractable where otherwise it would not be. I also implement a D-optimal design…
Summary I motivate clustering as a problem well suited to logic programming in the general case, and I volunteer a couple artisanal clustering algorithms in Prolog demonstrated on some mock data. Note: the code herein is my own. If you see bugs, or are a Prolog mage and can write it even more concisely, I’d be grateful if you could let me know. Introduction There are many clustering algorithms…
Summary I demonstrate a widely applicable pattern which integrates Prolog as a critical component in a data science analysis. Analytic methods are used to generate properties about the data under study and Prolog is used to reason about the data via the generated properties. The post includes some examples of piece-wise regression on timeseries data by symbolic reasoning. I also discuss the…
Introduction I often work with clients who have large “data lakes” or big star schema style enterprise databases with fact and dimension tables as far as the eye can see. Invariably said clients end up with a substantial SQL codebase composed of hundreds of independent queries with lots of overlap between them. I want to be able to treat SQL repositories like I’d treat other codebases. That is,…
Introduction “Data science” has a handful of fundamental metaphors for problem solving, few moreso versatile than the “point cloud”. That is, translate your data into points in a n-dimensional metric space and then do linear algebra to it. The point cloud metaphor applies most simply to numeric tabular data, but with a little creativity it readily extends to text, images, time-series and so on. In…
SUMMARY I show how assumptions about price structure can be used to build a compelling fixed effect (deterministic) price imputation model for the UK residential housing market. The model uses just public price paid data. I describe how the data is collected and processed, how the model is designed, and how it is fitted using the Jax Python package. I showcase some results, I discuss shortcomings…
SUMMARY I use lichess.org games data to investigate the extent to which fast thinking is the dominant factor affecting game outcomes at any time control. I show how to (1) frame a pseudo-experiment, (2) database lichess.org data, and (3) carry out the analysis. I argue that fast thinking is most prominent in quick games. I analyse a sample containing games from pairs of users who have played each…
I spent about 50 hours making things with a language called J. Its an APL progeny and it promises to make possible the expression of general programming tasks as if in mathematical notation. In J, arrays are first class citizens, and most functions natively support array operations. It also has fancy composition rules, so rather than the usual f(a,b), in J you have either f a or a f b.
INTRODUCTION The crux with IQ – so far as I understand it – is that performance across abstract reasoning tasks is correlated no matter what the tasks are. That is, being good at one type of abstract puzzle implies that you’re more likely to be good at any other such puzzle. If you got \(M\) people to do \(N\) puzzles and made an \(M\times N\) matrix of their scores, and then did some SVD on it,…
My name is Emir. I do research commercially, mostly by applying maths, stats and comp. sci. I’ve been at it about 7 years. I’m also a software engineer of 18+ years and an astronomer (PhD candidate). Aside from the PhD in progress, my academic background is in Analytical Philosophy (BA) and Applied Maths (GDip, Msc). My Linkedin is here. You can contact me by email. Construct my email address by…