Recent content on Alán's blog
Today I failed to videocall a prospective landlord. He wanted to FaceTime me to judge whether or not I am an adult who can (and is willing to) pay rent. I explained that FaceTiming my number would not work because I use an Android phone, not an iPhone. Hours later, he replied asking if we were FaceTiming or not. I explained the issue again and suggested a standard phone call instead. No response.…
To showcase the real-life usefulness of Duckdb (and SQL-adjacent Domain Specific Languages in general) I decided to use the public datasets made available by the Massachusetts Bay Transport Authority (MBTA). I have lived in Boston for a couple of years and wanted to test if my intuition of the busy lines and stations lined up with their data. There are multiple available (tabular) datasets:…
At some point I was struggling to get access to my Gmail account. Since I usually block unwanted scripts from running on my computer, Google likes to flag my Gmail login attempts as suspicious activity. This would be fine if it didn’t also make one of my only alternative identification methods an SMS. If I were to lose my phone or number I could be locked out of my account. Most accounts I…
I use dired-do-shell command in Emacs to run CLI commands from within its file manager dired . This workflow makes it easy to perform batch operations on files that would be annoying otherwise. The trouble arose when trying to use the duckdb CLI to print the schema of a parquet file, as the notation for wildcards in emacs ( * and ? ) conflicts with duckdb’s usage of the former. Thus running…
Duckdb, the (tabular) data exploration tool I use supports window operations. I recently discovered that it can also perform cumulative sums in a very efficient manner. Let us generate a toy dataset where we want to calculate the sum of one column relative to the order of another one. -- seeding for reproducibility, creating a table to hide output CREATE OR REPLACE TABLE seed AS SELECT SETSEED( 0…
To solve a multitude of challenges I have faced when processing high throughput microscopy data, have developed Nahual , a tool that allows me to move data across multiple Python environments that deploy deep learning models in the background. I usually keep these models “listening” in the background for the main analysis pipeline ( aliby ) to send them data to process. To be able to…
I wanted a simple way to see the progress of a data processing pipeline, and the internal progress bar tools were messed up by threading. I thus decided to use the number of output files in each folder as an indicator of progress. In my case the output of tree . looks like this: . └── steps ├── A01_001 │ ├── segment_nuclei │ │ ├── 0000.npz │ │ ├── 0001.npz │ │ ├── ... │ │ └── 0019.npz │ ├── tile │…
I was editing some markdown and had to insert a new figure in the middle. The problem is that this document already has an explicit figure numbering (e.g., “Figure 5”), so changing tens of figures felt dull. I like to run small (GNU) awk scripts for this type of tasks. # update_figures.awk { if ( match ( $ 0 , "Figure ([0-9]+)" , num )){ if ( num [ 1 ] > after ) gsub ( "Figure…
I needed to rename all occurrences of a pattern with another, where I knew there was no ambiguous situations. This uses ripgrep , xargs and GNU sed . source . rg old_pattern --files-with-matches | xargs sed -i 's/old_pattern/new_pattern/g'
One of the common challenges when analysing large bioimaging datasets is to bring it all together in one place. I usually use tools like DuckDB for database querying and copairs for selecting statistically significant subsets of the data. For one of my recent projects I built a marimo interface to explore the result of large-scale (~2TB images, ~2GB feature profiles) image-based profiles, then…
Create an empty branch with one empty commit Create new branch git checkout --orphan review-1-target Reset git reset . Clean branch git clean -df Add empty commit git commit --allow-empty -m 'Empty commit' Rebase a branch to put this commit at the root Push to your fork git push -u origin review-1-target Move to branch to review git checkout origin/main Spin-off branch from here git checkout -b…