One of my main inspiration these last few years, especially for my consultancy, is Nikhil Suresh, aka Ludic . Last week, I spoke to him to ask him about his experience setting up Hermit Tech , consulting practices and pitfalls, and funny stories. I got permission to share most of my notes (except the parts that might be legally dicey) here. This reflects my own notes, paraphrasing and moving…
Or, Deep Work is Illegible Or, Everything Worth Learning You'll Have to Teach Yourself I remember quite clearly how I learned Git. Spring quarter of my second year of undergrad, I took Data Ethics, which was a required course in the Data Science major. This was a mistake. The intended path was to take Computer Science for Data Scientists after completing the introductory courses, and take Data…
Or, Breaking 2 GB/s on Streaming JSON Queries This is a long one, with notes on the ongoing development and re-development of crowley , some light literature review, and armchair theorizing about how to think about JSON querying as a process. Headers: SIMD Streaming JSON Parsing Langdale & Lemire (2019) Redux Rust Implementation The Frustrating Catch What This Structure Enables Lurching Towards a…
Previously: Rust is 2x faster than Rust: PyO3 Edition Rust, Python, and You: Rust-Python Interop for Python devs Rust-Python Interop for Scientific Codebases All of which are basically the same blog post for slightly different audiences. In light of a conversation I had last week, I figured it was worthwhile to write up the main pitch for Rust in scientific programming, specifically for…
A few days ago I read Micah Kepe's jsongrep is faster than {jq, jmespath, jsonpath-rust, jql} . There, he promoted an approach to high-performance JSON querying using DFAs generated by a regex-inspired JSON query language, with seriously impressive benchmarks to boot. As so often happens, I wondered 'how can I get my old data science pals to use this?' and because I am a river to my people,…
After several months working part-time on optimizing McFACTS, I think we're rapidly approaching saturation: one by one, we've cut the tall poppies down, cooled the hot paths, until the largest contributors to the runtime are either bookkeeping functions which are already very simple as they are, or calculations that are genuinely time-consuming and which we've already optimized a good deal. Well,…
For the latest batch of optimizations speeding up/replacing AstroPy unit operations in McFACTS (expect my next blog post titled The Agony and the Ecstasy of AstroPy Units ), I've wound up just extracting the string representation of the unit type and matching on it directly: fn extract_unit (ob: & Bound <' py, PyAny > ) { // extract unit value let unit_obj = ob.getattr( "unit" ) ? ; // use python…
I was reading David Nicholson's Domain-driven software design is a good idea, still a couple days ago, and it struck a few thoughts loose in my head. By way of summary, though you should read the original, Nicholson introduces the concept of 'Domain-Driven Design' coined by Eric Evans in his 2003 book of the same name, though the general concept is older. In Nicholson's rendition, DDD involves…
Yesterday , I shared a rather lengthy post about various ways to optimize NumPy array operations with lower-level solutions. One of the findings was that a zipped Rust array could be faster than NumPy, but it depended on several factors: How many binary NumPy operations are you performing? NumPy performs an intermediate array allocation on each operation, so the more operations you can fuse into a…
Or, Zipped Arrays Rule Everything Around Me (see followup post ) I've been contributing on and off to the McFACTS simulation project for a few months now, and am currently in the process of testing and merging some pretty substantial optimizations to the core 'business logic' of the simulation, which mainly consists of long series of NumPy array operations. Pretty much all the optimizations in the…
You are reading this because you, or someone close to you, is learning Python for scientific programming. This post intends to be a launchpad, providing introductions to the main concepts involved, some guidance on the Python in general, on the libraries most commonly used for scientific Python, and on the best tools to use going into 2026. It also spends some time on improving your developer…
A few days ago, I read Victor Skvortsov's I Switched From VSCode to Zed on Lobsters. It was a neat, short article, not too applicable to me since I don't use Zed very often (I just fall back to it periodically when I can't get nvim LSPs working for a language), but had a decent overview of modern Python language servers. I was caught a bit off-guard to see this comment below the article, wrote a…
A week ago, I wrote a quick blog post on some attempts at Rust optimization of C-Python functions in the basil-core codebase for calculating orbital decay functions, and found that a quite straightforward, one-to-one conversion of the existing C code with PyO3 resulted in code that was simpler, more readable, and in many cases faster. In the past week, Vera and I have been working more closely,…
As a method of helping pass the time while waiting for CNNs to converge, I've been experimenting with converting some numerical C-Python libraries to Rust-Python with PyO3. Today, I'm looking at V.E. Delfavero's basil-core C library, which is the optimized core of the BASIL (Bayesian Astronomical Sampling and Integrating Library) Python package. Specifically, a set of recently updated C functions…
This is the outline of a talk I originally gave at the September 24th RustNYC meetup. The slides are also available here . Note that it is not a tutorial on how to begin using PyO3 for Rust-Python interop: my tutorial on that subject is here . Welcome to Rust-Python Interop for Scientific Codebases ! Slide 4 When evaluating Rust-Python interop, we can ask two very pedantic questions. Why Rust, and…
This week's Computer Vision homework has us reading through some ML papers. One of these is Adam: A Method for Stochastic Optimization Kingma & Ba (2015). Adam is a very widely used gradient descent optimization algorithm that comes as standard in lots of ML frameworks, though recently it's got some competition from the polar Muon algorithm. Reading the paper, I decided to quickly implement a…
For my internship this summer, I developed a set of Rust utilities on top of the Slurm job scheduler. Slurm is written in C, so to do that, I first had to build an API, and to do that, I first had to build an FFI. This is how I approached it. Bindings At least one person has tried their hand at this problem before: Peter Williams created the slurm-rs repo years ago, and gave up around 2018,…
In my ongoing quest to teach myself graphics, I sat down on Friday to work through Shirley, Black, and Hollash's Ray Tracing in One Weekend . I sit here on Monday without having finished it yet: perhaps Shirley et al were unaware of Hofstadter's Law. Or perhaps it might have something to do with the book being written in C++, a language I have never before used and of which I know very little.…
In the course of preparing a talk on Rust-Python interoperation, I have come to learn terrible, horrible things about Python's memory model. But first, what is the Python memory model? Even if you're a Python developer, you may not be aware of any such thing: indeed, that was my experience in a job interview. Asked to describe the Rust memory model, I managed just fine. Asked to describe Python's,…
This is an outline of a talk I originally gave at the Center for Computational Neuroscience at the Flatiron Institute, on August 5th, 2025. All code for this talk is available on github . Welcome to Python, Rust, and You: Modern Py-Rust Interoperation ! Please open up your preferred way to execute python: your terminal, Jupyter Notebook, python script, what have you. If you have the Rust toolchain…