Recent content in Home on rand[om]
I’m experimenting with a two-step pipeline for structured outputs. Instead of asking for JSON in a single prompt, I capture the raw free-text response first, then format it in a second call. Why bother? Schema validation errors don’t invalidate the LLM response. If the JSON is malformed, I only retry the formatting step. The original reasoning is still there. Most errors I see in data…
I call my LLM agents “Oompa-Loompas”. It started as a fun way to refer to them, especially when talking to non-technical people. They normally just know “ChatGPT”, but I use Claude Code, Codex, and a bunch of others. At some point it felt weird to keep saying “ChatGPT” when that’s not even what I’m using. So I needed a term that covered all of them.…
Claude Code has a nice feature where you can type @ followed by a file name to reference files in your conversation. The problem is that it’s broken. At least for me, it doesn’t work well when the .git folder is not in the repo root; like when using git worktrees. There’s an open issue about it. After some frustration, I decided to write my own file suggestion hook. The Hook…
In LLM coding, people will spend weeks learning frameworks and tooling, and spend basically zero time learning how to talk to the thing that’s actually writing the code. I’m pretty sure I’m the #1 power user of coding AI tools at my company (Claude Code, Codex, etc). People asked me to showcase how I use them. As I’m planning it, I notice the main idea I want to share is:…
Build tools add a lot of complexity. They solve real problems. But for small projects, they add layers of abstraction I don’t need. I want to write code and run it. No waiting for builds, no watching for changes, no debugging why the bundler failed. I wanted to test the React API without the build step. Preact is a good candidate: lightweight, same API, and it ships ES modules you can load…
I recently got frustrated with Codex’s command permissions. They don’t let you configure which commands should always be allowed or denied. There’s an issue about it, but it’s been open for almost 4 months as of writing this. Their sandboxing is less convenient than tools like Claude Code, since it’s more of an all or nothing approach. So I decided to build my own…
Creating new files is safer and simpler. The Problem with Appending Appending to files causes problems: Multiple threads writing to the same file requires locks Other processes might append to the same file without you knowing about it (very rare, but could happen) Buffered writers don’t flush immediately Race conditions corrupt data Hard to debug when writes fail It’s hard to remove…
This post shows how to run some Ansible playbook tasks in parallel. The approach requires tags for all tasks you want to run in parallel. How It Works The script runs multiple Ansible tasks simultaneously by: Defining a list of tags to run in parallel Using a ThreadPoolExecutor to create a thread for each tag Running the Ansible playbook (as a subprocess) with a specific tag in each thread The…
I stopped using shell aliases and moved everything into executable scripts. Main Benefits No need to source your .bashrc after adding new scripts No need to edit .bashrc , just drop executable files in your ~/bins folder (already in my $PATH ) Use immediately without reloading your shell Faster iteration (edit script, re-run command, repeat). Available in more places: Executable scripts in your…
I have been using the Cursor IDE for some time already and I really like its documentation indexing feature. Recently, I discovered a better way to leverage Cursor’s tools to make better references to official documentation and write better code. The local docs advantage Here’s what I’ve been doing: Instead of relying on URL-indexed documentation, I download official docs…
At some point in the past, I created a tool for a project (not relevant for this post). This tool generated multiple artefacts from an input (embeddings file, JSON metadata, and other files). The project was successful, but one thing I didn’t like about the “tool” I wrote was that it was not easy to manage those artefacts. The artefacts had to be uploaded and downloaded to some…
Some time ago, I found (and used for a while) the htl library. Before that, I was using raw template literals to build HTML, and then setting some element.innerHTML = newString . htl uses template literals, but instead returns a regular DOM Node object. This makes it a lot easier to add even listeners, setting other properties, and overall manipulating the object before appending it to the current…
I was recently reading the source code of mizu.js 1 and I liked the utility “function” 2 they have to handle keyboard shortcuts/combinations. I decided to slightly modify it to have it as part of my JavaScript utility functions. The function This function returns another function itself. You pass in a string that represents a combination of key presses like Shift+Alt+C , and it returns…
How to query a list of Python dictionaries using DuckDB, without previously converting to another format. We can pass the list of dictionaries to the query directly and use unnest to convert to a list of rows. We can also expand the struct to multiple columns. Note : all the dictionaries must have the same keys. # /// script # requires-python = ">=3.12" # dependencies = ["duckdb"] # /// import…
When you have a web application, almost always, you need to run some functions/scripts regularly. This what cron is for. I wanted to find an easy way to run “cron-like” jobs inside a FastAPI app. The app runs inside docker, and I just need those jobs to run as long as the app is also running. Since the app is “async”, I found I can just have recurring background tasks as…
Lately, I’ve been trying to learn more JavaScript. I wanted to explore web components. After checking out some frameworks and tools, mainly lit , tonic and facet , I wanted to build my own thing. It ended up being heavily inspired by tonic , but way simpler and with less features. I also really liked lit-html , which you can use as a separate package without having to use lit . It solves the…
DuckDB version 0.10.0 has been released , bringing some new array functions. You can use them to turn DuckDB into a vector database. (Make sure you have at least version 0.10.0 of DuckDB version to follow python3 -c "import duckdb; print(duckdb.__version__)" ) Insert data from numpy arrays import duckdb import numpy as np conn = duckdb . connect ( database = ":memory:" , read_only = False ) conn .…
This post could also be called “Or how to do React in Python” or “HTML as a function of state”. Most people use templating libraries like jinja2 to render HTML. I think that’s probably the best way to do it in production. However, for very simple / internal / proof-of-concept apps, I wanted to generate the HTML directly from Python functions to avoid needing extra…
These are just a few Python snippets that I have been using a lot for timing and profiling functions using only the standard library. You can also find the code here as a GitHub Gist import contextlib from functools import wraps import cProfile from typing import Optional import io import pstats import time from pathlib import Path from functools import wraps @contextlib . contextmanager def…
I recently read a post from vercel about porting turborepo to rust . In that post, they mention calling a go binary from the rust code instead of having to deal with FFI and C type compatibility. I wrote a post called “ Use a subprocess instead of a dependency ”, which follows a similar philosophy, although applied to something different. Here are some extra notes related to the vercel…
macOS AirPlay Receiver now listens on port 5000 by default, which conflicts with the default port used by some frameworks (like Flask ). This may cause some confusion if you’re trying to forward port 5000 from a different VM machine and you don’t see anything in your browser. To fix it, you need to uncheck the option in System Preferences -> Sharing -> AirPlay Receiver. Then it works…
I recently learned 1 a nice pattern to improve parameter handling in shell scripts. The pattern involves parameter expansion: #!/usr/bin/env bash set -euo pipefail function main { local arg1 = "${ A1 :? 'FAIL. Provide A1 var'}" echo $arg1 } main When calling the script (called t.sh throughout this post), it will fail: bash t.sh # t.sh: line 7: A1: FAIL. Provide A1 var But this will succeed: A1 =…
I like using Makefiles. They work great both as simple task runners as well as build systems for medium-size projects. This is my starter template for Python projects. Note: This blog post assumes some basic knowledge of how make and Makefiles work. Basic configuration I like using bash as the default shell. Then set some flags to exit on error ( -eu -o pipefail ), warn about undefined variables…
Before you read I found out the code here doesn’t work outside IPython, I didn’t realize that get_ipython() only works when running inside IPython. But the examples here work in isolated scripts if you run the script using ipython: ipython benchmarh.py . Introduction I like using IPython to measure the execution time of Python functions. You can also time functions using the timeit…
In Python you can create dataclasses with the wrong type. The type checker should show an error, but nothing prevents creating the object. This small function validates that each attribute is of the correct type by using the __annotations__ attribute of the dataclass . You can also create a base class to inherit from, but it won’t work if you override the __post_init__ method in the child…
As part of my recent experiments with mmap 1 I have learned how to share data between processes using a memory-mapped file. Here I’ll show how to do it between two independent Python processes, but the same principles apply to any programming language. Create a file to store the data I will use the tempfile module so that the file gets deleted after the script finishes. I prefer using this…
Here are some benchmarks I ran to compare the speed of running a list of regexes on all the Markdown files in my Obsidian folder (210 when I wrote this). The benchmark compares running the regexes on memory-mapped files versus loading the file contents as a string and running the regexes on that string. I’m using hyperfine to run the benchmarks. Considerations When we memory-map a file, we…
After trying a few note-taking apps, I ended up using Apple Notes. It was super fast on macOS and iOS. This was more or less my thinking: The syncing was good and it worked offline. But it had a couple of drawbacks. The format is not open. There are some scripts to reverse-engineer the Apple Notes SQLite database 1 , but that still needs a computer with iCloud sync, otherwise you can’t get…
This blog started as a place to share new things I learned about programming. I initially wrote Jupyter notebooks that I converted to Markdown. Then I started writing Markdown directly. After some time, I felt like having to open my text editor in my laptop to write a blog post was making me write less. It also felt like I had to write something longer than a tweet. That’s how I started…
Intro I was exploring different ways I could use Apple Notes as a CMS (Content Management System). I found out that you can add a Google account to sync some notes, but those notes are synced using the IMAP protocol to a folder in the email account. This post explains how to use the Gmail API to retrieve those notes. The idea for the CMS is: Write notes in Apple Notes, avoid using tables and…
I was recently reading the documentation of nnethercote/counts , in there, Nicholas explains a few pain points when using a table as a profiling output. But using SQLite as the “table” can relieve some of those pains. I frequently use SQLite for debugging and I wanted to share some techniques that have been helpful (and relate them to some arguments mentioned in nnethercore/counts ) Defining the…
Here are some utility functions that have been useful when writing automation scripts. requests I like using requests or httpx , but if you add a dependency to your script, now you’ll need to create an environment, install the depdencies, etc. just to run it. I like keeping automation scripts in a single file I can copy around. Instead of using the requests , module, you can use this. This…
I wanted to query a large TSV file stored in S3. To achieve this, I decided to convert it to Parquet and query it using DuckDB . However, I didn’t want to download the full file and then convert it. Instead, I wanted to stream the CSV file directly from S3 and write the output to a Parquet file. Here are a couple of approaches that worked quite nicely. Setup I’m running these…
Sometimes calling a subprocess is better than using a dependency/package. At least in Python, once you add a third-party dependency, distribution becomes slightly harder. I like writing automation scripts in a single .py file. If that script doesn’t use any third-party dependencies, distributing it is as easy as copying the file to the machine. Otherwise you need to package your project,…
tl;dr conn.execute ( "select * from ... where ... in (select value from json_each(?))" , ( json.dumps ( list_of_values ) , ) , ) Lists of values in SQLite SQLite doesn’t have an array type as Postgres does, but we can exploit the JSON extension to do operations with arrays. An example of why this may be needed. Let’s say you have a table with names, and you want to select all the id ’s…
Engineers like automating tasks. You find yourself running a set of commands, copying values between files and prompts, and you think I can automate this! Aside from the joke: “why spend 30 minutes doing something when you can spend 2 days writing a program to automate it”, I believe we don’t think enough about automation. Here I want to share some ideas. Broken automation is more painful than…
In this post, I want to share some reasons why using SQLite turns into a pretty convenient developer experience. This is a “live” post that may be updated in the future with more contents. Moving and sharing data, backups Sharing SQLite data is straightforward, just rsync the file to a new VM or upload it to any file storage service. When you want to share data from other databases, it usually…
One of the main pain points of using SQLite in production deployments or VMs is managing the database. There are lots of database GUIs, but only work with local SQLite databases. Managing an SQLite database remotely requires: Adding a new service to the deployment (like Adminer , sqlite-web or postlite ) Giving the new service permissions to access the volume with the database Exposing a port to…
Python generators are mighty, but they lack a couple of useful features. One of them is peeking the next item without consuming the generator. Even better, what if we could peek any number of items? Another feature lacking from generators is rewinding/backtracking. We will implement both of those features in a couple of different ways. Note: if you need a better tested implementation that lets you…
SQLite has a powerful extension mechanism: loadable extensions . Being an in-process database, SQLite has other extensions mechanisms like application-defined functions (UDF for short). But UDFs have some shortcomings: They’re local to an SQLite connection, not shared for every process connected to the DB They have to be defined in your program. That means that you need to have the function…
If you have ever used the built-in sqlite3 module in a multithreaded Python application, you may have seen this message. sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 4316169600 and this is thread id 6190804992. This code can be used to reproduce the error: from threading import Thread import sqlite3 conn =…
The best code is the one that is easy to understand. The problem Source People don’t focus on code readability. Readability is like writing documentation. It takes extra time, more key presses, and you need to stick to it to make it work in the long term. However, sometimes people write code like this: settings = { "foo" : 1 , "bar" : 2 , "baz" : 3 } results = [ parse_document ( settings , r…
In this post I want to explain a new deployment method I came up with while working on drwn.io . I wanted it to meet a few requirements: Simple Based on git tags Zero-downtime Easy rollbacks Creating an empty remote in the server Imagine you already have your project with some code that is being synchronized with a git service like GitHub. To have a git push based deployment, we need to have our…
A Bloom filter is a probabilistic data structure present in many common applications. Its purpose is answering the question: “is this item in the set?” very fast and not using a lot of space. The answers can be NO, or MAYBE YES. They work using hash functions, we learned about them some time ago. For example, one use case of Bloom filters is the following: you have a huge list of…
What is a Bloom Filter? A Bloom filter is a probabilistic data structure. It tells you if an element is in a set or not in a very fast and memory-efficient way. A Bloom filter can tell if an element is not in the set (“being 100% sure”) or that it may be in the set, but not “being 100% sure”. It only has 2 operations: add , to add an element, and query , to check if an…
Hash functions are a fundamental part of modern programming. They map a variable-sized input to a fixed-size output. We can use them to: Verify the integrity of files and make sure they have not been modified. Save a password in a database without the risk that if somebody steals the data, they will be able to know the passwords. Create a hash table to store items. With that, we can find them…
This works only in the same Python process, you can’t share an in-memory SQLite database between processes in this way. import sqlite3 # NOTE: you need to use uri=True # 3 connections to the same in-memory database (DB1 / memdb1) DB1_1 = sqlite3 . connect ( "file:memdb1?mode=memory&cache=shared" , uri = True ) DB1_2 = sqlite3 . connect ( "file:memdb1?mode=memory&cache=shared" , uri = True )…
Feedback is one of the most importante things when you are creating a product (probably the most important one!), so it should be easy for users to give feedback. In this post we’ll see how to implement a feedback button easily. This exact code is what I’m using at drwn.io , a little project I’m working on with a friend. The button does the following: after you use it, it will…
Some time ago I published a post about managing Conda environments. My workflow has changed quite a bit since then, and I’ve moved to venv + pip-tools , but I’ll leave that for another post. Today I want to talk about new experiments I’m making with mamba + pip-tools . If you don’t know about mamba, it’s kind of a miniconda replacement. It just works a lot better, and…
Storytelling is part of the human essence. Stories have let humans survive until today, they became the medium to move information between individuals. Programming can be considered another form of moving information. With different levels of abstraction, programming becomes a medium to transfer information between humans and computers. A programming paradigm mixing those two concepts is literate…