I have had mobile-pocket on my phone for a while, but lately it started telling me about their privacy policy and how they track me. That prompted me to try other apps and I tried My Barcodes but at the fourth barcode, it asked me for money. Then I decided to vibe-code a barcode wallet progressive web app (PWA) for my phone and start using that instead. It is fully open source, does not track, and…
There are quite some opinions out there on how to dockerize Python projects. I've had many a discussion on how to put Poetry inside Docker and now that I'm replacing Poetry with uv every where I can, I feel it's best to also post my opinion on how to do that. So let's jump into building a Dockerfile for a Python project maintained with uv .
Almost 3 years ago, I wrote down a Getting started with Python Poetry . Now that I have started to use uv instead, I decided to run through the same example with uv instead of Poetry .
There are plenty of date calculators on the web, but if they don't push adds in your face constantly, they push cookie warnings. And on top of that, because this is an ancient issue, they have you send your date on a roundtrip to what feels like an old PHP server in a dusty closet. So I decided to write my own: instant client-side results using Rust .
I love working with Polars instead of Pandas because of the simpler index and high performance. What I needed as a data engineer was loading and storing Hive partitioned datasets . This is supported through Arrow , but support fell short of what I required. I decided to write and publish Polario : a simple pure Python Polars Hive partitioned dataset handler library.
There is a large number of opinions about how to deal with data and organize tables. But I think there are a few things that I end up doing for all table related information, that I might as well try to formalize in a blog post. Be explicit when naming things. This will make working with the tables later easier because you will end up with fewer name collisions when joining and fewer mistakes.…
Writing a Polars DataFrame as a PyArrow parquet dataset onto remote storage turned out to be more than a 10 minute look up and write up. So I decided to have a quick write-up of my findings.
A while back I started a list of fun online activities for teams on Github. It was not really a success and I was unable to find anybody willing to add things to the list. I still think some of the content is nice to have, so I decided to archive it here in this blog post.
Snowflake has some table statistics available on the tables DBT creates. But if we want data observability metrics on what happened over time, we have to implement some custom SQL. In this blog post we will set up a dbt incremental model to track changes in these table statistics using the modification time as a unique key.
Here is the scenario: you have two private Github repositories, one with a Python package, another with a dependent project. And you do not have access to a Python repository you can use to host the package. One solution is to set up direct access from one repository to the other using a deployment key and have Python Poetry grab the package from the git repository directly. This blog post will go…
There is a plugin for vscode that allows you to run dbt code, but it is not available for the open source version of vscode . Alternatively you can define a build task in vscode that allows you to quickly run DBT on your current SQL file.
Using Poetry in a Github action is simple. However, because it's nice to copy-paste a good start, here is a quick snippet of how you can use it.
For some reason I can't find proper documentation on the AWS cloudwatch log filtering syntax. They support regex filtering using like /your regex/ but I can't find any documentation on the regex pattern syntax, so let's assume PCRE. Because of the syntax of cloudwatch query language, we can't apply modifiers outside of the forward-slashes. This leaves us with only being able to use inline modifier…
Snowflake has introduced a new GUI called Snowsight. This includes a nicer interface and primitive dashboarding functionality . In this blog I'll walk through how I used it to analyse our overall db run time. We start by optimizing our warehouse usage, then look at longest running queries, and finally visualize the timing of the complete run in a Gantt chart .
If you have a model in dbt and want to make sure it has at least N number of rows, here is how to write a generic test you can add to your schema.yml .
Poetry will enable you to easily manage dependencies, create packages and manage your development virtual environment. In this blog post I'll give you a quick introduction to get you started using Poetry to manage your Python project.
Yeah, Cloudflare pages is out of public beta . To celebrate, let's deploy some Elm code on there using Yarn .
Most build tools create lock files, including poetry for Python . Now what if you want to deploy the dependencies in the lock file on a Databricks cluster? One approach is to add a requirements file to your python package and on the cluster extract that extra file and use custom logic on the cluster to install the dependencies from the requirements file. At that point you are spreading the logic…
Programs can be configured. When you talk to people about program configuration, they often think of things like what theme the application uses, the locale formatting, wether you want notifications turned on or not, etc. This is user configuration, not application configuration set by a developer. These are different kinds of configuration. In this blog post I'll explain three types of…
There are a lot of people a bit lost as to what DevOps actually is. Luckily, there is a great page from Amazon explaining what DevOps is . Because the approach is really basic, the discussion on what you should do to be DevOps has gone all over the place. Some people even try to quantify it with research which is pretty nerdy: concrete examples of what it means to be DevOps rely heavily on the…