Recent content on Anders Poirel
This benchmark compares the join in Snowflake performance of surrogate keys generated via md5 versus surrogate keys generated via md5_number_lower64. Run this in a schema you have write permissions on and on a dedicated warehouse to avoid interference with the results. create or replace transient table benchmark_dim as select row_number() over (order by null) as natural_key ,…
In the extra-load-transform (ELT) paradigm of data engineering, a popular method to generate surrogate keys in the transformation step is to compute a MD5 hash of uniquely identifying columns and store it as a string. This is the approach of dbt’s built-in generate_surrogate_key macro. The arguments for using a hash instead of an incrementing integer sequence are well worn. Hash keys are…
It’s hard to get around the need to manually inspect real inputs and outputs when working on a data pipeline. Unit tests with well-formed synthetic rows won’t save you if the actual source data look like this: | ticket_id | category | source | ... |-----------+-------------+----------+---- | 32 | bug | customer | ... | 1 | BUG | customer | | 47 | feature | internal | | 16 | feat.
Python dictionaries are available without an import and extremely flexible, which means many Python programmers default to representing data as a dict. Here’s why and when you should use dataclasses instead. Note: I’m using dataclass here since it’s in the standard library. If you’re already using a similar 3rd-party library like the excellent attrs the advice here still…
PEP-249 (“Python Database API Specification” a.k.a. “DBAPI”) defines a standard interface for database access in Python, implemented by libraries such as psycopg and PyMySQL. One common task with DBAPI libraries that is not immediately obvious is identifying column names in query results, since fetching from a Cursor returns a sequence of list-like rows. For instance, in…
I’m a software engineer and data professional, most recently at Plasmidsaurus. To learn more about my professional work, check out my CV. The contents of this blog reflect my personal opinions only. Contact The best way to get in touch with me is email: contact@anderspoirel.net. Don’t hesitate to reach out about anything on this website! You can also find me on: GitHub LinkedIn…
Software engineering Alex Miller Antirez Armin Ronacher Bob Nystrom Brandur Cal Peterson Charity Majors Dan Luu Decyphering Glyph Fred Hebert Eli Bendersky Jason Fantl Jeremy Bowers Li Haoyi Hilel Wayne Hynek Schlawack Irrational Exuberance Martin Sústrik Matt Dugan Nathaniel J. Smith Sympolymathesy Swizec Teller Tania Rascia Tom MacWright Write More Tests zwischenzugs Mathematics, statistics &…
Technical Expertise I’m a software engineer, with a particular interest in enabling and doing “data work” writ large, whether this means building tool libraries, designing a data architecture, standing up platforms and infrastructure, analyzing data or fitting the occasional model. Languages & tools: Python (especially the data stack), Go, SQL, Apache Spark & Delta Lake,…