Photo credit C D-X on Unsplash A deeply relatable aspect of agentic coding is their use of progressive disclosure in skills and references. Most of my career successes have come from integrating ideas or approaches from unrelated disciplines. As a result, I’m enjoy foraging for information and surveying the landscape. I may not have the full “skill” in my “context”, but I try to always add new…
Quick Links Test Set Episode Website Apple Podcasts | Spotify | YouTube Episode Notes Emily’s had a wild ride through modeling, data engineering, machine learning, and back again, and she knows a thing or three about the evolution of SQL tooling (from nightmare multi-page scripts to the dbt renaissance). She reveals how building internal packages became her gateway to making work enjoyable. Plus:…
In October, the Python Software Foundation announced that it had made the difficult decision to forgo a $1.5M National Science Foundation grant. The grant was intended to improve structural vulnerabilities in the Python language and PyPI, but came with the unpalatable stipulation that no part of the PSF as a whole could “operate any programs that advance or promote DEI [diversity, equity, and…
Photo credit to Hans-Jurgen Mager on Unsplash polars API is a delight in part because of its consistency. Transformations are chained sequentially onto the DataFrame in a consistent series of steps without leaving the DataFrame. This helps developers get “in the flow”, produces highly readable and well-structured code, and cna make a very natural transition for users coming from R’s tidyverse who…
So, you build a great predictive model. Now what ? MLOps is hard. Deploying a model involves different tools, skills, and risks than model development. This dooms some data science projects to die on their creator’s hard drive. Tools like dbt and SQLMesh entered the scene to solve a similar problem for data analysts. These tools offer an opinionatee frameowrk for organizing multiple related SQL…
Data science stakeholder communication is hard. The typical explanation of this is to parody data scientists as “too technical” to communicate with their audiences. But I’ve always found it unsatisfying to believe that “being technical” makes it too challenging to connect with the 0.1% of the population so similar to ourselves that we all happen to work in the same organization. Instead, I believe…
Posit’s recently-announced project orbital translates fitted SciKitLearn pipelines to SQL for easy prediction scoring at scale. This project has many exciting applications to deploy models for batch prediction with near-zero dependencies or custom infrastructure and have scores accessible to operatilize from their data warehouse. As soon as I heard about the project, I was eager to test it out.…
Quick Links Podcast Episode Casual Inference is a podcast on all things epidemiology, statistics, data science, causal inference, and public health. Sponsored by the American Journal of Epidemiology. As a guest on this episode, I discuss data science communication, the different challenges of causal analysis in industry versus academia, and much more.
Quick Links Abstract Slides Slides This talk was part of a symposium on data science tools and opportunities for adoption in epidemiology. The full session description is provided below: Most applied research and education in epidemiology does not yet benefit from modern data science. Fledgling epidemiologists may receive cutting-edge education on the theory of epidemiologic methods, but remain…
Photo credit to the inimitable Allison Horst About a year ago, I wrote the original version of Python Rgonomics to help fellow former R users who were entering into the world of python. The general point of the article was that new python tooling (e.g. polars versus pandas ) has evolved to a point where there are tools that remain truly performant and pythonic while still having a more similar…
Literate programming tools like R Markdown and Quarto make it easy to convert analyses into aesthetic documents, dashbaords, and websites for public sharing. But what if you don’t want your results too public? I recently was working on a project that required me to set up a large number of dashboards with similar content but different data for about 10 small, separate organizations. As I…
Quick Links Abstract Slides Video Slides Video Post - Python Rgonomics Post - Advanced polars versus dplyr Warning Tooling changes quickly. Since this talk occured, Astral’s uv project has come out as a very strong contender to replace pyenv , pdm , and more of the devtools part of a python stack. Data science languages are increasingly interoperable with advances like Arrow, Quarto, and Posit…
Credible documentation is the best tool for working with data. Short of that, labor (and computational) intensive validation may be required. Recently, I had the opportunity to expand on these ideas in a cross-post with Select Star . I explore how a “good” data analyst can interrogate a dataset with expensive queries and, more importantly, how best-in-class data products eliminate the need for…
Photo credit to David Clode on Unsplash In the past few weeks, I’ve been writing about a stack of tools and specific packages like polars that may help R users feel “at home” when working in python due to similiar ergonomics. However, one common snag in switching languages is ramping up on common “recipes” for higher-level workflows (e.g. how to build a sklearn modeling pipeline) but missing a…
We’ve all worked with poorly documented dataset, and we all know it isn’t pretty. However, it’s surprisingly easy for teams to continue to fall into “documentation debt” and deprioritize this foundational work in favor of flashy new projects. These tradeoff discussions may become even more painful in 2024 as teams are continually asked to do more with less. Recently, I had the opportunity to…
Photo credit to Hans-Jurgen Mager on Unsplash A few weeks ago, I shared some recommended modern python tools and libraries that I believe have the most similar ergonomics for R (specifically tidyverse ) converts. This post expands on that one with a focus on the polars library. At the surface level, all data wrangling libraries have roughly the same functionality. Operations like selecting…
Documentation can be a make-or-break for the success of a data initiative, but it’s too often considered an optional nice-to-have. I’m a big believer that writing is thinking. Similarly, documenting is planning, executing, and validating. Previously, I’ve explored how we can create latent and lasting documentation of data products and how column names can be self documenting . Recently, I had the…
Photo credit to the inimitable Allison Horst Warning Some advice in this post has gone stale regarding IDEs, installers, and environment management tools. Please see me 2025 update for more recent thoughts following the release of uv and Positron Interoperability was a key theme in open-source data languages in 2023. Ongoing innovations in Arrow (a language-agnostic in-memory standard for data…
Last week, I enjoyed attending parts of the annual virtual Causal Data Science Meeting organized by researchers from Maastricht University, Netherlands, and Copenhagen Business School, Denmark. This has been one of my favorite virtual events since the first iteration in 2020, and I find it consistently highlights the best of the causal research community: brining together industry and academia…
Abstract In October, I joined a Halloween-themed panel along with Chad Sanderson and Joe Reis to discuss our horror stories of data quality gone wrong and how to build successful data quality strategies in large organizations. Key takeaways are summarized on Monte Carlo’s blog .