Recent content on A Blog
Five hundred years ago, Martin Luther nailed a list of grievances to a church door. His complaint was essentially that the institution had buried the faith under centuries of bureaucracy and toll-collection—indulgences, intermediaries, a whole priesthood standing between the believer and God—and that you could get back to what mattered by cutting all of it away.
I gave a talk to open the 2025 Arrow Summit in Paris, looking back at the history of the Apache Arrow project, some factors that have contributed to its success, and what the past can tell us about how to ensure a bright future. The talk wasn’t recorded, and while I’ve published the slides , they don’t alone capture what the talk was like, so I thought I would publish a version…
When my kids were little, we spent lots of time at the playground. The hardest part was when it was time to leave: they were having fun and didn’t want to go. When you have to drag a screaming kid off the play structure, it’s unpleasant for everyone, so we needed a way to get them to come willingly. The trick we landed on was to give them a 5-minute warning that we’re going to…
I’m not a morning person, but in the household division of labor, I’m the one responsible for getting the kids up and out to school in the morning. It’s not ideal. I’m not happily awake or fully caffeinated, and I just want the kids to comply the first time I tell them to get their shoes on. When we’re behind schedule getting out the door, I can lose my patience. My…
A rip current is a powerful flow that can drag an unsuspecting swimmer out to sea. When visiting a beach with rip currents, you’re told that if you get caught in one, you shouldn’t try to swim back toward the beach, against the current—it’s too strong. Instead, you should swim parallel to the beach to get out of the current, and then you can get back to shore more easily.
One of the great things about leading a software project is the sense of ownership over what you and your team are creating. Your mind is buzzing with the big picture of what your project can do and will be able to do. You also sweat the details, and particularly if you’ve been with it from the beginning, you have unrivaled breadth and depth of knowledge about the project. So, when…
When interviewing developers at Crunch , we sometimes do a live coding exercise using a JavaScript data visualization library. It’s basic, and we don’t assume any prior knowledge of the library. To start, we tell the candidate that they can use the full internet to help them solve the problem (we’re trying to simulate real working conditions, after all), and by the way, this…
This week, I wanted to analyze some web traffic patterns back over several years. For contemporary analysis, we sync the last couple of months of load balancer (AWS ELB) log files from S3 to a EC2 instance, and I’ve written some tools in R to query them using dplyr methods. The elbr R package I wrote does some clever things to take advantage of the directory structure for fast filtering by…
At Crunch , we’ve built up our product over the span of six years, and what we’ve built helps a lot of people analyze their data quickly and painlessly. As ever, our ambitions are great and we want to build more. Sometimes this requires changing data contracts and APIs, and that poses an additional challenge: not only do we have to design a good solution to our users’ problems,…
The 3.2.0 release of httptest is now on CRAN . This is a minor release with several enhancements suggested by package users. Details can be found in the release notes . A few of the changes improve the behavior of httptest in interactive use. There is one new function, use_httptest() , which configures your package to use httptest when you’re setting it up. Thanks to everyone who filed…
In putting up the slides for the talk I gave at last week’s Bay Area R meetup , I figured it would be good to dig up the slides from previous talks and post them here too. (In fact, I linked to one of them in the slides from last week.) So, there’s now a “Talks” section of the site.
As a product owner and development leader, I often hear that “we have to pay our technical debt” in order to proceed with our product plan. As the metaphor goes, when you choose to take shortcuts now, you incur a debt that you’ll eventually have to repay with interest. The implication when a developer raises it is that those interest payments are too high and are preventing us…
Recently I analyzed some web traffic for my team at an all-hands meetup. I suspected there were some misconceptions about the frequency of certain kinds of requests, and that some basic descriptive statistics could help. Whenever I start an analysis like this, I also think about how to make it easier to repeat it. If the question is interesting enough to merit analysis now, it will probably be…
When you submit a package update to CRAN, they ask you to check some boxes confirming that you’ve done your due diligence on your submission: that it passes its checks. One confirmation is that you’ve addressed any failures on the CRAN package check page, which looks like this . Hey, all passing today! This page is a window into one of the key features of CRAN: it runs…
Publishing a website for your R package is simple with pkgdown . As I recently discussed , you can build a basic site and host it on GitHub Pages with just a handful of commands and clicks. That’s a great start, but why stop there? If you have a personal website or blog, it would be nice to have your package page mesh with it—that is, it should fit with your “brand”. Your…
pkgdown is an incredibly powerful tool for building beautiful websites for R packages. With pkgdown and services like GitHub Pages , deploying a package website is so simple and straightforward that I’m starting to see it as an essential part of writing a package. What’s more, I’ve found that the act of preparing a package website has led me to improve the package itself,…
The 2.3.2 release of httptest has just been published on CRAN . Since the 2.0.0 release in June, several key additions have been made. See the NEWS for the full list. If you follow those links, you’ll see the first new “feature”: httptest now has a proper website, using the pkgdown site generator! In terms of functionality, the biggest change has been the addition of a framework…
At Crunch , I occasionally pick up small coding projects to help with project management . Recently, I wanted a way to get automated reporting on our open issues in Usersnap , a service embedded in our web application that allows users to report bugs from within the app. My thought was to set up a cron job to report daily summaries in our Slack channel on things like the number of reports created,…
At Crunch , I lead both engineering and product management. Over the last few years, what that entails has changed greatly. How I spent my day when we were getting started, with a team of four developers and zero users, is very different from how I spend my time now with 16 on the team and thousands of users around the world. In fact, my day-to-day has evolved through several phases as we’ve…
I’m a big advocate of testing and test automation, both on the team I lead and in my own projects . Tests provide valuable evidence that your code works. Without tests, you’re engaging in faith-based development : you believe your code works because you believe in your own infallibility as a coder. Theology aside, as a practical matter, tests are liberating because they allow you to…
Over on the Crunch.io dev blog , I wrote about how we used Travis-CI to automate the building of our various static sites. I thought I’d do the same for my personal blog, though with one twist: using R and the blogdown package on top of Hugo. This blogdown GitHub issue pointed to Yihui’s guide to building bookdown projects on Travis , as well as a project that had adapted that flow for…
This page left intentionally blank.