A blog on Tech, Data, Architecture and People
In the past few years, we’ve seen a Cambrian explosion of new columnar formats, challenging the hegemony of Parquet: Lance, Fastlanes, Nimble, Vortex, AnyBlox, F3 (File Format for the Future). The thinking is that the context has changed so much that the design of yore (the previous decade) is not going to cut it moving forward. This seemed a bit intriguing to me, especially since the main…
This is just me sharing my experience trying out AI coding. Don’t get fooled by the curmudgeonly engineer having to change his ways. If you get past my stumbling through what might be obvious beginner mistakes, and the (sometimes humorous) grumbling that accompanies it, you’ll get to the happy ending of the tool I was able to create fairly quickly, using languages and technologies I am totally…
The topic of software architecture has become a bit cringe. Some people will roll their eyes at the mere mention of it. My impression is this is because it has often been a very top down practice. An architecture committee approval must be secured before starting anything new. Nothing gets done unless it’s been vetted by “the architect”. The people who are closest to the problem being solved must…
As I looked back at my experience working in data engineering for this post, I realized I never really consciously decided to specialize in data. It just kind of happened. The company I was working for was acquired by Yahoo! where Hadoop was emerging as the next industry leap in data processing. As I dug deeper in the platforms I was using and became interested in open source software, I…
Lineage has long been a requirement for anyone processing data - whether for complying with regulations, ensuring data reliability or, to quote Marvin Gaye, plainly just knowing what’s going on from provenance to impact analysis. However, our industry has historically had difficulties collecting data lineage reliably. From the early days of lineage powered by spreadsheets, we’ve come a long way…
In 2018, I wanted to describe how the components of databases, distributed or not, were being commoditized as individual parts that anyone could recombine into use-case specific engines. Given your own constraints, you can leverage those components to build a query engine that solves your problem much faster than building everything from the ground up. I liked the idea of calling it “the…
Recently I’ve signed up for an improv class and it’s been a lot of fun. It had been way too long since the last time I had taken classes, back I was at Twitter, and I wish I had done this earlier. That class I took ten years ago was part of “Twitter University”, a program designed to help employees develop their skills. There, you could learn about many topics from programming Scala to Improv’.…
Much the same as there was a common need for a columnar file format and a columnar in-memory representation, there's a common need for lineage across the data ecosystem. In this chapter, I'm telling the story of how OpenLineage came to be and filled that need.
In 2015, a discussion started in the Parquet community around the need for an in-memory columnar format. The goal was to enable vectorization of query engines and interoperability of data exchange. The requirements were different enough from Parquet to warrant the creation of a different format, one focused on in-memory processing.
15 years ago (2007-2011) I was at Yahoo! working with Map/Reduce and Apache Pig, which was the better Map/Reduce at the time. The Dremel paper just came out and, as everything I worked with seemed to be inspired from Google papers, I read it. I could see it applying to what we were doing at Yahoo! and this was to become a big inspiration for my future work.