In my previous post I walked through how I designed a new query language called Entity History Query Language (EHQL) for a group of semi- and non-technical analysts working with vehicle maintenance data. Here I will show you how to implement a working skeleton of that language using Python and Apache Spark.
I recently had the pleasure of designing and implementing a custom query language1, along with an integrated execution environment. It was my first time doing anything like this, and it became a passion project I dedicated many nights and weekends to. I learned more working on this side project on and off over the course of about 18 months than I did at my day job in the past eight years. More…
What’s the difference between a data constraint and a data query? Is there anything that can be expressed in one form but not the other? My sense is that there is no such thing.
Say you run an online book store and want to build a data pipeline that figures out who the top-selling authors are. Logically, the input to the pipeline is a log of every individual book purchase on the store for all time, along with details about each book like who authored it. And the output is a list of the top-selling authors per month.
Many years ago, working as a database developer at a video game company, I was tasked with designing the database behind an in-game wallet service. The wallet service would store each player’s current balance and transaction history. In other words, it was a simple banking database.
I was recently talking to some coworkers about the mix of data technology we have in our stack. Apache Spark, HDFS, Amazon Athena, Amazon S3, AWS Glue… The list is long. The technologies obviously work together somehow, but to a newcomer it may not be clear how each technology relates to the other. And in the details of how a given technology works it’s easy to lose sight of what purpose it serves…
A couple of co-workers who are new to database technology recently asked me why we use both Postgres and Redshift in our stack. They’re both SQL databases and seem to do the same thing. So why not just use one technology? It would be simpler.
In the movie Die Hard with a Vengeance (aka Die Hard 3), there is a famous scene where John McClane (Bruce Willis) and Zeus Carver (Samuel L. Jackson) are forced to solve a problem or be blown up: Given a 3 gallon jug and 5 gallon jug, how do you measure out exactly 4 gallons of water?