The City of Chicago now publishes trip-level data for every ride-hail trip taken since November 1, 2018. Chicago isn’t the first major US city to make ride-hailing trip data publicly available—New York has published Uber and Lyft data since 2014—but the Chicago dataset includes additional fields, most notably fare amounts, that provide new insights into the ride-hailing landscape. In particular,…
The New York Police Department provides data for every motor vehicle collision in NYC since July 2012. Each record includes location coordinates and other metadata, most notably the number of injuries and fatalities, segmented further by motorists, cyclists, and pedestrians. I wrote some code to process the raw data , and built an interactive heatmap of 1.4 million collisions between July 2012 and…
If you’ve been on a New York City subway platform since January 2018, you should have noticed a countdown clock that displayed an estimate of when the next train would arrive. Although there’s no official record of when trains actually stopped at each station, the countdown clock data can be used to approximate. Over the past 5 months, I’ve collected and processed some 24 million stops’ worth of…
I wrote an open-source app called NBA Shots DB that uses the NBA Stats API to populate a database with all 4.5 million shots attempted in NBA games since 1996. The app also processes a dataset provided by Sportradar of over 1 million NCAA men’s shot attempts since 2013 into a format that can be merged with the NBA data. Both datasets include similar information: location coordinates, player and…
Every day in New York City, millions of commuters take part in a giant race to determine transportation supremacy. Cars, bikes, subways, buses, ferries, and more all compete against one another, but we never get much explicit feedback as to who “wins.” I’ve previously written about NYC’s public taxi data and Citi Bike share data , and it occurred to me that these datasets can help identify who’s…
The City of Chicago has released a public dataset containing over 100 million taxi rides since 2013. I adapted my analysis of the similar New York City dataset to process the Chicago data, and created a GitHub repository with the relevant code. The Chicago dataset does not include data from ridesharing companies like Uber and Lyft, but the data makes clear that taxi usage in Chicago has declined…
The Simpsons needs no introduction. At 27 seasons and counting, it’s the longest-running scripted series in the history of American primetime television. The show’s longevity, and the fact that it’s animated, provides a vast and relatively unchanging universe of characters to study. It’s easier for an animated show to scale to hundreds of recurring characters; without live-action actors to grow…
Update March 2019: Click here to view a new dashboard that tracks additional monthly taxi and ridehailing metrics from the TLC. This page will still update, but I recommend the new dashboard for more comprehensive up-to-date data. The New York City Taxi & Limousine Commission publishes summary reports that include aggregate statistics about taxi, Uber, and Lyft usage. These are in addition to the…
The NBA’s Stats API provides data for every single shot attempted during an NBA game since 1996, including location coordinates on the court. I built a tool called BallR , using R’s Shiny framework, to explore NBA shot data at the player-level. BallR lets you select a player and season, then creates a customizable chart that shows shot patterns across the court. Additionally, it calculates…
In the conclusion of my post analyzing NYC taxi and Uber trips , I noted that Citi Bike , New York City’s bike share system, also releases public data , totaling 22.2 million rides from July 2013 through November 2015. With the recent news that the Citi Bike system topped 10 million rides in 2015 , making it one of the world’s largest bike shares, it seemed like an opportune time to investigate…
Note: this post was originally written in November 2015, and was expanded with updates in September 2016 and March 2018 . There is also a dashboard available here that updates monthly with the latest taxi, Uber, and Lyft aggregate stats. The New York City Taxi & Limousine Commission has released a staggeringly detailed historical dataset covering over 1.1 billion individual taxi trips in the city…
It’s fairly commonplace these days for news outlets to reference prediction markets as part of the election cycle. We often hear about betting odds on who will win the primary or be the next president, but I haven’t seen many commentators use prediction markets to infer the electability of each candidate. With that in mind, I took the betting odds for the 2016 US presidential election from Betfair…
LearnedLeague bills itself as “the greatest web-based trivia league in all of civilized earth.” Having been fortunate enough to partake in the past 3 seasons, I’m inclined to agree. LearnedLeague players, known as “LLamas”, answer trivia questions drawn from 18 assorted categories, and one of the many neat things about LearnedLeague is that it provides detailed statistics into your performance by…
[M]ortgages were acknowledged to be the most mathematically complex securities in the marketplace. The complexity arose entirely out of the option the homeowner has to prepay his loan; it was poetic that the single financial complexity contributed to the marketplace by the common man was the Gordian knot giving the best brains on Wall Street a run for their money. Ranieri’s instincts that had led…
FiveThirtyEight and Inpredictable teamed up to make a nice interactive visualization that shows every NBA team’s chances of winning at every minute of every game, according to Inpredictable’s basketball win probability calculator . I was curious what the FiveThirtyEight graph would look like if every team didn’t begin every game at 50% win probability, so I took all of the NBA in-game gambling…
NAND gates are functionally complete , which means that you can implement any boolean function using only NAND gates , even one as complicated as Twitter’s corporate strategy statement: Twitter’s corporate strategy statement, via the Wall Street Journal
Latest TCBI data TCBI 1 mo 1 yr TechCrunch has established itself as a leading resource for startup-related news, so I thought it would be fun to analyze every TechCrunch headline to see what we might learn about the startup funding environment over the past few years. Without further ado, I present the TechCrunch Bubble Index , or as I like to call it, the TCBI: What is the TCBI? The TCBI…
I was pleasantly surprised when somebody shared my traveling salesman animation to reddit and the post made it all the way to reddit’s default front page (i.e. the top 25). The gif racked up over 1.3 million pageviews on Imgur, a testament to reddit’s traffic-generating prowess. Before the post made it to the front page, though, it was brought to my attention that it was on the second page of…
In honor of the Ryder Cup, here’s a fun puzzle for the mathematically inclined golfer to consider: how many different paths are possible in an 18 hole round of match play golf? If you’d rather not wade through the math then you can skip ahead to the “practical exploration” section of this post to see some actual match play data, but if you like puzzles then let’s assume the following match play…
I built an interactive Shiny application that uses simulated annealing to solve the famous traveling salesman problem . You can play around with it to create and solve your own tours at the bottom of this post , and the code is available on GitHub . Here’s an animation of the annealing process finding the shortest path through the 48 state capitals of the contiguous United States: How does the…