Open source is a flex

I first started using Databricks in early 2023. Back then I was heading up analytics for Delhivery, and the CTO insisted that all our models and code “be put in production”. This being before the time vibe coding was a thing, and us being mostly a team of analysts rather than software engineers, we struggled.

That was when the data engineering team bailed us out, introducing us to Databricks - the best feature of it then was that you could “put notebooks into production” (some of the more technical people on my team had a problem with that, but we got our way). It was a gamechanger for the rest of my time in Delhivery (I left later that year).

As I started reading up more about Databricks, I was mildly shocked to see that the core technology - Spark - was open source! In fact, it began as an open source project and then Databricks was built around it as a commercialisation initiative.

Later that year I started an AI-for-data company, and soon discovered that having an open source core was almost “standard practice” in the data engineering world. Confluent was built around Kafka. dbt has an open source “dbt core”. Astronomer (that Coldplay concert company!) commercialises Airflow. DuckDB is entirely open source (the company Motherduck builds services around it).

My initial hypothesis around open source was around lock-in - especially when it comes to data engineering and storage, customers are wary of solutions that lock in their data. What if the company goes belly up? What if the pricing is suddenly jacked up? (these are questions we actually got from our customers at Babbage). What if we get a new leader who wants a new technology (a surprisingly common reason why companies switch data platforms)?

With an open source portion core to the product, this lock-in is easily addressed. Purchasing CIOs know that even if one of the above contingencies were to happen, they can continue to use the open source version of the technology and keep their data (and pipelines and everything downstream) safe.

My next hypothesis about why companies go opensource is community contributions - when you open source a part of your product, other users are able to contribute to it, and this can make the product richer. Of course, it comes with its own headaches in terms of reviewing contributions (apparently a much bigger problem now in the times of vibe coding), but you don’t need to respond to every single feature request or think up every single product idea.

When I first started my “data and strategy consulting” practice in 2011-12, I was hoping to do a wide variety of work. Apart from working with corporates, I was also planning to explore the not-for-profit sector, given that it seemed to be data-rich and analysis-poor. And at that early stage of my career, I didn’t even care that much about how much I was going to earn from this - just the insights and potential publicity might have worked then.

However, in about a year (during which I took up and completed one not-for-profit assignment; and agreed on and backed out of another thanks to some differences on access to clients), I swore off this kind of work. I had got disillusioned based on my encounters with what I called “data hoarders”.

I don’t know the situation now, but back then a lot of data that was supposed to be public and open wasn’t actually so. data.gov.in was active but had limited datasets. There were people with the right kind of access who had got control of some datasets, and then wouldn’t let go of them. “Write to me, I’ll send you”, they’d say when you’d talk to them at conferences and events. And then you would have to jump through hoops to get it.

And then there would be people sitting on data who would refuse to share it with you saying “it is not safe to have this data out in the public” (yes, it was very safe to have it with these gatekeepers!). It was apparent that access to data was a key, and controlled, resource in this sector.

I must mention that not everyone I encountered was like this - I encountered a whole bunch of genuinely helpful people and I’m graterful to them (I should give a shout out to this online community called “datameet” where people are rather generous in sharing data etc.).

Of course the situation is very different now. You have initiatives such as the Trivedi Centre and SHRUG that collect and share loads of open data (I’m pretty sure there are many more such; these were just top of mind). The Takshashila Institution, which I’m associated with, also has open github repos where they share data and code.

If you come from the prior that people hoard data and code and access, since that is what gives them power, you come to realise that open source is actually a flex. It is like the peacock’s tail in evolutionary biology. “We are able to share all of this for free and still we are running a viable business, so there is something to what we are doing”, is the message that goes out.

The flex in the peacock’s tail is “I can devote considerable resources to maintaining this beautiful tail, yet I’m able to survive and avoid predators, and so I’m evolutionarily really fit”. The peacock showing off his feathers in his mating dance is sending out exactly this message, and a peacock with better plumage will get better mates.

Open sourcing a part of your software stack is something similar - “we are able to devote resources to produce this piece of software that we make absolutely no money off, so we are a sound business” is the message that goes out. Of course - investors and shareholders might have a different interpretation of this message, so the corollary to this is “we are able to charge customers precisely for our extreme cutting edge. Whatever is beyond what we have shared. And we are able to charge them enough just for the cutting edge that we can open source the rest of it and still be a viable business”.

So if you are a not-for-profit that collects data and disseminates it for free, you are showing that access to your data is not the cutting edge - it is possibly the analyses or discourse that you’ve built on top of it. Basically - you are higher up the value chain than what you have disclosed!

If you are an LLM lab, and you have released your model with open weights, you are saying that the weights are not your secret sauce - there is something else you offer that offers real value to your customers. For example, Thinking Machines (started by former OpenAI CTO Mira Murati) has published an open weights model, with the reasoning that customising / fine tuning these models for enterprises is where the real money is.

And then sometimes you might choose to go open source / weights purely for trust issues. According to me, there are two reasons why Meta decided to open source the Llama suite of models - firstly their weights had been leaked anyway; and more importantly, the models came out at a time when Meta didn’t have the best reputation in terms of data security and privacy (remember Cambridge Analytica?).

In 2023, a proprietary Llama model would not have found too many takers since people would have been wary of giving their data to Meta. Making it open weights helped them get around this hurdle. A similar logic explains why most cutting edge models coming out of China are open weights - people in US, India and other places will be reluctant to put their data into Chinese models hosted in China, and so the only way for these models to get traction is by making them open weights.

The other thing that open source / weights does is to shine a light into the insides of your models (it is similar to how MTR or Taaza Thindi have made their kitchens in a way that people can see them and the food being cooked - showing off their cleanliness and quality). And since people can now inspect your code, they know how good or bad your overall software stack and engineering practices are. Again - assuming you’ve done it well, it is a flex and a selling point (you will never open source your code if you yourself don’t think it’s good enough!).

It was only during my time at Delhivery (what I’ve started this post with) that I recognised the value of putting models in production, and Git discipline. And so, even though I had been largely pro-open source until then, I’d myself not open sourced too much until recently.

Vibe coding has changed all that - I’ve retrospectively open sourced a whole bunch of my analyses in the last 15 years. And now that I’m back to being a consultant, I’m trying to open source a lot of stuff I’m working on outside of my client stuff (where the IP belongs to them).

For example, I’ve open sourced a whole suite of data visualisation based skills here.

I’ve built an iOS app called “Pensieve” where I can log my thoughts. I distribute it through TestFlight to a few users, but the code is again open source (it’s entirely vibe coded, FWIW).

Some of you might know me for my Bangalore weather charts. I’ve open sourced the code behind that as well, so you can extend it to get similar charts for your own city!

Last month, someone on Twitter (who I initially mistook to be a data hoarder) tweeted about how he had “exclusive data on pre-2008 Indian constituencies”, and started releasing them state by state. I had the entire dataset thanks to Datameet, and shared it in a public repo.

As you can see, a lot of this (except maybe the visualisation stuff) is stuff I could never really make money off anyway. So I might as well get some good karma by sharing the code and data! Yes, that is another reason for making things open source!