Maria's Repair Café

I want to tell a story that led to the idea I want to talk about later in this article. A customer complained that his web shop was slow during times of increased traffic. I fired up a database shell and had a look at the processes the database server was executing. One of them was really expensive. I already had an idea that it was a missing index, but I wanted to check how well Claude.ai did with it. It did not only confirm my suspicion, but directly suggested the correct statement to add the missing index. Since I could not bear our customer suffering, I did an emergency measure and immediately added the missing index. The customer was happy, he told me that the shop was very responsive now. The project manager lauded the quick resolution. However, I had to disagree about “quick”. I had noted the same query half a year ago and put it in a ticket with the comment “this has to be optimized”. I also told the project manager that the issue had been there from the start of the project several years ago. Nobody really bothered about bad performance. So it only got more pressing when more and more people where using the shop.

I remembered a quote of a former colleague. He used to utter in such situations that there is nothing so urgent that it can’t become more urgent by procrastinating it. How right he was! The project manager suggested that I should be the one to solve performance issues from now on. As much as I like solving performance issues, I told him that I did not believe that this was a good idea. I told him the following quote: “Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime.” He understood what I wanted to convey and replied that I now had the task to teach my colleagues.

How should you go about spreading knowledge in query optimization? In times gone by, every project had a database administrator. He took care about everything related to schema design and optimization. You only had to beg him to add a new table or new columns and after a few days you would get your schema change and and could go on with the task where this extension was needed. Life was much more calm back then! Nowadays, we write migrations to evolve our database schemas, but often, performance stays behind. And we lack the skill to do anything about it. A sure sign of alarm is, when a project tries to cure performance issues by throwing hardware at them. If your database size is in the range of maybe 1 GB, you just don’t need 256 GB. That is, when your schema is well tended. If you are lacking indexes to perform joins, you will end up multiplying the involved tables and that needs a lot of RAM and processing power! Only you can’t add it fast enough since your hunger for RAM increases by the size of the involved tables multiplied. At some point, you just can’t add more RAM and you will go for distribution techniques. But your problem won’t go away but only grow worse!

Similarly, solving performance issues won’t scale when put on the shoulders of one “expert” (who even wouldn’t define himself as such). Even worse: What if your expert is run over by a bus? Also, I wasn’t optimistic about teaching. Especially not in a remote situation where you could not see if the students where following. So what to do? It had to be something fresh and modern to keep them interested and awake. That’s how the idea of Maria’s repair café was born. It is an open space where Maria and Claudius (I suspect this guy to be an AI) help you with your slow queries, even identifying them in the first place. Maria, of course, is the daughter of Monty Widenius who created MariaDB and named it after her.

The format would be hands-on, the result would be unknown upfront and therefore suspense would be high and keep everybody awake. No theoretical discussions, but real action. Everyone would contribute, Maria would supply query execution times and query plans and Claudius would help analyzing the situation. Everybody could participate in the discussion. And, hopefully, at the end of such a session, everybody would have learned something and some performance issues would have been mitigated.

What do you think about such a format? Are you keen on reading about how it works out?

Read more about how a generative or learning culture relates to high performance organizations in my book “DevOps Mindset in Software Development; From Apprentice to Journeyman”.