Recent content on Jens Rantil
These past years, I have had a lot of people reaching out and wanting to talk “AI” ( or LLMs ) with me. Either the AI conversations have been around AI strategy for themselves, or for their company. Sometimes they have worked for a company claiming to be “AI first”. Assuming that the conversation is about customer impact (which “AI usage” is not 1 ), my first…
I was once assigned as the onboarding buddy for my then-new colleague Jeff . I had been at the company for five years and was almost a founding engineer. I knew the ins and outs of the company. Not only did I mostly know the way we worked, our company culture, our technical stack, and our design choices, but, more importantly, I had the history and context of how things came to be.
Code freezes commonly stirs up strong emotions within the engineering community. Either people find them useful, or hate them. Some companies have a code freeze on Fridays, others do it over the December holidays, or around important dates when systems must be extra stable, such as Black Friday. Code freeze? Link to heading In short, a code freeze is when changes to a software system are suspended…
The term AI is everywhere these days. Suddenly, it feels like every single company frontpage on the Internet says it is either “an AI company” or “using AI to do X”. It is as if the whole world has discovered that computers can do smart things. But let me tell you what feels like a little secret these days: AI has been around since the 1950s . In 1950, Alan Turing published…
I recently read the blog post “In Praise of –dry-run” . I love --dry-run mode! It can be immensely useful to avoid Bad Things™️ from happening. The blog post reminded me of a small terminal/shell/Bash trick that I have used for years, but never shared with the Internet: An example Link to heading Imagine you want to download many profile pictures from the Internet. All…
If a tree falls in a forest and no one is around to hear it, does it make a sound? In my article about incidents , I wrote: “Raise an incident when you think a postmortem would be useful for what you are seeing.” This also includes raising an incident for a near miss. A near miss refers to an event that could have caused an incident, outage, security breach, or service degradation —…
An incident framework usually has the concept of an incident priority : priority 1 (“Systems are DOWN!”), priority 2 (“I can log in, but there is a bug.”), etc. Incident priorities are mostly used for two things: Operational: To help operators and on-callers quickly guide them on how serious an ongoing incident is and help them prioritize their work. A low-priority…
Incident process? Link to heading Many companies have an incident process. An incident process (often called an incident management process) is a defined set of steps an organization follows to detect, respond to, manage, resolve, and learn from incidents that disrupt normal operations or pose a risk. flowchart LR A[Incident detected] --> B B[Incident declared] --> C C[Mitigation in place] --> D…
Kagi Translate supports translating from English to LinkedIn Speak. I rarely rant on this blog. But for a change, now is the time! In Swedish, there is the expression “Axelklapparklubben” which literally translates to “The Back-patting club”. It’s a mutual admiration society, an expression that is used to describe a setting where every participant is only telling each…
I have never been a big fan of how some people equate “contract testing” with using something like PACT . There are other ways of doing contract testing, but a lot of engineers don’t seem to know this. Today, I thought I would write something about this. Contract testing? Link to heading Contract testing is about making sure that two pieces of software (usually two services) can…
Charade. A pretense or false show — something that’s meant to look real or sincere but is actually fake or deceptive. Like a theater. Sometimes, I have felt like work, the professional domain, is like a real-life role-playing game. Stepping into an office has occasionally been like stepping into an improv theater where rules apply that don’t otherwise: Feelings should be left outside the…
I recently stumbled across the post “Does Visual Studio Rot The Mind?” and I was reminded of a blog post that I read a very long time ago and could not find on the Internet. I thought I would write down the gist of that article here such that I can share links to it later. There is this theory that the best and most beautiful code was written before the nineties, before the…
This post will be the first in a series of blog posts where I will be talking about the limitations of Apache Kafka as a task queue and how we overcame these limitations at a previous employer where I managed tens of fairly high-throughput Apache Kafka clusters. This post will lay the groundwork for explaining how Apache Kafka works, such that the rest of the posts are easy to follow. Hopefully,…
I have used the same one-on-one template with my last six managers, including other regular check-ins. I am very happy with it! A recent conversation with a friend made me realize I better share the template with the world. If nothing else, I should share it for my own sake - in case I need it again in the future. The template has given me a lot of structure to my one-on-ones which I previously…
Amazon’s founder Jeff Besos is famous for his “disagree and commit” management principle used in leadership. According to Wikipedia , it means “that individuals are allowed to disagree while a decision is being made, but that once a decision has been made, everybody must commit to implementing the decision”. I have a slight variation on this which I call “Raise…
Many people believe pair programming has to be a session where coding is the only action. I don’t think so. Pair programming is a session that involves things like knowledge transfer, discussions, debugging, and coding. I see similarities between this flawed mindset and people thinking that typing faster on a keyboard makes you a more efficient programmer. Creating software is not just…
Throughout my career I have heard soooo many people complain about the general concept of process - the standardized series of steps needed to perform a task. Saying something like “I think we might be helped by coming up with a process for this” has given me the looks as if I brought a skunk to a garden party. Nobody wants a process. Processes exist to structure, manage, and optimize…
In safety systems engineering (SSE) people talk about “safety type 1” and “safety type 2”. The first one is about prevention & reliability , the second is about resilience . Type 1 has been around for a long time, and type 2 is fairly new . Prevention will not save us Link to heading Software systems (and software companies) are complex. And for complex systems, type 1…
I recently read the article “Why I don’t like discussing action items during incident reviews” by Lorin Hochstein. The article lists a few reasons why action items should not be discussed during incident reviews 1 . While I personally think an incident review is a great place to think about possible action items, I am missing one major reason to not decide on action items in that…
Earlier this year, I was on a complex multi-team project that was handed to us by our product organization. The project had a lot of dependencies between teams and could have been a disaster. Today I would like to share an approach we took to structure the project into what eventually made it a success. The project was to deliver a new product feature, and the deadline was three months in the…
In software engineering jargon, people often speak of “performance” without defining specifically what they mean. Improving performance usually involves “increasing throughput”, or “reducing latencies”. Throughput is measured in “units over time”, for example, “bytes per second”, or “cars passing by every day”. Latency ,…
In the article “Natural solutions” , I wrote […] product engineering is equally much about defining the problems we are trying to solve. and I think that statement is somewhat more controversial than first meets the eye. The quote says something about the role of a product engineer and implies things like: Engineers are not just implementors of well-defined tickets and…
I recently wrote about the solution vs. problem space . For years I have had this belief that given a problem that is well-defined enough, a solution will naturally unfold . In other words, solutions are the natural fallout of well-defined problems . But what is “a well-defined problem” anyway? It means having the answer to the following: Problem you are trying to solve. Scope of the…
Today I wanted to write something about the recent CrowdStrike Falcon incident and about one contributing factor, namely centralised risk management . The importance of local risk assessment Link to heading I recently watched the trailer for the movie “Safety Differently” by Sidney Dekker. One of the things it talks about is that the farther you are from a system, the less effective…
In the article Scaling the Practice of Architecture, Conversationally by Andrew Harmel-Law he talks about the concept of Architectural Decision Records (ADRs) as a way to align and document architectural decisions. In short, an ADR is a record stored in a central log where architectural decisions are documented. Usually, the decision log is checked into version control somewhere. A record includes…
As engineers, we are great at solving problems and working in the “solution space”. But, unfortunately, many engineers are quite bad at defining the problem(s) we are trying to solve in the “problem space”. Let me explain! Working in the solution space involves things such as designing, coding, drawing, improving performance, or refactoring. It’s usually what many…
There is a belief that “software architecture” only involves designing the “big picture” of how software is built. For example, drawing boxes and arrows how microservices integrate, or how larger modules of a code base should be organized and coupled together. I beg to differ. I think of architecture simply as “how to organize and associate technical components”…
Innovation is the skill of going from idea to action. If innovation is low, ideas risk getting lost or discarded and you risk missing out on great opportunities. As a tech organization, how do you make sure that you are innovating? In my opinion, innovation doesn’t “just happen”. It’s part of Creative Management . While the dream scenario is that any engineers can propose…
This is a follow-up on How to Ask Good Questions . Just like asking questions is an art form, answering them requires some skill. Here are some general recommendations on things to think about when answering a question: Keep a friendly tone Link to heading A person asking a question is implicitly in a vulnerable position; They expose themselves to not knowing something. I think that makes it extra…
Info This article is about staging environments, but it almost equally applies to the concept of single-tenant environments. The ultimate quality assurance is when customers get their hands on your software and actually try it. That’s when you know if your software does what it’s supposed to do or not. This happens in production. Every technical decision has a tradeoff, but certain…
I think our software industry sometimes is blindsided when it comes to security. Don’t get me wrong, I am of course in favor of good security practices. But I think we could get better at understanding the trade-offs when improving security. A story Link to heading At a previous employer, we partnered with a big bank. They were using our SaaS’s REST API over the Internet. For security…
The Agile Manifesto states Responding to change over following a plan This implies that we must design our systems to be malleable to change and allow for evolutionary design . Becoming good at evolutionary systems design requires practice, and sadly, programming education doesn’t teach this enough! Because of this, I ran a workshop at $previousEmployer to teach how you can arrive at a…
Background Story Link to heading Imagine you’re mentoring an engineer who’s eager to brush up on their coding skills after a hiatus from programming. This was precisely the scenario I found myself in recently. A mentee of mine had been away from coding for quite some time and was keen to get back into the swing of things, particularly focusing on data structures and algorithms.
I was on my first podcast discussing the topic of “SRE & Full Ownership” ! We recorded it on the 8th of December, it was published & announced on the 25th of January, and I finally took the time to listen to it fully - and vetted it worthy of sharing. 😅 I thought I would write a little bit about the experience. Info Links to the podcast: Soundcloud Spotify Apple Podcast How it…
Info This post is part of my blog series about My Simplicity Toolkit . I highly suggest you read the introductory post before reading this one. In my initial post on this blog post series I wrote about the verb “to complect” . It means to braid something together. In this article, I am giving examples of things related to testing that tend to be unnecessarily complected together.…
I recently read the excellent article Data is not a Strategy . It reminded me that I’ve been wanting to share some of my own learnings and thoughts when it comes to data-driven teams: I was part of a big push to introduce service levels at my previous job. The objective of the project was to make our company data-driven when it came to operational excellence. A thing I kept seeing was that…
I have been very close to burnout two times in my career. At the same employer. Both times, the following held: Tight deadline. I had an urgent project on my shoulders with a very very tight deadline. In one of the cases, I also had a planned vacation starting within a week - and I feared having to cancel the entire sailing trip with my friends (it was my boat). Low capacity. I had known the…
I recently read the article “What I learned getting acquired by Google” by Shreyans Bhansali. Shreyans wrote On the other hand there was the discovery that most Search improvements are manually reviewed by engineers through ‘side-by-side’ comparisons between old and new results…on spreadsheets! The above quote reminded of how hard, and often understated, quality assurance (QA)…
Info This is the second part of two talking about my simplicity toolkit when programming. If you have not read the previous post I highly encourage you to do so first. This post is part of my blog series about My Simplicity Toolkit . At the beginning of my previous post about simplicity within programming, I talked about the numerous benefits that bringing simplicity to programming can be: Easy to…
Info This post is part of my blog series about My Simplicity Toolkit . I suggest you read the initial post before reading this post. In the previous post I talked about the difference between simple and easy. From the perspective of programming and writing code, simple code/functions/methods/objects are not trying to do multiple things ( “complect” ) unless needed. Some of the benefits…
Info Blog series: My simplicity toolkit: Intro (this page) My simplicity toolkit: Programming (part 1) My simplicity toolkit: Programming (part 2) My simplicity toolkit: Testing My simplicity toolkit: Architecture My simplicity toolkit: Patterns “Perfection is Achieved Not When There Is Nothing More to Add, But When There Is Nothing Left to Take Away” Antoine de Saint-Exupery When it comes to…
Back in the day when The Cloud (AWS) was The New Hot Thing™️, I thought autoscaling would solve most of our problems related to costs, availability, performance, and scalability. Over the years, I have learned that autoscaling has a high maintenance cost, adds complexity, and doesn’t solve all those problems I initially thought it would . In this article I will give examples of…
Enterprise hostage situation: First @disqus roles out ads on all free tiers. A week later they e-mail all customers and offer a deal to remove the ads. — Jens Rantil (@JensRantil) October 12, 2023 I just disabled blog post commenting on this blog after Disqus , my former blog post comment system, rolled out a pretty horrible (or shall I say…disqusting?) change. On the 26th of…
As engineers, we all occasionally need to reach out and ask for help. Asking questions is an art! By asking a question in a good way, you will receive the answer you are looking for, faster. A well-phrased question is also respectful of the intended audience’s time - not wasting their time to have to ask clarifying questions etc. As someone who has been answering questions and mentoring…
Shower thought: Why do we talk about MTTR (Mean Time To Resolution), when it’s a latency? Wouldn’t it make more sense to talk about P99TTR (99th percentile Time To Resolution) given that latencies usually have a long tail? 🤔 #MTTR #SRE #DevOps — Jens Rantil (@JensRantil) March 10, 2021 Today I would like to talk about why Mean Time To Recovery (MTTR) is a wrong metric to look at. For the…
Over the years, I have heard many people say something like Over 50% of people think they are better than the average car driver. They say this as if it’s funny or sounds smart. But the fact is that actually could be true, and the reason is because mean and median are not the same thing . When people take a basic course in statistics, they are likely to come out of it thinking that the world…
I tend to love a “quick feedback”. Usually, this applies to software development: Unit tests should execute fast, CI/CD should fail fast, and minimum bureaucracy between coding and deploying, quick deployments, automated alerts when something fails, etc. Usually what many people think about when talking about “quick feedback” are the above engineering practices. But the…
Today I would like to talk about a missed opportunity when giving and receiving feedback – mainly the discussion that happens afterwards. A classical way to give feedback is using the observed-perceived method: Describe the observed behavior from your point of view. Do it as soon as possible about the specific behavior that occurs. Describe the perceived impact of the observed behavior.…
Throughout my career I’ve been drilled to “have a customer-first mindset”. I have written user stories, done user testing, measured customer impact, and had managers challenging me on what the “customer value” is of what I am working on. This has had a very big impact on me and how I approach all technical work; I have often asked questions such as “How will…
Today I would like to talk about caching strategies for aggregate queries over time-based data which is updated often. This is something I spent significant brain-cycles on in my previous job and I would love to share some of my findings. Example data & use case Link to heading For the sake of the rest of this post, let’s say we have a relational database table containing financial…