Recent content on Sushant Dhiman
Subscribe to get posts on backend engineering, distributed systems, Go, databases, and infrastructure. I write about the things I’m building, the problems I encounter, and the lessons I learn along the way—practica ideas, experiments, and engineering insights from building systems in the real world.
Let’s say you are building distributed software that supports replication (such as a database). There will be multiple nodes running in a distributed environment for that same application. In order to support replication, all the data from one node needs to be copied to other nodes in the network as well. So you came up with an idea of a leader/follower-based system. But now there is a…
Hi, I’m Sushant Dhiman. I’m a backend software engineer passionate about backend systems, networking, databases, backend architecture, distributed systems and low-level software. This website is where I document my software engineering journey by sharing what I learn, build, and explore. I believe the best way to learn engineering is by building things. Instead of only reading about…
A lot of engineers use load balancers every day. But most of them never go deep enough to understand how it works. I did, and I implemented a Layer 4 load balancer in C to understand how it works. It is not a production-grade solution but is enough to give a core conceptual idea of modern load balancers. Why I built it? I’m a great fan of a quote by Nobel Prize-winning physicist Richard…
Why Kafka Exists? You are building a backend system with multiple services (auth, payments, notifications, analytics). Each service needs to communicate with other services. Issues come when Service A needs to call Service B, and if Service B is down, Service A also fails. This type of communication increases system complexity and scaling problems. This is why Kafka is built. Kafka introduces a…
Most tutorials explain how to write a .proto file and call a gRPC method. But very few explain what actually happens after that call. I recently implemented gRPC in one of my projects and while learning it, I tried to understand its architecture from first principles — not just how to use it, but why it is designed this way and why it is fast. This post is a simple breakdown of that.
Most backend engineers think that HTTP is a stateless request-response protocol. That’s technically correct but not complete. For the last few days I have been reading a lot about HTTP in depth. I’m going to talk about same in this post. In This Post What is HTTP? What is a protocol? What happens when you browser any website? How is this HTTP structured How the server actually reads…
Recently I was learning how Kafka works internally. So in order to understand it, I built a commit log system from scratch in Golang. Some people might think that Kafka is a message queue and not related to logs. But in reality Kafka is an append-only log system. I’ve discussed this more in this post. In this post we will implement the log system discussed in the previous post from scratch.
Recently I was exploring RabbitMQ and Kafka to do some POCs. I found that Kafka provides very high throughput, which refers to the ability to process a large number of messages in a short amount of time, as compared to RabbitMQ (not RabbitMQ streams). I started exploring why Kafka is so fast. The main reason behind why Kafka is so fast is the way it stores data on disk and some operating system…
I wrote production systems for a startup in Java & Spring Boot for 1.5 years before switching to Golang. That was a high-paced startup, so we were shipping features every week. There I got my hands dirty on some absolute backend technologies that are powering business-orientated backends, such as Java, Spring Boot, MySQL, Redis and Kafka. Recently I made a switch to Golang because I wanted to move…
Hi, I hope you all are doing well. Recently I was exploring RabbitMQ, and I found it fascinating. Previously I’ve used Kafka. RabbitMQ is very different from Kafka. This article is mostly useful for beginners or people who haven’t used RabbitMQ. If you are an experienced developer, you might not find anything new in this post. What is RabbitMQ? RabbitMQ (Rabbit Message Queueing) is an…
Hi, I am on my way to becoming a better engineer. So I am building stuff that people don’t build and learn outside their job. I am currently learning about operating systems and how they work. But instead of following the old textbook reading approach, I am doing this by building some projects along the way. In order to understand processes, I have decided to build a shell from scratch. This…
Recently I was watching a GopherCon talk on YouTube that was published in 2016. The speaker spoke about “nil”. After watching that 30 minutes talk, my views about Nil are completely changed. In this post we will discuss exactly the same things that he talked about. This post is not going to be a very long post, so it will be easy for you to get the key points from that talk. But I will…
If you are following my blog, you might know that my plan for this year is to learn and program low-level systems. I wanted to develop an operating system since I was in college. I never got enough time back then. But now I have plenty of time to work on skills other than my job. I spent last month learning about how a computer is basically made and how a CPU performed all these magical tasks from…
This is the 4th episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way. Recently I was reading how Uber’s backend system handles so many real-time location events, and I found that the system which handles it is quite interesting. So I’m talking about my learnings in this blog post. All the information provided in this post…
This is the 3rd episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way. In this episode we are going to discuss: What are rate limiters.? What problem the solve? How they work? Rate Limiting Algorithms. IMPORTANT - This post is for beginners who want to understand rate limiters. If you are an experienced developer and…
This is the 2nd episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way. Web crawlers are software that only a few people know about, but they play a major role in today’s tech. Without web crawlers, our search engines will not be able to show us search results. In this post we will discuss how web crawlers work by learning…
Email is a technology that you use every single day. It is actually very interesting how emails are handled behind the scenes and reach the recipient’s mailbox. In this post I’ll explain it in a simple way. Basic Terminologies These are some terms that you must know before reading this post. SMTP : Simple Mail Transfer Protocol. A TCP-based communication pattern used to receive and deliver mail.…
Hi, 2025 has come to an end. It was a wonderful year for me, and I hope it was for you too. I worked with different tech stacks, including Golang, Java, Spring Boot, Linux, MySQL, Cassandra, ClickHouse, Kafka and encryption algorithms. But one thing that bothers me is “we work with computers on a high level” and we really don’t know how a computer works, how memory works and all that stuff. That’s…
Hi, I’ve been reading about Apache Cassandra since the last 2 weeks. I spent time reading a lot of online blogs and then ended up reading Cassandra: The Definitive Guide. In this post we will discuss what makes Cassandra so much faster, more scalable and how it can handle massive writes without bottlenecks. Before that we need to understand What Cassandra really is and how is it different from…
This weekend I was diving deep into concurrent programming to learn that, I decided to write a threaded file compressor, which will not only compress files concurrently but also limit the memory usage. I have implemented this in GoLang, but these concepts can be implemented in any language. What I’ve Developed? I designed a tool that reads files in chunks and limits the overall memory used by…
Hey, everyone. It’s another weekend, and I was exploring what to build. So I decided to build a simple yet completely functional load balancer. Let’s discuss it in this post. What is a load balancer? A load balancer is a component in a system that distributes requests evenly between servers. It is used in large-scale systems where a large number of requests need to be handled…
Hi, It’s been a long time since I wrote something in this newsletter. Recently I was learning about language parsing and abstract syntax trees. After getting some knowledge about this, I decided to write a JSON parser from scratch. Parsing: It is the process of analysing the structure of a string (basically any programming language syntax). Parsing helps us to determine the meaning of the…
Recently I’ve been learning about distributed systems and I came across a very interesting concept “Consistent Hashing”. It is one of the most basic thing in Distributed Systems. What is Consistent Hashing? Consistent hashing is a key distribution technique that ensures easy and smooth mapping of keys to servers to minimize data movement when nodes are added or removed. Unlike…
I made a configurable rate limiter. In this post I am going to talk about that. We will discuss its architecture and how it works. Rate Limiter Rate limiting is a technique that controls the number of requests that can be made to an API within a specific interval of time. Although it is not a very new concept, it has been used for a long time. But I haven’t made a traditional rate limiter;…