Recent content on Dhaval Shah
Background This is the fifth post in a series on AI-augmented software engineering across the disciplines that matter most for production grade enterprise systems. The earlier posts covered ground that's probably more familiar with software engineering fraternity: Three Fintech Architecture Post-Mortems The GC Summary Report Wasn't Wrong Same JSON Storage Problem, Different Database A Black Friday…
Background An older post on this blog covered the technical details of this incident - the GC types, the flags, the before-and-after metrics. This post covers something different: the process of how the investigation ran, where it lost time, and which decisions (with more structured approach) would have changed. The incident On Black Friday, an online checkout platform running at over 1000 TPS.…
Background My earlier article on AI augmented Fintech post-mortem review walked through an Oracle JSON storage problem: a payment transaction table stored as JSON, a query that needed to filter on a nested field, and the chain of fixes - that the team eventually needed. Out of curiosity - a reasonable question came out of that post: does Postgres have the same problem? The honest answer is partly…
Background In Part 1, we established that access patterns & not entities should drive your design. In Part 2, we fixed the database by replacing brittle three-table JOINs with a covering index, reducing P99 on the listing query. The schema was right. The index was right. The listing API itself was fast. And then it quietly started taking the database down - not because of a bad query, but because…
Background My old article compared Virtual Thread based implementations of Spring Core Reactor and JDK 21, building on the comparative analysis before it. Both articles measured the usual things - total processing time, memory footprint, GC pauses, CPU time - and both relied on the summary report generated by GCEasy. The conclusion from that data was straightforward: JDK based implementation…
Background In the first part we established the foundational principle that separates systems which survive scale from systems that collapse under it - design for access patterns, not for entities. In this second part we double click on the database design of the User Profile system and try to understand its impact from performance engineering and scalability standpoint. Specifically - why the…
Background Architecture decisions rarely break at design time. They break a year later, under double the load, when an unwritten assumption proves false. I’ve seen this play out in synchronous coupling, storage models, and service splits - while architecting and designing FinTech platforms. That’s why I now run every major design decision through an AI-augmented review process. What follows are…
Background This multi part series walks through the real architectural decisions and real mistakes behind designing a globally distributed KYC User Profile system - a system that should be capable of serving billions of users at 10M+ RPD with sub-500 ms P99 latency. It's mainly about weighing trade-offs and making architectural decisions. Not theory. Not a tutorial. No "here's how Redis works." If…
Background In the previous article, we explored resiliency and fault tolerance as critical enablers of high‑volume inter‑cloud data transfer. Yet even the most resilient system cannot succeed without deep visibility into its live operations. This final part of the series focuses on observability Observability - A First-Class Requirement RICS builds observability into architecture with two pillars:…
Background Parts 1 and 2 established the what and the how fast — a stateless, chunk-driven pipeline with deterministic block IDs, reactive backpressure, and an adaptive concurrency controller functions under real cloud conditions. But speed and elegant architecture count for nothing if application crash at chunk 73 of 120, or if a transient Azure 503 silently corrupts a partially assembled blob.…
Background In Part 1 we established RICS's foundational architecture — a stateless, chunk-driven pipeline with a hard separation between the data plane and the metadata-driven control plane. That foundation deliberately deferred one critical question: Once the architecture is right, how do you make it fast? Part 2 answers that. We dissect the two mechanisms that directly govern throughput — S3…
Background Imagine you need to move terabytes or petabytes of large files from AWS S3 to Azure Blob Storage, and that too with Reliability & High Throughput The naïve approach (download the whole file, upload whole file) collapses under real‑world realities: Network glitches Process crashes Cloud throttling Feasibility of buffering a 1 TB of file in memory What looks like a simple file-copy…
I am a Principal Software Architect with more than 2 decades of experience designing and scaling distributed platforms. Specialized in high-throughput, low-latency backend systems built using Java, reactive microservices, and event-driven architectures. Proven track record architecting resilient platforms processing 1000+ TPS with sub-100ms latency, designing distributed pipelines handling 200K+…
Let's work together and fix whats' slowing you down Independent consulting architect — 20+ years scaling fintech, SaaS, and payment platforms. If your platform has - High‑stakes architecture choices needing an unbiased expert review Performance bottlenecks directly affecting customer experience and revenue Scalability limits blocking growth and investor confidence Cloud costs outpacing platform…
Introduction If you’re here, chances are, something in your product or platform or MVP isn’t moving the business forward the way you expected - New features take longer to ship because the tech stack is hard to manage Systems slow down or break under traffic spikes Cloud spend rises faster than customer growth, with no clear driver Outages or poor performance erode customer trust and revenue…
Background As software engineers we spend so much of our time building these beautiful, complex data pipelines. We design them to move mountains of information... but let's be honest, we often cross our fingers and hope they don't break. And when they do? A pipeline that fails without making a peep - especially in a system like Azure Data Factory (ADF) - is a genuine nightmare. Working with Azure…
Background My previous article focussed on comparing solutions for performing parallel execution using Spring Core Reactor and JDK 21. This article will follow my previous article where I will provide comparative analysis of Virtual Threads based execution for Spring Core Reactor and JDK 21 based implementation. Keeping the same use case that we referred to for comparing Spring Core Reactor & JDK…
Background If you've ever had to implement business logic that involves concurrent execution along with heavy IO operations, you'll know it can be challenging! Recently, I had a chance to tackle such a use case and made some interesting discoveries about parallel execution in the Java ecosystem. I'll keep this concise and focus on three main areas: Understanding What part of requirements High…
Background For someone like me who's been an ardent fan of Rafael Nadal since 2008, hearing the news of his upcoming retirement absolutely shatters my heart. His departure from the sport feels like losing a constant source of inspiration and joy. The courts will never be the same without his relentless pursuit of chasing down every ball, as if his life depended on that point. This is a truly…
Background In today’s world of distributed computing, network is the backbone of high-performance, low-latency enterprise applications. Understanding and optimizing the Linux network stack is crucial for achieving peak system performance. Here’s a breakdown of the key components: Network Components: Hardware: Includes routers, NICs, switches, and more. Software: Encompasses the OS kernel, device…
Background Lot of my pet projects have been built using Spring Boot 2.x version. Same might be applicable for all the enterprises and organizations who have been building Microservices based applications for their products / services using Spring Boot. During last year, Spring community made a major version upgrade and thereafter released Spring Boot 3.2 version with bunch of new features. First…
Background As I concluded my year-end break in 2022, I engaged in a profound professional introspection. The realization that struck me most profoundly was the depth of my understanding of the intricate software systems I've been architecting, maintaining, and optimizing for numerous years. The stark truth was that I had merely scratched the surface of comprehending the infrastructure of the…
Background Recently I stumbled upon a tweet on my twitter timeline, that discussed about approaches / strategies that can be adopted to improve performance of enterprise applications. What struck me while going through its responses and various similar posts on Linkedin is - tons of material preaching and teaching of how to improve performance of a system is uni-dimensional. While I completely…
Background In this current era, Distributed Architecture has become de-facto architectural paradigm, which necessitates implementation of loosely coupled Microservices which would talk with each other via REST API Message Oriented Middleware As far as Message Oriented Middleware is concerned, Apache Kafka has become quite ubiquitous in today's world of Distributed Systems. Apache Kafka is a…
Background In today's contemporary world of enterprise software where massively used applications are expected to scale and run seamlessly at extreme high loads e.g. Scaling Hotstar for 25.3 million users, system performance becomes one of the key tenant of architecting high throughput, low latency applications along with capability of ease in scaling as per business / end consumer needs . System…
Background In couple of my previous articles I not only tried helping understand fundamentals of JVM memory management, but also delved deeper to understand how Garbage Collection works along with its optimization. So with this background, I am sure you would be convinced that behavior of garbage collection may have ramification on performance of an application. With newer versions of Java,…
Background In one of my previous post we saw tangible advantages (w.r.t throughput, latency and resource utilization) of refactoring existing Microservice application from imperative to reactive constructs. So an obvious question that comes to an inquisitive mind - Can we apply Reactive principles to the underlying communication layer Answer to above question is - Yes and RSocket is the way to go…
Background In my previous post I tried demonstrating how to implement an optimal and performant REST client using RestTemplate In this article I will be demonstrating similar stuff but by using WebClient. But before we get started, lets try rationalizing Why yet another REST client i.e. WebClient IMO there are 2 compelling reasons - Maintenance mode of RestTemplate NOTE: As of 5.0 this class is in…
Background In this contemporary world of enterprise application development, Microservice Architecture has become defacto paradigm. With this new paradigm, an application is going to have myriad set of independent and autonomous (micro)services which will be calling each other. One of the fundamental characteristics of Microservice Architecture is Services must be easily consumable Hence most of…
Background As software industry is embracing the new Microservice Architecture paradigm, myriad applications have been built with Spring Boot framework. By the time organizations have got its early versions of microservice applications in production, industry has found out newer and better avenues for further optimizing microservices, so that systems can be more robust, resilient and responsive…
Now that twenty-twenties i.e. 2020 has just commenced, I just thought to reflect upon the bygone year 2019 as a casual and occasional blogger. This in my opinion will definitely help me to introspect, analyse and thereby identify hits and misses, which eventually will lead me to improve upon my shortcomings / limitations - yes I am trying to be self critical here as that’s one of the way to…
In one of my previous post I elaborated basics of JVM memory and also explained how garbage collection works. In this post we will try to understand various types of GC and how to tune and optimize them with a real world example. GC Taxonomy 1. Minor GC When garbage is collected from Young space it is called Minor GC. Characteristics Triggered when JVM is unable to allocate space for a new object…
Background From first part of blog we gathered understanding about basics of Chaos Engineering. Now we will further deep dive to understand how to perform Chaos Engineering with a working example - which to me is going to be quite interesting. First lets start with understanding basics of working example which will be used to demonstrate following- How to perform chaos engineering within an…
Human body is vulnerable to lot of diseases. So in order to protect human beings from diseases vaccines have been invented. Vaccines mainly work due to process called 'Hormesis', by which system or organism adapts to harm in order to become stronger. Just as our body is susceptible to diseases and germs, so do our systems in software world. Hence vaccines and vaccination can be considered as an…
Personally speaking, I am of the belief that only way to excel (professionally) is to collaborate and learn from experts. Since I am constantly striving to improve professionally as Software Craftsman, I try my best to exercise available options. One such option is attending some good Technology Conferences - as they are the platforms from where the so called experts of industry share their…
Spring Boot helps developers to implement enterprise grade applications which can be pushed to production in no time. Once application gets into production and if we strongly believe in vedic philosophy of Karma :), we are bound to experience Murphy's Law. Whatever can go wrong, will go wrong Considering nature of Distributed Architecture, Observability and Monitoring of application and…
Software engineers have been ardently following Test Driven Development (TDD) as an XP practice for having necessary safety nets. I have even tried covering different schools of TDD with an example in one of my previous posts. Considering recent surge in using Spring Boot for developing Microservice applications, I felt a need to understand and learn how to do TDD whilst implementing Spring Boot…
Everyone of us as Software Engineers would have experienced memory leaks, OOM errors in our Java/JVM applications? In order to dissect such issues it is extremely important to understand the whats' and hows' of JVM memory and its management. JVM - Memory Management One of the many strengths of the JVM is that it performs automatic memory management. As we all know memory management is the process…
Background As per the current trends, Microservice Architecture has become a common paradigm using which enterprise applications are built. With this paradigm shift, an application is going to have myriad set of independent and autonomous (micro)services. So how does a developer do testing within Microservice Architecture? Answer is very obvious - Create integration tests that invokes microservice…
Considering the extensive usage of Spring Boot for building Cloud Native Architecture, I embarked on the journey of utilizing it in my reference Cloud Native application. When I ran my first application I was literally flabbergasted with the magic Spring Boot does under the hood, using which it camouflages the complexity and challenges of building enterprise applications. In order to understand…
Whenever the buzz word 'Cloud Native Architecture' gets to our ears, first thing that comes to our mind is - It sounds too jazzy and latest and would have something to do with technology since it has a word called 'Architecture' within it :). Because of this it creates a perception in the minds of lot of people that 'Cloud Native Architecture' is just about TECHNOLOGY. Unfortunately, this is one…
This is the concluding blog of 2 part series on 'Anatomy of Test Driven Development'. Continuing from where we left in first part of this series, this will mainly talk about 'How' part of TDD along with its avatars. How to do TDD We start with a Test. To get this to work we need to frame the test. We need to make sure that there is enough code in place to make it compile or to make it to work and…
Since we being one of the most intellectual and so called logical species on this earth, we need proper rationale behind each and every action that we do in our personal and professional life. Hence I thought to pen down a two post series highlighting rationale behind following one off the most important and underrated XP practice of software devlopment i.e. Test Driven Development What is TDD…
Recently I was required to identify a memory leak in one of the enterprise application running in production. Fortunately we were able to haveheap dumpsfrom the production environment. After analyzing few heap dumps I was able to trace a uniform pattern; which I felt might be one of the potential root causes for the memory leak. Within all the heap dumps, the same object was holding almost 25 % -…