Recent content in Amar Prakash on Amar Prakash Pandey - ᕦ(ò_óˇ)ᕤ
You’re building something with Claude Code. The agent asks a question, you explain the context, it makes a good decision. You commit. Session ends.
Next session. Different task, same file. The agent is about to do the exact thing you just told it not to do.
You explain again.
This isn’t a bug. It’s just how LLM sessions work — every session starts completely fresh. No…
Why these words sound scarier than they are If you’ve ever Googled “what is a monad”, you’ve probably landed on a Stack Overflow answer that says something like “a monad is just a monoid in the category of endofunctors” and immediately closed the tab.
Fair.
Here’s the thing: you don’t need category theory to use functors and monads. You need…
Why this topic is confusing Currying and partial application are concepts that almost every Python developer has seen, but very few feel confident using. For a long time, I understood what they did, but I didn’t know when using them actually made code better.
A clear distinction: currying vs partial application Currying transforms a function that takes multiple arguments into a chain of…
Why this matters Python gives you multiple ways to solve the same problem. Loops, comprehensions, generators, map, filter, reduce, any, sorted, and more.
For a long time, I knew these tools existed, but I didn’t really know when or why to use them. I could read functional-style code, but writing it confidently was another story.
This document is not a tutorial on functional programming in…
Picking EC2 Spot instances shouldn’t feel like rolling dice. Most teams either guess and get bitten by interruptions, or overbuild complicated logic to dodge them.
Here’s the clean split that keeps you sane:
Spot Optimizer is a Python library/CLI that figures out the right instance types and counts for your workload. EC2 Fleet / Spot Fleet are AWS services that launch and…
Picture this: You’re maintaining an open-source project, and you receive a pull request that promises to revolutionize your codebase. Security improvements, architectural enhancements, performance optimizations – it has everything! Sounds perfect, right?
That’s exactly what happened with my project spot-optimizer. The contributor was skilled, the intentions were great, but there…
As developers, it’s common to wear multiple hats — working on company projects during the day and tinkering with open-source or personal side-projects at night. If you’ve ever accidentally committed to Git using your personal email in a company repo (or worse, pushed your work identity to a public repo 😅), you know how frustrating it can be.
I used to run into this problem often, especially…
When working with large datasets in Spark, joins are a common operation. But what happens when data distribution isn’t uniform? Let’s dive into a real-world scenario to understand why dynamic skew join optimization is not just useful, but often essential.
The Problem Setup Assume we have two large tables and we’re trying to join them using the following Spark SQL:
SELECT * FROM…
Everything seemed right—ample resources, a well-sized cluster, and yet, the Spark job kept failing with an out-of-memory error. Logs pointed to memory allocation failures, but with a 63-node cluster, each equipped with 64GB RAM, this shouldn’t have been an issue. We tweaked configurations, analyzed logs, and even considered scaling up the cluster. But the real solution? It wasn’t what we…
When working with large-scale data processing using Apache Spark, understanding how jobs and stages work is crucial to optimizing performance. This blog is for those who already have some experience with Spark and want to dig deeper into the internal mechanics of jobs and stages.
Spark Transformations and Actions In Spark, operations are classified into two main categories: Transformations and…
When designing databases, there’s a constant balancing act among three main factors:
Read times Update cost Memory/storage overhead The RUM Conjecture suggests that optimizing any two of these factors will negatively impact the third. Essentially, you can only choose two out of the three to prioritize in any design.
Example: Log-Structured Databases Consider a log-structured…
The CAP theorem, also known as Brewer’s theorem (named after computer scientist Eric Brewer), defines a fundamental trade-off in distributed systems: any distributed data store can provide only two out of three guarantees at any time:
C: Consistency A: Availability P: Partition Tolerance What Do These Terms Mean? Consistency (C): Every read receives the most recent write or an error. This…
Apache Spark’s shuffle partitions play a critical role in data processing, especially during operations like joins and aggregations. Properly configuring these partitions is essential for optimizing performance.
Default Shuffle Partition Count By default, Spark sets the shuffle partition count to 200. While this may work for small datasets (less than 20 GB), it is usually inadequate for…
In Apache Spark, efficient data processing often relies on the use of broadcast joins. However, when the dataset exceeds a certain size, specifically 8GB, you may encounter the following error:
Caused by: org.apache.spark.SparkException: Cannot broadcast the table that is larger than 8GB: 13 GB This error arises because Spark is attempting to broadcast a dataset that is larger than the maximum…
1. Rigidity Rigidity is the tendency of the system to be hard to change. Code that has dependencies that snake out in so many directions and you cannot make an isolated change without changing everything around it. Rigidity causes compile time error. 2. Fragility A system is fragile when a small change in one module causes other unrelated modules to misbehave.
It is the tendency of the code to…
Docker is a software framework for building, running, and managing containers on servers and the cloud. Here are the several best practices for using Docker in production to improve security, optimize image size and write cleaner and more maintainable Dockerfiles.
1. Use Official Docker Image as Base Image Always use the official or verified base image when writing the docker file. Let’s…
What is GitOps? Treat the infrastructure as code the same way as application code.
Separate repository for Infrastructure as code. DevOps pipeline. How does GitOps works? Infrastructure as Code hosted on Git repository.
Version controlled. Team collaboration. Use branching strategy to merge code in git repository. With CI pipeline to test the code. With CD pipeline to apply the changes to…
TL;DR. Code is here.
Finger detection is an important feature of many computer vision applications. In this application, A histogram based approach is used to separate out the hand from the background frame. Thresholding and Filtering techniques are used for background cancellation to obtain optimum results.
One of the challenges that I faced in detecting fingers is differentiating a hand…
We will talk about Google Summer of Code but before that let’s talk about what Open Source Development is. Yes, it’s very important.
What is open source development? Open-source software development is the process by which open-source software, or similar software whose source code is publicly available, is developed. These are software products available with its source code under an…
Hello, I’m Amar Prakash Pandey :)
I’m a developer and hacker with a knack for solving problems in unconventional ways—finding simpler solutions where conventional methods fall short. Curiosity drives me, and I’m always experimenting with new ideas, whether it’s building efficient software, designing data platforms, or creating tools to make developers’ lives easier. You’ll often find me…
Here are some of the projects that I have worked on. You can find the source code and demo link for each project below.