Ramblings on programming, tech and life.
When you are debugging a production incident, you are forming hypotheses, testing them against evidence, ruling things out, trying not to confuse correlation with causation, and making triage calls under time pressure. The hard part is the reasoning – figuring out why the system is misbehaving, deciding what to stabilize first, knowing when to stop chasing a red herring. But a surprising amount of…
I wanted a browser terminal at terminal.mrkaran.dev that works from laptop, tablet, and phone without special client setup. The stack that works cleanly for this is ttyd + tmux. Architecture # Browser -> Caddy -> ttyd -> nsenter -> su - karan -> tmux(main) Two decisions matter most: ttyd handles terminal-over-websocket behavior well. -m 1 enforces a single active client, which avoids cross-tab…
I recently set up LiteLLM with AWS Bedrock as the LLM provider. The docs cover the happy path, but there are a few gotchas that cost me some debugging time. This post covers what I learned, from basic setup to per-project cost tracking with Application Inference Profiles. Model Format # Bedrock models use the bedrock/ prefix followed by AWS’s model identifiers:…
In my previous post , I introduced Gullak, an expense tracker I built to categorize transactions using LLMs. While the initial prototype used SQLite, I’ve since made a fundamental shift in how Gullak stores data. It now uses the ledger-cli format—a plain-text accounting standard that has been around for over 20 years. The Core Insight # Ledger-cli uses a format that is, at its core, just text…
The industry is currently obsessed with defining standards for how Large Language Models (LLMs) should interact with software. We see a proliferation of SDKs, function calling schemas, and protocols like MCP (Model Context Protocol). They all aim to solve the same problem: bridging the gap between natural language intent and deterministic code execution. But we might be reinventing the wheel. The…
About eight months ago I wrote about Logchef – a log viewer I’d been building to scratch my own itch with log exploration at work. Back then it was basically a nicer way to query ClickHouse without writing raw SQL every time. Today I’m shipping v1.0, and it’s evolved into something I didn’t quite expect. Let me walk through the major features that made it to 1.0 and some of the engineering…
About a month ago, I downloaded my CIBIL report expecting a routine check. Instead, I found loans from lenders I had never interacted with, written-off accounts, overdues from fintechs I had never installed, and even two-wheeler loan enquiries. I don’t even ride a bike. My credit score had collapsed to under 680. I stared at the report trying to understand how this could happen. The Root Cause: A…
Everyone is worried that AI will replace programmers. They’re missing the real revolution: AI is turning everyone into one. I’ve been noticing a new pattern: people with deep domain knowledge but no coding experience are now building their own tools. Armed with AI assistants, they can create custom workflows in a matter of days, bypassing traditional development cycles. Are these solutions…
Introduction # For the past five years, I have maintained a homelab in various configurations. This journey has served as a practical exploration of different technologies, from Raspberry Pi clusters running K3s to a hybrid cloud setup and eventually a cloud-based Nomad setup . Each iteration provided valuable lessons, consistently highlighting the operational benefits of simplicity. This article…
I recently spent some time debugging a WireGuard tunnel that was acting weird. The handshake was successful, pings worked perfectly, but any TCP connection failed with connect: no route to host . Classic misleading error message. The routing was fine. The Setup # Server with a public IP running WireGuard ( wg0 ) with IP 10.100.0.1/24 . Client connects and gets assigned 10.100.0.2/32 . I…
So, for the last 3-4 months, I’ve been busy building Logchef . This tool basically grew straight out of my day job managing logs at Zerodha , where I’ve been managing logs for almost half a decade. I wanted to share a bit about how Logchef came to be. Like many, we journeyed through the complexities of ELK (a management nightmare) and found its OSS fork, OpenSearch, didn’t quite hit the mark for…
I’ve been introduced to Nix by my colleagues at work. Being a Linux user for over a decade and a serial distro hopper, I was curious to learn more about it. I’d seen Nix mentioned before, but the comments about its steep learning curve made me wonder if the effort was worth it. I decided to give it a try by reading this excellent beginner’s guide however got bored very quickly and decided to…
I’ve been playing badminton more regularly since the start of 2025 - almost 4-5 days a week. I recently moved to a new part of the city, which meant I couldn’t play with my old friends anymore. PlayO has been super helpful for finding games with new people. On PlayO, a host creates a game and up to 6 people can join one court for a one-hour badminton doubles session. However, on hectic days I…
My Obsidian vault has gotten quite messy over time. I’ve been dumping notes without proper frontmatter, tags were all over the place, and some notes didn’t even have proper titles! I needed a way to clean this up without spending hours manually organizing everything. I’d been playing around with Claude’s API lately, and thought – hey, why not use an LLM to analyze my notes and add proper…
2024 was indeed an important year for me as it marked several significant milestones. Quite happy with how this year was! Here’s my reflection on this memorable year. Life # Got married to the prettiest and dearest Saumya 💗 Did my first international trip, exploring Europe Bought a fun toy - Maruti Jimny 4x4 Relocated to Bangalore after working from home for 4+ years since Covid Attended several…
Just yesterday, GitHub announced integrating Claude 3.5 Sonnet with Copilot. Interesting times ahead. In my experience, Claude has been remarkably better than the GPT-4 family of models for programming tasks. I’ve tried a bunch of tools like Cursor, Continue.dev but finally settled with Aider for most of my tasks. In this post, I want to write about my workflow of using Aider when working on small…
I recently discovered Outline a collaborative knowledge base. I wanted to self-host it on my server, but the mandatory auth provider requirement was off-putting. My server is on a private encrypted network (Tailscale) that only my approved devices in the tailnet can access, so I don’t really need authentication for my personal single-use apps. I found a few guides using Authelia/Keycloak, but…
A couple of weeks ago, I decided to start logging and tracking my expenses. The goal was not to record every minor purchase but to gain a general insight into where my money was going. In this post, I’ll dive deep into the behind-the-scenes of building Gullak —an expense tracker app with a dash of AI (yes). Why # My wife and I have a simple system for tracking our expenses during trips: we use…
Last month, I did a wonderful trip travelling through the scenic landscapes of Switzerland. My wife and I were in Lucerne and had scheduled a day trip to Mt. Titlis for the next day but were wondering what to do that evening. After strolling along the Chapel Bridge and enjoying an amazing lunch by the waterfront, my wife and I collectively decided to book a Lake Lucerne cruise for the evening. It…
I have an upcoming trip to Europe, which I am quite excited about. I wanted to set up a Tailscale exit node to ensure that critical apps I depend on, such as banking portals continue working from outside the country. Tailscale provides a feature called “Exit nodes”. These nodes can be setup to route all traffic (0.0.0.0/0, ::/0) through them. I deployed a tiny DigitalOcean droplet in BLR…
Earlier this week, I had stumbled upon 1brc , which presents a fun task: loading a huge text file (1 billion lines) in Java as quickly as possible. The One Billion Row Challenge (1BRC) is a fun exploration of how far modern Java can be pushed for aggregating one billion rows from a text file. Utilize all your virtual threads, leverage SIMD, optimize your GC, or employ any other technique to create…
Introduction to SadServers # Recently, I stumbled upon sadservers , a platform described as “Like LeetCode for Linux”. The premise is: you are given access to a full remote Linux server with a pre-configured problem. Your mission is to diagnose and fix the issues in a fixed time window. With the goal of documenting my journey through these challenges and sharing the knowledge gained, I decided to…
This blog post is ignited by the following Twitter exchange : I don’t take the accusation of unsubstantiated argument, especially on a technical topic lightly. I firmly believe in substantiated arguments and hence, here I am, elaborating on my stance. If found mistaken, I am open to corrections and revise my stance. Some Historical Context # In my professional capacity, I have run and managed…
At $dayjob, we use AWS Pinpoint to send out SMS to our customers. We’ve also written a detailed blog post on how we use Clickhouse + vector stack for our logging needs. We additionally wanted to store the delivery logs generated by the Pinpoint service. But like with anything else in AWS, even simpler tasks like these usually tend to piggyback on other counterparts of AWS - in this case, it…
To set the stage, it’s crucial to understand what we mean by “bridge networking”. In a nutshell, it is a type of network connection in Linux that allows virtual interfaces, like the ones used by virtual machines and containers, to share a physical network interface. With Nomad, when a task is allocated, it creates a network namespace with its own network stack. Within this, a virtual ethernet…
You know those budget freaks? People who log and categorise every Rupee they’ve spent over the month? The financially sane people? I am definitely not one and I suck at it. I moved cities a couple of months back and had some big ticket spends off late, mostly financed by credit card. I wanted an easy way to list down where all I’ve spent most money and spot some recurring expenses so I can be…
At work, we use a Vector pipeline for processing and shipping logs to Clickhouse . We also self-host our SMTP servers and recently started using Haraka SMTP . While Haraka is excellent in raw performance and throughput, it needed an external logging plugin for audit and compliance purposes. I wrote haraka-plugin-outbound-logger to log basic metadata like timestamps/subject/SMTP response…
A few months ago, a proposal for adding a structured logging library in Go was introduced by Jonathan Amsterdam . At present, Go has a minimal and bare-bones log package which works all right for basic use cases. However, the current library has a few shortcomings that this proposal aims to solve: Emitting logs with different severity/levels Structured output: Makes parsing of logs harder…
CoreDNS is an extensible DNS server (which is actually a fork of Caddy v1) that can be used to serve DNS records for a domain. It is written in Go and is very easy to extend. It has a plugin system that allows you to write your own plugins to extend its functionality. In this post, I will be writing a plugin for CoreDNS that will allow it to serve DNS records for Nomad services. I recently came…
I’d been mulling around reading a computer science paper and implementing a project based on it. Distributed systems, Networking and Databases are some of the things that fascinate me a lot. However, I had been looking to implement a more approachable project to avoid getting inundated initially. And I happened to chance upon the Bitcask paper through Avinash’s project: CaskDB . After giving a…
Application orchestrators like Nomad , Kubernetes etc., allow you to deploy multiple applications on the same host. Logs are stored on the underlying node wherever the applications are run. However, it’s pretty common to treat these instances as ephemeral if they’re a part of an autoscaling group. Therefore depending on the node’s availability to search these logs is not a practical idea as the…
which is a non-standard/non-POSIX compliant program. I faced many issues getting which to work in a chroot environment (Nomad). So basically, which is a simple shell script program to find out the dependency by searching the $PATH (which is what makes it less deterministic). It’s also somehow symlinked 3 levels deep in Debian: $ ls - laht /usr/bin/which lrwxrwxrwx 1 root root…
On June 28, I received a text from Anand nudging me to submit a CFP for IndiaFOSS . Until then, I’d not planned to attend IndiaFOSS because travelling to participate in a conference didn’t seem enticing. However, the thought of visiting Bangalore (a city where I spent five years before Covid took the joy from our lives) stayed in my mind for the rest of the day. Call it serendipity or just that…
At work, my colleagues and I stumbled upon a hair-pulling networking issue involving a specific problem when connecting to a Kafka cluster. We use the franz-go library in our Golang applications to interact with an external Kafka cluster. These Go apps are hosted inside a Nomad cluster and running with the exec driver. The Issue # Solving issues where a specific condition happens only sometimes is…
Nomad is a general-purpose cluster orchestrator and scheduler. Up until Nomad 1.3 was released, it had no native support for discovering other applications running in the cluster. This is sort of a very elementary requirement when scheduling tasks in a cluster. Nomad relies on Consul to discover other “services” and has first class support for registering and fetching service records which makes…
I’ve often been asked how to expose public and private services running on DigitalOcean droplets/RPis when self hosting apps. Most people don’t have access to a static IP for their RPis. I felt I’d summarize my approach to this in this blog post and hope it’ll be useful for others trying to do the same. Tools I use # Tailscale : I love this and I’ve written about it in the past . I am one of…
ClickHouse is an extremely performant columnar DB used for fast analytical processing. ClickHouse supports data replication using Apache Zookeeper which needs to be deployed externally. While Zookeeper works well after you’ve tuned it properly, it was still an additional maintenance overhead. The good news is that you don’t have to worry about this anymore. ClickHouse 21.12 release annoucement…
Recently, I’d posted on Twitter that my feed has become messy overtime. Maybe I followed some accounts that were of no interest, maybe Twitter’s algos don’t really know what to show to me. Whatever, I wanted a fresh start. And, I’ve done this in past. I’ve removed all the accounts that I follow but they don’t follow me back. This little trick is helpful to start afresh while still not offending…
Today I faced an issue that questioned my sanity. Since I didn’t find many “related” issues on StackOverflow/Google-fu except a lone GitHub thread where a kind stranger hinted at what could be the issue, I am writing about it here in the hopes (although I don’t wish this torture on anyone) that it helps someone! To give some context, I am running a Gitlab CI job that bakes an AMI using Packer…
This week I was occupied with optimising a Golang program I’d written at work. I wanted a way to reproduce the issue under heavy load on my development environment and Load Tests are a good way to do that. The service in question is a RESTful API so it’s relatively easy to use any HTTP load test tools. The endpoint had an input parameter uuid which accepted a valid UUIDv4 as the input. To my…
I’ve been toying around Clickhouse and Vector at my day job and find both of these tools pretty interesting at what they do. A short summary for those unaware of these tools: Vector helps you build a pipeline for collecting, transforming and processing different kinds of observability data (logs and metrics). Clickhouse is a columnar based DB used as a warehousing tool for generating reports and…
Over the past 2-3 years, note-taking apps have become all the rage. Note-taking is an extremely subjective topic and a lot of it depends on the individual’s workflow. There’s no one size fits all and maybe that justifies the ever-expanding landscape of such apps. I’ve tried a few popular ones (Notion, Roam Research) in the recent past but never really quite stuck to any after the initial hype…
It’s been a long time since I’ve written a post on Hydra (my home server). I use Hydra as a testbed to learn new tools, workflows and it just gives me joy to self-host applications while learning something in return. History # A brief history of how Hydra’s setup evolved over time: 2019 : A pretty minimal K3s setup deployed on 2 RPi4 nodes. I couldn’t continue with this setup because: Some of the…
There are some things you expect to just work. Sadly trying to make Gitlab Runner with AWS ECR turned out to be quite a daunting task and the little documentation in this area doesn’t help. There’s even a 4 years old issue and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been. Anyway, since I spent a lot of time figuring out how to make a…
In this post we’ll see how to create a simple job queue in Golang. There are tonnes of libraries and posts out there doing overly complicated stuff, however if your need is pretty minimal or you want to understand the concepts from ground up, this post aims to do just that and nothing more. We’ll be using concepts like WaitGroup , Channels and Contexts to build our own Job Queuing mechanism. It…
Have you ever been stuck in a situation where you’ve spent countless hours digging through lot of options, overthinking their pros/cons and having a tough time to decide? If yes, I’ve been in the same boat myself and the more I think about this, I think it’s causing some serious issues in my productivity. Since I’ve started to self host a lot of services, I’m overthinking and overanalysing…
I’ve been writing on this blog for about 2 years now. This has been the longest I’ve stuck on to the same technology stack for my blog. I’ve previously jumped from a Jekyll based static site to a Medium blog before finally settling for Hugo . I’ve been using Hugo since 2018 but I don’t recall as to why I went ahead with it. Maybe it was increasingly popular at that time and everyone touted Hugo as…
Terraform has a straightforward way of importing existing records (managed outside Terraform) via terraform import command. The usage is documented here and works well if you have a handful of records to import. However when you work with custom Terraform modules and have a whole bunch of records to be imported, you’d look out ways to script the entire workflow. I did this a few weeks back at work…
Twitter is an amazing thing! For all the shitposting and meme-ing that’s done there, there are some really cool people you get to interact with which wouldn’t have been possible IRL. I happened to stumble upon Swapneel last year in Bangalore at a meetup. He occasionally posts about RIPE Atlas Probes measurements on his Twitter feed and that made me curious and learn more about the Atlas Probe.…
I like monitoring stuff . That’s what I do at work and when my home ISP started giving me random problems and I decided it would be nice to monitor my home network as well. There are a couple of ways to go around this, a very popular and OSS solution is SmokePing . SmokePing is written in Perl and is used to visualise network latencies. It’s quite a great solution but for my current stack which…