Recent content on
Every once in a while you find yourself caught between two warring bureaucratic processes. This was me back in 2024, while I was trying to finalize my immigration paperwork. I was missing a key document that only Cook County (the Chicago district I lived in at the time) could provide. The only problem was that their standard document format didn’t satisfy USCIS’s specific requirements,…
I recently saw a video from the Thinking Machines folk where the speaker was being corrected in real time as she spoke. “It’s pronounced ah-sai-yee” it quipped as she struggled with the word for the purported superfood berry she had for breakfast. In a way, it was cool to see an AI actively listen and steer a human to saying the “right thing”. But it got me thinking; is being right the point of…
I started freelancing this year. This has meant jumping between unfamiliar codebases and getting productive as quickly as possible. A year ago this process entailed scouring through READMEs and directly pairing with a more veteran engineer to further familiarize myself with the codebase. Today, the process couldn’t be more different. I prompt an agent to walk me through general architecture, set…
In a past job, I broke networking within my first week of starting. The change? A single line commit that added a customer domain to a reverse proxy control loop that directed traffic to a specific server. api.somecompany.com, api.anotherfancynewcompany.ai { reverse_proxy http://localhost:4200 } At a glance, this code is legible, it should work; we’re manually proxying traffic from two domains to…
The other day, I was working on translating data from a file into json to visualize the results. Easy enough, given that this was an operation I’d done countless times in the past. Still, I blanked. After spending months delegating all coding related tasks to an agent, I’d completely lost the muscle memory of a simple fs.readFileSync call. I ended up sheepishly reaching for Claude, which wrote the…
Who am I, if not what I do? This was the question that haunted me at the start of the year. I began 2025 willingly unemployed, which was more uncomfortable than I’d imagined. Growing up in Asia, I’d been conditioned to believe that “a life well-lived” followed a clear rubric: education, career, family, being a “good person”. But who am I, bereft of that plan? If not in relation to someone else,…
In Pulumi, stacks can separate an infrastructure build by environment or by more granular concern. In a recent setup I worked on, we went with the latter approach. One stack handled compute, another handled networking and a third managed global artifacts like container images. Given that Pulumi stacks are all about independent interoperable units, the stack differentiation helped us reason about…
In a previous role, I worked on establishing an observability stack using OpenTelemetry (OTel) which mostly involved setting up instances of OTel collector to run across a distributed network. The main goal of this effort was to decouple data collection from data export, to more reliably gather data from various services for export. In the simplest of setups, logs, and metrics can be sent directly…
Across the startups that I’ve worked at, a recurring theme has been using Nomad and eventually migrating off of it. In one of them, which you might already be fairly acquainted with , Nomad was an initial iteration at constraint-based deployments enabling regional rollouts and seamless rescheduling across a fleet during maintenance events. For a company like Fly.io that gives customers the ability…
I was recently let go from my role. This wasn’t my first experience with an unexpected termination. In fact, this time last year I found myself in a similar situation albeit under different circumstances and with far more surprise. Yet this one felt like an especially low point in my career; unexpected in a similar way, but predictable given the circumstances. I was technically on an unofficial…
One of the hardest problems in local-first software is sync. In previous posts, I talked about OTs/CRDTs, the role of the cloud relative to the client, and general conflict resolution. But I never quite landed on the core theme behind those ideas; sync is the bread and butter of data reconciliation in local-first software. Briefly, a sync engine is responsible for collecting changes, resolving…
The goal of every data reconciliation algorithm is to ensure all replicas converge on the same state of the world, in spite of concurrent changes. The best way to achieve this is through a deterministic merge where the same result can be guaranteed regardless of the order or timing of updates. The backbone of a deterministic merge is one that is associative (grouping doesn’t matter), commutative…
When it comes to data reconciliation in the context of building real-time, collaborative user experiences, two major techniques are often discussed: Operational Transform (OT) and Conflict-free Data Replicated Types (CRDT). OT was developed in the late 80s and pioneered the foundational principles of real time collaborative editors. CRDTs emerged much later (around 2006), and were partly motivated…
In my last post, I argued that conflict in local-first is an inherently human problem. Without understanding the intent, and expectations of the end user, reconciliation is not always meaningful, given that correctness doesn’t equal usefulness from a user perspective. Before we dive into the technicalities of how conflicts get resolved, it’s worth exploring why drift happens in the first place and…
Conflict is a recurring topic of discussion in the local-first space. This is unsurprising given that real time collaboration and intermittent offline access inevitably introduces drift between replicas. When multiple users collaborate on a single application or even when a user works on the same application from separate devices, change conflicts occur. This echoes a foundational principle in…
With the rise of mobile in the early aughts, where updates and collaborative edits felt instantaneous (since they wrote directly to local storage and used shared network drives) the traditional client-server model of the web proved clunky. Instead of instantaneous updates that were common to mobile interactions, web updates incurred added latency from the extra round trip it had to make. The…
There’s nothing quite like the clarity of losing your job on a friday that forces you to contend with the bigger question; who and what kind of future do you want to build for? For the last year, I’ve been deep in the local-first ecosystem, learning alongside peers and building the infrastructure to power local-first software. Amidst the hype of AI and agent-driven code, I’ve witnessed a quiet hum…
As the saying goes there is no cloud, just someone else’s computer. Simplistic as it is, the saying captures a universally accepted reality: app data is better off hosted elsewhere living on someone else’s hardware (I’m looking at you GCP and AWS). The issue with this cloud-first sentiment is that it relegates the cloud as a panacea. Sure, giving control to the cloud frees us from the burden of…
The rise of the local-first movement was no anomaly. For years, browsers were no match for native environments that offered superior performance and offline capabilities. The gap was painfully obvious, native apps could store gigabytes of data and had access to a whole file system while browsers barely scraped by with megabytes. Modern browsers have since caught up over the last few years and the…
In my last post, I mentioned that what makes an app local-first is its ability to keep running even when a connection falters. At first glance, this might look a lot like the offline-first movement that arose alongside the progressive enhancement wave of the mid to late 2010s. But there’s a subtle distinction. Offline-first apps focused primarily on staying functional during network interruptions…
I’m frequently amused by how often local-first software gets mistaken for community initiated software. The ethos is spot on, but the confusion is revealing: cloud first has become the de facto standard for building apps. Local-first challenges this notion and subverts the dynamic between cloud and device. It puts the device at the center of operations. In this way, local-first apps remain…
I’ve been pretty quiet about my work life lately. But spoiler alert, I have a job (and have been at it for the last 3 months)! If you spoke to me earlier this year or read a previous post about my job search, you’d have heard that the journey to a job post layoff for me was an emotional rollercoaster. Interviewing was not a strength of mine. I spent months getting wooed by companies impressed by…
A few years ago, I received news of a professor’s passing in the most bizarrely impersonal way we tend to receive news these days: via a twitter mention from a college ex I hadn’t spoken to in years. The memorial service was set for that weekend in Central Massachusetts where he and his family lived. At the time, I was living in Cambridge, which was just a short train ride away. But I never made…
I got laid off for the first time back in September (surprise!) and it took a giant toll on me emotionally. For the last 10 or so years I’d managed to muscle my way through engineering jobs despite not having a formal degree in CS. I’ve had the immense privilege of working at some really cool companies, working my way from the front of the frontend through the bowels of the stack and (for a brief…
2024 was the year when I finally transitioned from surviving to thriving. With everything that transpired over the last few years, I got lost in the chaos of my reality and it’s taken me a second to “get back”. I’m not quite who I used to be, but I’ve become a version of myself that I’m quite pleased with, same same but different. The year came and went, for the…
Change is inevitable, especially so in applications that ebb and flow with system changes and user requirements. Generally, changes are codified in the form of an event; an event like a ddos attack for instance triggers a reaction or a response from one or more nodes, which ultimately mutates the overall state of the universe in a system. Undoubtedly, events rarely happen as one offs. At any given…
Distributed nodes share neither memory nor a logical clock. In order to effectively keep a system in running order, nodes need some way to communicate with one another to share data and coordinate complex tasks. Interprocess communication (IPC) plays an integral role in this. IPC is the mechanism that enables a server to query a database instance and resolve a request with the associated data…
In a distributed system, failures are to be expected. Whether it be a fault of the network or the node, failure is disruptive to the entire system. Failed processes struggle to recover and receive updates. Healthy nodes that try to connect to a failed node might fail completely or receive stale information, which can jeopardize the integrity and overall (C)onsistency of the system. To mitigate…
In a distributed system, communication is crucial to ensuring no node becomes stale. Updates can happen in 2 main ways, through consensus, or through gossip. In a consensus model, nodes attempt to reach quorum by mutually agreeing on the order of updates or periodically voting on and rotating the leader in charge of distributing updates to fellow nodes—I covered the ins and outs of this in an…
In an earlier post, I glossed over a concept that forms a fundamental tenet of systems design; the CAP theorem. To reiterate, the CAP theorem states that of 3 desirable traits in a distributed system, Consistency, Availability and Partition tolerance, only 2 can be prioritized at any given time. It highlights the tradeoffs we make when we design for a distributed setting. The CAP theorem was…
What does it mean to be consistent? In an earlier post, we talked about the importance of building consensus when it comes to ensuring consistency in a distributed setting. Loosely, consistency focuses on keeping all nodes in a system up to date on changes as they happen in real time or near real time. This ensures the reliability of the system as a whole, so users are never (fully) let down when…
In a distributed setup where nodes share the load of processing requests and storing/retrieving data, reaching consensus is crucial to ensuring consistency in an application. To return to the concert ticket sales example from the last post, we wouldn’t want 2 users buying the same seat since that violates the condition of assigned seating. Consensus can then be described as ensuring all nodes…
Demystifying Distributed Systems: the what and why If you deploy an app today, dealing with a distributed system is an inescapable problem. Whether it’s understanding how to balance requests, or synchronizing data replication, at least some aspect of your deployment will require orchestrating and coordinating multiple servers to operate in unison. The goal of “distributedness” here is to have…
Today was a really hard day. I lost the man who raised me. The man who was there for all my milestones. We didn’t always agree but he had a heart of gold and was always there for his family, his friends, his neighborhood, his community. Words can’t begin to express the immense grief I feel. The grief I feel as I process this loss alone from the confines of a hotel room 5 miles away, 3 days…
Didn’t make a lot of progress today and decided to play around with shapes instead. Here’s a heart -> https://codepen.io/shortdiv/pen/zYxGQBm?editors=1010
Geometry is the basis for drawing any shape in ThreeJS. As I covered in earlier posts, geometry in ThreeJS consists of vertices and faces, which can be defined by hand in order to create custom geometry. Of course, this task of defining your own vertices and faces is ambitious and requires a firm understanding of how math works in ThreeJS—knowledge which I currently do not have. To keep things…
Most shapes in ThreeJS and WebGL can be created using primitives many of which, you can use to create composite geometries, like this really neat christmas tree . Creating complex and unique geometries however takes effort and can be difficult to achieve by simply compositing, mutating and ”extruding” existing primitives in ThreeJS. A better approach, as I highlighted in a previous post, is to…
Walking at angles When drawing in 3D and even in 2D, we rely on shapes to make up a larger geometry, which then go on to form a more complex scene. This may seem rather straightforward but to a machine the task of drawing shapes could not be more complicated. This is largely because most machines only know how to render triangles. Take the humble square. Ordinarily, a square is drawn from point to…
Because we’re starting from the basics, this week we’ll focus largely on shapes and rendering shapes to the screen. This task may seem trivial but I assure you there’s a lot happening to occupy a week’s worth of content. To keep things simple, we’ll create our 3D images using ThreeJS, a JS library that abstracts a lot of the complexities of WebGL so you can write graphics with the power of…
At the start of 2018, I got the opportunity to be part of the first mini batch at the Recurse Center. Recurse Center is a self-directed, community-driven programming retreat in NYC aimed at attracting programmers who want to grow their programming chops in a rigorous yet, supportive environment. Students who attend Recurse generally come in with a dedicated project focus. This helps with setting a…
A key step in the data visualization process is the conversion of data to the right format. When working with geospatial related information in particular, data needs to be in geoJSON to be accurately superimposed onto a map. In many instances, geospatial data available via Open Data Portals like data.gov is already in a proper format to be mapped. However, there are times when the available data…
Redirect Rulez Browsers rely on URLs to access content on the web. One way to think about URLs is as a giant phone book for the web. When a browser wants access to a specific resource, it looks it up in the “phone book” and makes a request for it by visiting the address specified. However, like physical addresses, virtual addresses are not permanent and are subject to change. When a person moves…
Some of the most cutting edge and effective web performance optimizations, like prefetch and preconnect, involve being proactive. We make predictions to determine where a user is likely to go next and load resources ahead of time so page load is as fast as possible. While there is somewhat of a science to making these predictions, thanks to analytics tracking and extensive user testing, they are…
If you’ve spent some time in frontend land, chances are you’ve heard someone say “it’s just javascript” when referencing a framework, library or snazzy new tool. While the phrase has an element of truth in it—a framework like React compiles down to JavaScript for instance—it often betrays the full picture. For one, a frontend framework like React or Ember often involves complex-ish tooling,…
Componentization is the central concept driving frontend development today. It enables us to encapsulate core logic of a user interface into smaller chunks thereby making them easier to reason about. While each of these components exist in isolation, they work in concert to build a unified interface. This relative isolation also means that components are reusable and can be easily mixed and…
With 2018 drawing to a close, I’d like to take a moment to review some of my personal highlights from the past year. Attended Recurse Center 2018 was by far the first time that I hit the ground running at the start of the year. I successfully matriculated(?) at the Recurse Center in their first ever mini retreat and spent a full week in January learning and building alongside a bunch of…
If you’ve been keeping up with the JavaScripts, chances are you’ve heard snatches of excitement about the latest and greatest in web technology, WebAssembly. WebAssembly is a binary executable for the web that promises near-native performance for web applications. This means that graphics heavy applications like Photoshop, and AutoCAD can now be run in the browser without the need for clunky third…
In the last few years, virtual reality (VR) and augmented reality (AR), also known as cross reality or XR, have developed far beyond the niche realms of the gaming industry. There are now compelling use cases for using XR for building applications focused on education (Aurasma, Math Alive), home improvement (Wayfair, Ikea Place) and even beauty (Meitu) that are accessible to the masses. Despite…
One of the most memorable cinematic portrayals of a voice recognition system is Hal 9000, the seemingly omniscient computer from Stanley Kubrick’s 2001: A Space Odyssey. Hal captured our collective imaginations and opened our eyes to the possible future of a voice based human-computer interaction. Since the movie’s release in 1966, the landscape of voice recognition systems have changed…
The task of building for accessibility, while worthwhile, can be a rather daunting one. Strategies to incorporate it into a product frequently involve a fair amount of work, much of which tends to be manual and admittedly, tedious. The web accessibility community has made many strides to improving this by making accessibility more accessible for developers. In addition to the best practices and…