Recent content on Igor Kupczyński
We’ve received customer reports of workers producing bursts of UNAVAILABLE: No connection established. Last error: connect ECONNREFUSED x.x.x.x:xxxx in the logs. UNAVAILABLE is the canonical gRPC transient failure code and ECONNREFUSED is a lower-level TCP error meaning the SYN went out and an RST came back: the destination was reachable, but nothing was listening on the port. This article is a…
TL;DR # Two Cursor bugs bite when you nest git repos under a gitignored parent and open them in one window: Semantic search returns nothing when the .code-workspace lists the root folder ( . ) alongside its own nested subfolders. The agent’s Grep and Glob tools go blind on the nested repos. They build their search off the git tree and honor .gitignore directly, and the .cursorignore /…
A collection of load balancer surprises I’ve encountered. I’ll add new entries over time. NLB idle timeout silently drops connections # TIL: NLB doesn’t close the connection gracefully after idle timeout > .. no traffic for 350 seconds, then the connection is removed from the NLB flow table. If a client attempts to send traffic after the connection is no longer tracked, then NLB…
A Vagrant-based sandbox for running AI coding agents in full autonomous mode—without risking your host machine. The problem # AI coding agents work best when they can run autonomously. Claude Code’s claude --dangerously-skip-permissions and Codex’s codex --full-auto let agents execute commands without asking for approval at each step. But this is scary for good reason—agents have…
The following incident is fictional, but the architecture lessons are real. The incident # Monday morning. You’re reviewing last week’s metrics and notice a gap in the graphs — your staging environment was down for about 5 minutes on Thursday afternoon. The AWS console shows the instance is healthy, but uptime is only 4 days. Digging into the EC2 console, you find it: AWS had sent a…
Hatchet is a durable task orchestration platform, built on PostgreSQL. Competitors include Celery or Temporal. Hatchet scaled from 20k tasks/month to 1B+/month without abandoning Postgres. Some patterns caught my attention. PostgreSQL as the Universal Backend # Conventional wisdom says separate your queue (Redis/RabbitMQ) from your state (PostgreSQL). Different access patterns, different tools.…
I recently reviewed a Go codebase that implemented connection limiting in a TCP proxy. The goal was to protect the backend from being overwhelmed by too many concurrent connections. The initial implementation used a semaphore to gatekeep the Accept() call. While this seems intuitive, it relies on the kernel’s backlog behavior in ways that can cause significant operational issues. TL;DR #…
If you’re using AI coding agents—and who isn’t these days—you’ve probably noticed each tool wants its own configuration file. Cursor reads .cursor/rules , Claude looks for CLAUDE.md , and every new agent seems to invent yet another convention. This proliferation of tool-specific files is annoying. Enter AGENTS.md : a unified standard that consolidates project context for AI…
I found this excellent article by Utkarsh Kanwat: Betting Against Agents . Compounding Works Both Ways # We usually talk about compounding in positive terms—interest accumulates, wealth grows, skills build on skills. But with AI agents, compounding works against you: the failure rate of an agentic process compounds with each step . This framing really clicked for me. I’ve noticed that…
I recently gave a talk at an internal Elastic conference about prompt engineering techniques in 2025, accompanied by live demonstrations using Mistral via Ollama, VS Code Copilot, and Windsurf. Here’s a brief overview of the key topics covered. Why Prompt Engineering Matters # Prompt engineering dramatically improves output quality, enables structured responses, and allows the use of…
I often change my default browser. I use Chrome for work and Safari for personal. It would be nice to have this in Raycast. Raycast is a Spotlight-like app launcher for macOS. Ideal UX: type chbr , get a list of browsers, pick the one you want to be the default, and hit return . Sounds simple enough. The problem? I’m neither a TypeScript nor a macOS developer. Vibe coding to the rescue # In…
I came across an interesting article by Mai-Lan Tomsen Bukovec, AWS Vice President of Technology, about a framework for Principal Engineer roles . The framework originated from their observations of how Principal Engineers work at AWS, particularly in the S3 team. I’m summarizing it here for easier reference. The Six Principal Engineer Roles # The framework defines six distinct roles that…
Recently, I encountered a fascinating incident where some API endpoints received unexpected traffic. Services were getting requests they weren’t configured to handle, which shouldn’t have been possible with our ALB routing rules. Here’s what I learned. Details of this incident and the service involved are anonymized to not distract from the key learning. TL;DR # When all weighted…
I came across an interesting method called “Chain of Density” that promises to generate increasingly concise and information-rich summaries. Let me share what I’ve learned. Chain of Density is an iterative summarization technique that aims to create progressively denser summaries of a given text. The process involves repeating two key steps multiple times: Identify 1-3…
Paul Graham’s “Founder Mode” Paul Graham’s essay Founder Mode is doing the rounds in social media: In effect there are two different ways to run a company: founder mode and manager mode. Till now most people even in Silicon Valley have implicitly assumed that scaling a startup meant switching to manager mode. There are as far as I know no books specifically about founder…
TIL S3 supports conditional writes Check out the official AWS announcement and the more detailed S3 Conditional requests guide . Amazon S3 adds support for conditional writes that can check for the existence of an object before creating it. (…) You can perform conditional writes using PutObject or CompleteMultipartUpload API requests in both general purpose and directory buckets. ……
Great book by the star CEO of Snowflake. Amp It Up by Frank Slootman was highly recommended on Twitter X and Hacker News. Frank Slootman was the CEO of a couple of high-profile companies that he took to IPO: Snowflake, Service Now, and Data Domain. In the book, he shares his management philosophy and supporting stories. The main pillars are: Raise your standards Align your people and culture…
Go binaries are not inherently container-aware, specifically in that they do not account for memory and CPU limits. TL;DR When running a Go application in a container with set CPU or memory limits, you should inform the Go runtime of these limits using the GOMAXPROCS and GOMEMLIMIT environment variables. For example, if you run your application as follows: docker run --cpus = '4.0' --memory =…
Remember that Java slogan, Write once, run anywhere ? Go compiles directly to binaries, which target specific architectures. There’s no intermediate byte code. You cross-compile, and once you produce a binary for the target architecture: GOOS=linux GOARCH=amd64 go build , it just works. Well, at least usually. With Cgo , go can interop with C code. There’s some magic involved, the…
Bookmarks on salary negotiation: Classic from Patrick McKenzie Salary Negotiation: Make More Money, Be More Valued 101 on offer negotiation from Rishi Taparia Negotiating your startup job offer Other links in no particular order Ten Rules for Negotiating a Job Offer 15 Rules for Negotiating a Job Offer How to negotiate salary: 9 tips from a pro salary negotiator Books: My favorite is Never Split…
Use cases for Private Link and differences in its implementation across the major Cloud Providers. Changelog November 27, 2024 : Updated to reflect AWS PrivateLink’s new cross-region connectivity support . This feature brings AWS PrivateLink’s capabilities more in line with Azure and GCP’s offerings. January 30, 2022 : Original post published. Private Link # Private Link allows…
Theory of constraints applied to IT operations. I’ve read The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, George Spafford. This book is amazing. If you follow the lean manufacturing practices, or enjoyed The Goal (see The Slowest Hiker ), or simply want to improve the way IT operations work with the rest of the org (and improve the org in…
How to find out a container responsible for a given open port on a host. Recently, I’ve debugged an issue with a server running multiple docker containers. They interact with each other by exposing TCP ports on the host and communicating over these ports. In this post: I describe a surprising discovery about host network mode containers (spoiler: they don’t show under PORTS in docker…
I’ve created a telegram bot to give me a daily updates to a webpage I was tracking. This is a hobby project and the hosting price is an important concern. The problem Context The solution Why not open-source the code? Notes on the project Telegram bot API is really good for experimentation Anyone can chat to your bot Google App Engine Secrets are hard to manage If you don’t know GAE…
This post has a companion github repo github.com/igor-kupczynski/remote-debug-example – check the repo for a simple example. Coming to go from the JVM background I often find myself looking for analogs of the features I rely on the JVM ecosystem. One example of such a feature is remote debugging – which is built into JVMs. Read on to find out how to accomplish this in go. Remote debug…
Natural project planning is a project planning method described by David Allen in his bestseller Getting Things Done . In this post, I describe the method. Natural project planning consists of a couple of steps: Define purpose and principles. Visualize goals and outcomes. Brainstorm. Organize the ideas. Define next actions. What are the goals of natural project planning? Help clarify the steps…
Updated: 2022-09-29. To reflect that BoringCrypto is now part of the main branch. Updated: 2022-11-20. To add a section on TLS 1.3 and FIPS. Updated: 2023-01-07. To add a section on automatic configuration at compile time: Force FIPS-compliant crypto at compile time . Updated: 2023-08-07. To add a note of microsoft/go . If you work with US government entities or corporations in regulated markets…
I’ve read Zero to One: Notes on Startups, or How to Build the Future by Peter Thiel. Here are my notes on the book. Background # Peter Thiel is a well known Silicon Valley founder and investor. He is often portrait as a contrarian and is quite a controversial figure. He is a founder of PayPal and Plantir. I’ve heard very good reviews of the book and it was on my list for quite some…
How to build a great engineering organization — advice from Elastic’s SVP Engineering. Update 2019-11-29: Kevin published Leadership @ Elastic blogpost. Kevin’s post is more of high-level and strategical than the content here. I’ve recently come back from Elastic’s Global All Hands 2019 (or simply GAH 2019). It is a conference-style internal Elastic event. The goal is to…
How to setup Server Name Indication with Traefik. In this blog post we discuss briefly what an SNI is, show a simple example of setting up SNI with Traefik — a popular reverse proxy written in go — and then show some options for implementing it in “plain” go. This blog has a companion repo traefik-tls-demo , be sure to check it if you want to try the example. Server Name…
How to improve any process. One of the two most often requested features of Go is the support for generics. Arguably, generics are super important and they are taken for granted in most of the modern programming languages. In fact, Go developers get a lot of flak from their Haskell-minded colleagues being forced to use a language with such a poor feature set. And yet, Go is one of the fastest…
In the previous post , we’ve presented an AWS Lambda function to automatically invalidate resources in CloudFront distribution when underlying objects in an S3 bucket change. Instead of clicking and copy-pasting in the AWS Console, we can use terraform to set this function up. In fact, I’ve already made it a part of my terraform static aws website — terraform module which sets up…
There are only two hard things in Computer Science: cache invalidation and naming things. – Phil Karlton In this post we will focus on cache invalidation. Problem statement # This blog is a static site hosted on S3 with CloudFront being used as a CDN / cache. Blog post about this setup . UserAgent requests kupczynski.info/posts/abc.html from CloudFront . If this is not in the CloudFront cache,…
One of the VPNs I connect to sets the DNS server for the link. It is an OpenVPN with DNS option. The problem is that is doesn’t really work out of the box in Ubuntu (at least 18.04 and 18.10) — the VPN DNS is not consulted. At the time of writing, I’m on 18.10, and I write it from that perspective. Ubuntu uses systemd-resolved service to provide a local DNS resolution. We need to…
Today I’ve rediscovered Fire and Motion blogpost by Joel Spolsky. I recommend you to read it. It start with how you can get only a couple of productive hours per day. And this is OK. Joel mentions a colleague, who worked only from 12 to 5, and still had more done than the average other team member. These productive hours are called the flow . One of the tricks to put yourself in the flow is…
The procedure changed on recent ubuntu compared to what I used to do. I used change /etc/security/limits.conf . The procedure on ubuntu 18.10 that worked for me is this. Go to /etc/systemd/system.conf Uncomment DefaultLimitNOFILE and set your limit there, e.g. $ grep NOFILE /etc/systemd/system.conf DefaultLimitNOFILE = 65535 Restart Profit Before: $ ulimit -n 1024 $ ulimit -Sn 1024 $ ulimit -Hn…
After a upgrade from 18.04 my external monitor stopped working. Installing lightdm was the solution. Update 2020-11-10. Abish Vijayan posted a solution for Ubuntu 20.10 in the comments below. Made it to work. Disabled Wayland in gdm3. https://askubuntu.com/questions/975094/how-to-disable-wayland-in-17-10-in-gdm3-login-screen Update 2019-11-15 on Ubuntu 19.10 I still need the lightdm to support…
I’ve got a legacy windows app that I need to run from time to time. The app is written in java , but uses some 32 bit JNI dll s. I’ve used to keep a windows VM around to run it, but recently I was able to “port” it to wine . In short, you need install wine on ubuntu: # I've needed a 32 bit version $ sudo dpkg --add-architecture i386 $ sudo apt-get -y update && sudo apt-get…
How to minitor health of SSD drives, case study on my failed Kingston SUV500MS480G. I use my old Samsung Series 7 ultrabook for some personal tasks. It originally had 120 GB SSD, but I’ve upgraded it recently to Kingston 480GB SSD. I’ve first installed Ubuntu 18.04, but then upgraded to 18.10. Everything seem fine, but after few hours of usage and a restart it started complaining about…
Back to Linux after almost 3 year hiatus. Switching back from MacBook to Thinkapad and Ubuntu as the main work machine. My MBP 13’’ aged quite nicely, but compiling large scala codebases and running VMs / containers became quite frustrating. I was eligible for a refresher at Elastic and I’ve decided to try Thinkpad P51 + Ubuntu. This is a 15’’ beast with 64 GB of RAM.…
Fix annoying issues with Zoom and HiDPI on Kubuntu 18.04. I’ve got a new laptop for my work for Elastic recently. This time I’ve opted for a Thinkpad and not for a MacBook. I’ve installed Kubuntu 18.04. Linux on desktop or laptop gives you a lot of opportunities to blog about something you’ve fixed. Here comes a first one. How to make zoom — the conferencing software — play…
Notes I’ve took while doing a microbenchmark of some code. Not really structured, but maybe useful for future me. Webpage: http://openjdk.java.net/projects/code-tools/jmh/ Getting started # Use an archetype: mvn archetype:generate \ -DinteractiveMode = false \ -DarchetypeGroupId = org.openjdk.jmh \ -DarchetypeArtifactId = jmh-java-benchmark-archetype \ -DgroupId = info.kupczynski \…
Classic advice on time management. A colleague of mine (thanks Stu!) recently recommended me a classic article Management Time: Who’s Got the Monkey? It is classic indeed — published in 1974 . Some of the wording is in need of modernizing (e.g. the boss-subordinate hierarchy changed into much more flat organizations), but I think it aged quite well. I’ve enjoyed it and I…
This post describes how to enable Class Data Sharing (CDS) for a java app and what are the benefits of doing so. I’ve recently seen a talk about CDS by Volker Simonis which was the inspiration to write this blogpost. Table of Contents What is Class Data Sharing? Prepare CDS Intro — we need OpenJDK 10 Makefile Create a list of classes used by Elasticsearch JVM error Prepopulate the class…
Conference notes. Geecon 2018 was an important event in Center- and Eastern- European JVM conference agenda. It was over a year since I’ve attended my last non-elastic related event so I thought it is a great venue to pick up some ideas and look where the industry is heading nowadays. And what a great venue it was! In this pots I’ll present mostly raw notes I’ve took during the…
I just don’t grok the vi (yet). Three weeks ago, I’ve ditched most of my emacs config and installed spacemacs . The main driver was that I wanted to try modal editing instead of emacs key chords. I could’ve tried vim of course, but I’m already invested in emacs and feel at home there. Evil-mode # Luckily, there is a brilliant vim emulation in emacs, called evil mode . I…
I’ve read 12 Rules for Life: An Antidote to Chaos by Jordan Peterson. Background # I’ve heard about Professor Jordan Peterson because of my colleague, who is a fan of him. I’ve watched the famous BBC interview with Cathy Newman. The interview itself is fun to watch as a lesson in debating. Feel free to google it, there are a plenty of recording and analysis of the interview on…
Talk about career options for pupils of my old highschool I was asked to do a short talk on the career options in IT for Henryk Sienkiewicz grammar school. I agreed without hesitation. It went quite well, I hope it helped some of the folks to made their mind. It started with an interview (to be publish at schools website), and then a talk in a classroom format. For posterity, I’m attaching…
Some of my coworkers use Eshell inside emacs as their main terminal emulator . I had a little bit of envy, so I decided to give it a try. Unfortunately, I’m not going to use it as my main shell. At least not yet. How fast can you print? # What is the issue then? — performance. I’ve run a build which outputs a lot of, mostly useless, warnings to the terminal. And boy, it took…
I’ve recently read Insanely Simple: The Obsession That Drives Apple Success by Ken Segall. This post summarizes my notes taken while reading the book. Background # Ken Segall is a marketing director who worked with Steve Jobs at Apple and NeXT for 10 years or so. He also worked with Dell, Intel, IBM and other tech companies. In the book he offers a lot of stories on how Apple and Steve Jobs…