Recent content on
We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. Sometimes it self-healed, sometimes it kept restarting and we lost logs. This post…
I recently ran into an issue after upgrading the containerd client library to v2.3.0. The root cause was a change in one of containerd's dependencies, continuity, that started enforcing fsync when copying files. containerd/continuity is a library in containerd that provides filesystem utilities: file copying, directory diffing, disk usage accounting, metadata preservation, and extended attribute…
A container platform shipped a change to detect CPU architecture mismatches at image pull time. If an image's config says arm64 but the host is amd64, fail fast instead of letting the container start and die with exec format error. The implementation is correct, tests passed and reviewers approved. However, it caused a production impact and a quick rollback. It turns out customers had been running…
If you build software that runs in someone else's environment, anything that isn't a SaaS you operate yourself, the following code can break your customers: 1if not hostname.endswith('.'): 2 hostname += '.' Appending a trailing dot to a domain name before you resolve it feels "more correct" when you own the domain name. It can even be a performance optimization (more on that later). But in…
I recently ran into a Go test timeout that turned out to be a deadlock. That was surprising because Go has a built-in deadlock detector. The simplest possible deadlock is a goroutine that locks the same mutex twice. Running the following code exits with fatal error: all goroutines are asleep - deadlock! as expected. 1package main 2 3import 'sync' 4 5func main() { 6 var mu sync.Mutex 7 mu.Lock() 8…
Bottlerocket v11.0.0 dropped the socat package from bottlerocket-core-kit (#742). If you had a service that relied on socat to bridge a Unix Domain Socket (UDS) to a TCP port, you need a replacement. This post shows how to use systemd-socket-proxyd instead. Why socat was used soci-snapshotter exposes its metrics endpoint over a Unix Domain Socket at /run/soci-snapshotter-grpc/metrics.sock. The…
CGO_ENABLED defaults to 1. That means a standard go build produces a binary that links against C libraries (e.g., glibc) at runtime. For many parts of the Go standard library, there is a C-backed implementation and a pure-Go implementation. CGO_ENABLED selects which one gets compiled in. Pure-Go alternatives also exist for third-party libraries, so it is likely you can turn off cgo by setting…
We run a container platform. For privacy and security reasons, we do not collect kernel logs because customer workloads use the same kernel as the host and kernel messages can contain sensitive customer data, such as command-line arguments surfaced in audit logs. However, we recently hit a blind spot: foo.service was killed with no trace in its own logs or systemd logs. No error, no panic, no…
This post is part 1 of a series of learnings from nilaway. nilaway is a static analysis tool that detects potential nil panics in Go code. It does report false positives, but it's far from naive. One limitation is that nilaway is flow-sensitive (it understands if x != nil) but not value-correlation-sensitive (it doesn't understand "if y == SomeConst, then x is non-nil"). From my experience, most…
I recently encountered IMDS (Instance Metadata Service) request failures with this error: 1'caller': 'actor/actor.go:101', 2'error': 'operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, x available, y requested The root cause: the aws-sdk-go-v2 IMDS client exhausted its retry token bucket. The AWS SDK for Go v2 implements client-side rate limiting ("retry…
WaitGroup and channels are two powerful primitives in Go for synchronizing goroutines. A common pattern uses a WaitGroup to wait for goroutines completion: 1wg.Add(1) 2go func() { 3 defer wg.Done() 4 for { 5 select { 6 case <- done: 7 return 8 case task <- tasks: 9 handle(task) 10 } 11 } 12}() 13wg.Wait() In this post, we'll explore an interesting use of WaitGroup and channels where the WaitGroup…
While prototyping Bottlerocket, I discovered it doesn't recognize additional EBS volumes specified through Block device mappings on Xen. For example, launching the same AMI on t2.medium (Xen) and t3.medium (Nitro) with "DeviceName=/dev/xvdcz": On Nitro, the device appears at /dev/nvme1n1 and /dev/disk/by-ebs-id/xvdcz. On Xen, it appears at /dev/xvdcz. In the prototype, the xvdcz volume is…
I recently debugged a resource leak where a systemd service kept restarting while leaving a process behind after each restart. The root cause isn't particularly interesting: a backward-incompatible third-party dependency upgrade. But the debugging process and lessons learned are. Thousands of zombie processes from a systemd service I have a foo.service that runs /usr/bin/start-foo, which spawns a…
I recently debugged a mysterious latency issue: after migrating a systemd service from path-activation to socket-activation, there was a consistent ~1 second time-to-available latency. The culprit was a bad practice—starting the daemon program as a new process in socket-activation. Let's dive into the details. Starting soci-snapshotter On-Demand Using Socket Activation soci-snapshotter is an…
A bug caused around 0.5% of container workloads to fail to start during load test. This post walks through the bug and its fix, an interesting mix of Linux namespaces, Go concurrency, and syscalls. The need to run a program in its own network namespace and mount namespace soci-snapshotter is an open-source containerd snapshotter plugin that enables pulling OCI images in lazy loading mode. It…
When I first moved building Bottlerocket AMI from an EC2 host to AWS CodeBuild, I was hit by a very slow build. On an EC2 instance, I built both the x86 and Arm versions on x86 instances, and fresh builds finished in 5 minutes. However, on CodeBuild with more vCPU and memory, the build process was painfully slower. The x86 CodeBuild uses a compute of "145 GB memory, 72 vCPUs". The build finishes…
Early this year, we migrated containerd from v1.7 to v2.0.5. However, we quickly noticed image pulls from Amazon Elastic Container Registry (ECR) began failing for both public and private ECR repositories. For example: 1# public ECR 2FATA[0031] failed to resolve reference 'public.ecr.aws/aws-cli/aws-cli:2.31.5@sha256:9cb6ab9c8852d7e1e63f43299dca0628e92448d1c7589f7ed40344e7f61aad59': \ 3unexpected…
In Detect and fix rare cases where the primary ENI does not serve default traffic , we used IMDS "meta-data/mac" to get the primary ENI's MAC address. However, we encountered the following errors in 0.5% of EC2 ARM instance launches: 1failed to get IMDS /mac: operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, 2http response error StatusCode: 401, request to EC2 IMDS…
There are a few programs we install in Bottlerocket that cannot be built from source. For these programs, we download the binary from a secure repository and install it using an RPM spec like this: 1# foo.spec 2Name: %{_cross_os}foo 3 4Source0: foo 5 6%install 7install -d %{buildroot}%{_cross_sbindir} 8install -D -p -m 0755 %{S:0} %{buildroot}%{_cross_sbindir} A teammate discovered that the foo…
Bottlerocket is a Linux-based operating system optimized for hosting containers. We use Bottlerocket to run millions of containers each day. There are three key differences between Bottlerocket and common Linux distributions like Amazon Linux 2023: The rootfs is read-only. There is no package manager (e.g., yum) in Bottlerocket. Each package in Bottlerocket must be built into the OS variant.…
Bottlerocket is a Linux-based operating system optimized for hosting containers. At my work, we migrated from Amazon Linux to Bottlerocket and experienced the following benefits: Developer-friendly: Easy to understand and fast to build. RPM spec and configuration TOML files are all you need. Every developer can build a Bottlerocket AMI on an EC2 instance in just a few minutes. For example, I can…
I recently dealt with a server livelock issue caused by memory page thrashing. This post refreshes the Linux memory basics I found useful for debugging the issue. Much of the content is from Chapter 7 of Systems Performance: Enterprise and the Cloud. Virtual Memory Virtual memory is an abstraction that provides each process and the kernel with its own large, linear, and private address space.…
During testing, we encountered a rare scenario when launching EC2 instances with multiple ENIs: the primary ENI (device index 0) does not serve default network traffic. This occurs in approximately 1 out of 10,000 launches (0.01%). For example, when configuring two ENIs on an instance—ENI-0 (deviceIndex=0) from subnet-0 and ENI-1 (deviceIndex=1) from subnet-1—Linux may recognize eth0 as being from…
Security-Enhanced Linux (SELinux) is a mandatory access control (MAC) system that enhances Linux security. "Mandatory" means access control is strictly enforced by predefined policy rules—users and processes cannot modify these rules at will, ensuring security is not left to individual discretion. SELinux is available in major distributions, including Amazon Linux 2023 (AL2023) and Bottlerocket.…
Go is known for its backward compatibility, simplicity, and six-month release cycle. But that can sometimes lead to code that works yet isn't as modern as it could be. This post is a living document where I note modern Go idioms I've used to improve clarity and maintainability. Use GOOS and GOARCH in file names for build constraints When targeting specific operating systems or architectures, Go…
Shell scripts are infamous for security issues and surprising behavior, so when possible, it's better to avoid using shell. For instance, we built a container platform using the Bottlerocket OS, and we didn't even install a shell. If someone needs to run a shell, it must be run inside a container. That said, shell is still handy for ad hoc scripting. In this post, I'll share a few surprising…
I recently worked on getting amazon-ssm-agent to run inside containers on Bottlerocket. During that process, I ran into a TLS issue connecting to amazonaws.com. The root cause turned out be interesting and we'll walk through it in this post. Running amazon-ssm-agent in a container: why and how? To enable sessions between a container and the outside world, we followed the same approach as the ECS…
A recent faulty release disrupted service for some customers. The root cause was a concurrency bug involving x/sync/errgroup and context cancellation. This post shares three practices we learned from the incident. These practices will help us catch similar issues during code review or alert us to problems in production. What does the buggy code do? I've simplified the program for this post as…
We've been using journald-to-cwl to ship journal logs from EC2 instances to CloudWatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panicking on expected errors in Go is generally a bad idea. Where Do False Positive Alarms Come From? We run many Go programs…
This week, I needed to install the Amazon SSM Agent and was surprised to find that GPG (GNU Privacy Guard) was the only way to verify the download. I had assumed that software downloads verification had largely transitioned to PKI (Public Key Infrastructure). This short post is a refresh on GPG. OpenPGP is an open standard for encrypting and signing data, originally derived from PGP (Pretty Good…
While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat program The following Go program vm-demo.go demonstrates memory bloat by allocating…
It's a beautiful day, and it started with a simple code review: 1# tools/foo/main.go 2- fmt.Println('found it') 3+ log.Println('found it') The author explained the advantages of using a logging library over plain printf. The rationale was straightforward, so I approved the change without hesitation. However, two hours later, another code review came through—this time reverting the change because…
We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk usage compared to the same workload on AL2. In this post, I'll share my investigation. AL2023 Has Less Free Disk Space with ext4, Compared to AL2 Although disk usage metrics increased on AL2023, the "Used"…
Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. These fall into two categories: voluntarily choosing to die, and involuntarily being…
As Amazon Linux 2 (AL2) approaches its End of Life on June 30, 2025, we have started migrating our container platform from AL2 to Bottlerocket. The migration encountered a few speed bumps. In this post, we'll examine one of them: missing container disk I/O stats. Why are container I/O dashboards blank? Since Bottlerocket shares the same kernel used by Amazon Linux 2023 (AL2023), I will use AL2023…
I've been building a service for a month, and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigorous" manual tests so I launched 100 EC2 instances. Surprise! Around 28 instances failed to launch. What's going on? All failed instances were stuck in the "initializing" state, and the only way to connect to them was through EC2 Serial Console. There, I noticed…
Given an array of numbers A, find out whether it contains a 1-3-2 pattern. A 1-3-2 pattern is a subsequence of three numbers, A[i], A[j] and A[k] such that i < j < k and A[i] < A[k] < A[j]. For clarity, let's call the 1-3-2 pattern the Bronze-Gold-Silver pattern. If A[j] is Gold, then we should consider the minimum number from A[0:j) to be Bronze, because it gives us the largest range for picking…
Tony Hoare invented QuickSort in 1961. At the time of its publication, the best comparison-based sorting algorithm was merge sort. Merge sort divides an unordered array into two equally sized subarrays, sorts each subarray, and then merge the two subarrays to produce a sorted array. Merge sort is simple to understand. However, quicksort is just as simple as merge sort but more elegant. In…
I often find myself confused by "upstream" and "downstream", in the context of software development. They bother me so much that I avoid using them in my own writing and I have to pause whenever I see them. In this post, I'll show a simple rule that helps you remember the difference: downstream adds value to the output of upstream. Downstream adds value to the output of upstream. Let's take a…
Modern CPUs operate significantly faster than memory. A 4.5 GHz x86_64 CPU operates 30 times faster than 6000 MHz DDR5 memory with CAS Latency 36. When accounting for latencies from the bus and memory coherency protocols, memory can be 100 times slower than registers. To mitigate this speed gap, CPUs use layers of caches organized around cache lines, typically 64 bytes each. However, programming…
Given a non-decreasing array and a target value, we can find the target in logarithmic time using binary search. My first programming language was C++, and the C++ Standard Template Library (STL) provides two functions for this task: iterator lower_bound(first, last, value) returns the smallest index with a value greater than or equal to the target. Put another way, if you were to insert the…
Problem Given an integer array A, in one step, remove all elements A[i] where A[i-1] > A[i]. Return the number of steps performed until A becomes a non-decreasing array. See examples at LeetCode 2289. Solution The naive approach executes steps one by one. Store integers in a Linked List. At each step, find all integers that are smaller than its left neighbor and remove them. However, the time…
Problem In a connected network consisting of N nodes, each node is connected to either one or two neighbors, forming a line topology. The task is to develop a program that runs on each node, calculating the total number of nodes in the network. Each node is aware of its neighboring nodes and can exchange messages with them. It is important to note that nodes do not share memory; the only means of…
Problem An event consists of multiple properties, each defined as a key-value pair, where the key is a string and the value is of a primitive type such as numbers or strings. Importantly, each event must include a mandatory 'Name' property. Given a list of events, the task is to count the number of events based on specified properties. To illustrate, let's consider an example with four events and…
ClickHouse is a popular OLAP database. It speaks SQL and earns the reputation of "fast and resource efficient". But the support of SQL comes with surprises if not careful. In this blog, I show that a simple query of nested WITH clauses in ClickHouse generates factorial number of subqueries. The simple query is short, reads nothing, process nothing and returns nothing. Yet, it uses a lot of CPU and…
In distributed systems, because there are too many requests to be handled by a single server reliably, requests are handled by a cluster of servers. In order to get high availability, the technique of distributing requests to servers needs to satisfy the following three requirements. Even distribution. Each backend take about M/N requests, where M is the number of requests and N is the number of…
A function is monotonic if it preserves the order of its arguments, i.e., if $x \le y$, then $f(x) \le f(y)$. In this post, we examine a class of problems where the argument is an interval. By identifying monotonic functions, we can reduce the number of intervals to enumerate by an order of magnitude, from $O(n^2)$ to $O(n)$. This algorithm is often known as sliding window, because enumerating…
Given an array of numbers $A$, for each number $A[i]$, find the largest subarray that contains $A[i]$ and $A[i]$ is the minimum of the subarray. For example, for $A=[2, 0, 3, 5, 1, 1, 0, 2 1]$, the largest subarray for $A[4]$ is $[3,5,1,1]$. Let's represent the subarray for $A[i]$ as the left boundary $l[i]$ and right boundary $r[i]$. $$l[i] = \min_{j \le i }\lbrace \forall_{j \le k \le i} A[k]…
Every Go program has a runtime. The runtime implements garbage collection, concurrency, stack management, and other critical features. We can configure the runtime by setting variables. In this post, we will look at GOMAXPROCS, a variable that configures concurrency. You may get free performance boost by setting GOMAXPROCS when running Go in containers. What is GOMAXPROCS? The GOMAXPROCS variable…
We plan to load test our product before public Beta, with two goals in mind. Find out bottlenecks: figure out road maps of performance improvement and prepare for oncalls. Understand how much workload we can support with fixed resources. This shapes the pricing strategy and determines number of Beta partners to onboard, without burning runway. Because it is complicated to generate meaningful loads…