Recent content on howardjohn's blog
Recently I have been spending a lot of time analyzing and optimizing memory usage in our Rust reverse-proxy, agentgateway . One thing that repeatedly came up was a surprisingly large amount of memory allocated to innocent-looking Tokio mpsc channels. In my naive understanding, I would have assumed the following allocation pattern: struct BigStruct { data : [ u8 ; 1024 ], } fn main () { //…
When trying to use any Kubernetes CRD (or even core types) I am pretty much always pulling up a reference doc of some sort to see what fields are available and what they do. However, I have pretty much always found the options for this to be pretty bad. To remedy this, I have been working on a new tool that automatically generates documentation for Kubernetes CRDs, and I wanted to share it here.
This is part two of a series. Envoy Quirks Part 1: Clear Route Cache Envoy Quirks Part 2: Filter Chain Match (this post) One of Envoy's core features is, of course, its ability to match traffic and route it to the appropriate destination. This is done at two levels generally: Filter Chain Matches define the top level matching of traffic, matching on attributes of the TCP and TLS handshake like…
After years of working on a project running our own testing infrastructure, I have been working on Agentgateway utilizing the free GitHub Actions runners. While the free tier is quite generous, the performance is... not. As someone who has spent a lot of time optimizing build times, I found myself quite disappointed that after going through considerable lengths to optimize our workflows, our…
The go.mod file contains a mandatory go <version number> directive. Since Go 1.21, when a change was introduced to make this include a full patch number ( 1.21.0 instead of 1.21 ), a number of projects have started using this wrong, hurting everyone. The version is the minimum version your project can be compiled with . It is not the version you use to compile your project, but the minimum version…
With the retirement of ingress-nginx I've seen a lot of frustrations with Gateway API migrations due to differences in some of the resource models. The common problem is that Ingress users are often running self-service models, where application teams fully own their ingress configuration, including TLS certificates. This would look something like so: apiVersion : networking.k8s.io/v1 kind :…
This is part one of a series. Envoy Quirks Part 1: Clear Route Cache (this post) Envoy Quirks Part 2: Filter Chain Match Over the years working with Envoy (via Istio), I've come across quite a few quirks and gotchas. I thought it would be fun to share some of them, and how to work around them. Many of these surprise even Envoy experts! To start things off, lets talk about the "Clear route cache"…
Recently I had a need to run a bunch of steps in a bash script, with interweaved dependencies for a CI pipeline. While a simpler approach would do a linear flow like: Setup Kubernetes cluster. Deploy dependencies to Kubernetes. Build image 1. Build image 2. Run tests. could work, its slow; we don't take advantage that we could build our images while the cluster is setting up, for instance.…
When building out Agentgateway , we had a desire to introduce an embedded expression language to allow users to write custom logic to be evaluated at runtime. This is tremendously useful for a variety of use cases, such as: Extracting fields to log ( request.headers["user-agent"] ). Evaluating authorization conditions ( jwt.sub == "admin" || request.path == "/public" ). Manipulating fields in…
As I ramp up coding agent usage, I found myself wanting to share the Rust build cache across multiple copies of the same project (via git worktrees) to avoid multiple-minute cold builds. This was harder than expected, but I was able to get something working. First, the failed attempts Attempt 1: just copy the target directory This seemed like a good idea, but took about 2 minutes which was almost…
The new Pod Certificates feature in Kubernetes v1.35 promises to "drastically simplify service mesh architectures". It will not.
While typical HTTP request routing happens based on the request headers/path, sometimes routing based on the body is useful. In particular, as AI use cases become more prevalent, the need to route based on the model field of the JSON request body is pretty handy to serve multiple models or route to different external LLMs. The Gateway API Inference Extension solution to this is to deploy an…
How I run Agentgateway on my homelab setup.
Common Expression Language (CEL) is a great little language for embedding users' custom logic into an application. However, typically when I discuss it, I get a response something along the lines of "AHHHH!! I HATE CEL!!!" I, too, have been in that position. However, when building Agentgateway I decided to fully embrace CEL throughout the entire stack, and the results have been great. The problems…
I've spent the majority of my career building Istio's control plane, Istiod, with an emphasis on making it highly performant and scalable. And while it has come a very (very) long way, it's still a long way off from what a control plane could be. It's not alone. When I worked on building an open benchmark of Kubernetes Gateway control planes, I was surprised to find that no implementation met what…
An often overlooked part of the Rust license is that within a year of usage, users are required to make a comparison to their (previously) favorite language. While I am a bit late, I am ready to pay my dues. Given the overabundance of Rust vs X content, I'll try to cover only areas that haven't been discussed to death. Lifetimes and Borrows Did I say I wouldn't cover areas everyone has already…
In the past few years, CPUs have gotten really fast. Shockingly fast! Yet most people are stuck on previous generation mobile chips (whether by choice, or by their companies choice), at a huge detriment to their productivity. Meanwhile, AI coding subscriptions like Cursor are all the rage these days. I'll skip the debate on exactly how useful these tools are, and focus on the pricing. Cursor is…
AI workloads introduce new requirements on networking infrastructure, but the same core requirements that service meshes solve not only remain, but are exacerbated. Your AI workloads still need a service mesh - they just need a better one
Gloo Mesh’s ambient multi-cluster mode sets a new benchmark for scalability.
Go 1.24 introduces new support for "Tools" , which allows easy consumption of tools (which are written in Go) as a dependency for a project. This could be anything from golangci-lint to protoc-gen-go . In this post, I will cover usage and limitations. Basic usage Adding a tool to a project is nearly the same as a standard runtime dependency, with the additional -tool flag: $ goimports # I don't…
Is it a per-node proxy? Is it a sidecar? The reality is somewhere in between
Reimagine service mesh with Istio’s ambient mode—lightweight, efficient, and scalable. Gloo Mesh now extends this innovation to multi-cluster environments, delivering unmatched reliability, simplicity, and scale.
A deep dive into the termination behavior of Istio ambient mode and how it impacts your applications.
How Istio ambient mode brings an enterprise-ready service mesh to Amazon ECS.
How ambient achieves massive scale without toil.
When we first started designing what eventually became Istio ambient mode , there were many directions we explored, both in terms of implementation, and what our goals were. What resonated most, though, was that we wanted to provide an incredibly easy onboarding story for a subset of functionality. This subset, ultimately, was getting Mutual TLS deployed for all service-to-service communication…
Over 2 years ago, I started working on some ideas to build better Kubernetes controllers. In this post, I wanted to give a bit of a retrospective on how things have gone since then. Over the years working on Istio and other projects, I observed a number of major issues with controllers: Most code was about error-prone event handling and state reconciliation, rather than business logic. Most tests,…
External Blog Post .
Tools to create reproducible development environments are basically everywhere these days, from Development Containers to Nix wrappers to questionable Docker hacks . However, all of these (that I have found) have a common flaw that bothers me: they all require eagerly fetching the entire environment to get anything done. This kills the premise of these environments providing any easy on-ramp for…
JetBrains IDEs (IntelliJ, GoLand, etc) have a nifty feature called Language Injection that lets you get full language features when a language is embedded within another. For example, a SQL query within a string within a Go file. A few of these come out of the box, but they are pretty limited -- I only had some XML ones prior to enabling the Databases plugin which added a few SQL ones.…
How Istio ambient mode greatly simplifies and improves egress gateway use cases.
How Gloo mesh brings HTTP observability to your mesh, without compromises.
An overview of options to deploy mTLS on Kubernetes
Core problems with the API make it a challenging to use in a secure, scalable manner.
Zero trust architectures should not treat nodes as highly privileged components.
Istio's installation has a long, winding, complex history, leading to an interesting current state . In this post, I hope to explain some of the historical context of how we arrived to the current state, and where I think the project is going. This is all my personal perspective and memory of things that happened years ago, so there is likely some divergence from reality. The Past When I first…
go run bash-script@latest
How we built a best-of-both-worlds experience with Istio ambient mode.
Installing Istio... hard could it be? A simple istioctl install is all you need... right?
Does Istio ambient introduce a SPOF? No.
How I iterate on Grafana dashboards
Sidecarless? Why not podless?
How Istio tests its networking proxy without Kubernetes, Docker, or root.
tl;dr: it just works
Like most other Kubernetes controllers in, Istio is written in Go and relies on the client-go library. While this provides an excellent low-level building block, usage in higher level code in Istio led to a variety of issues that led us to develop our own higher level, opinionated client for Istio. This post covers the issues we faced and how we incrementally solved them. Background knowledge At a…
Rage bait for YAML templating haters
Which features I recommend using, or not using, in Istio
In Analyzing Go Build Times , I went over how to analyze and understand Go build times, and what factors impact build times. A close cousin to build times is build sizes . Large binaries can lead to a variety of issues such as: Generally, slower build times Increased costs of storage Increased costs and time to distribute Increased memory usage at runtime (more on this in another article,…
Exploring an extreme service mesh architecture to maximize extensibility.
The OSI model attempts to build a model for network communications, where increasingly high level layers are built upon lower layers. This is only slightly useful in practice, as the real world is not so simple. In service mesh, generally discussion is reduced to L4 and L7, or TCP and HTTP. This oversimplifies the problem, leading to some confusion. Thinking in terms of termination Simply saying…