Recent content on Hello internet
It’s another Vector post! As with the last, I won’t be a post describing Vector’s basic usage (Vector’s docs do that pretty well! ) but in short, Vector is an extremely competent tool for building observability pipelines. This post is about a neat trick you can do with its log namespacing feature, which effectively splits the log event into the log body . and its metadata,…
Vector is for building observability pipelines. It can work as an agent/sidecar too, but where it really shines is as an aggregator . Funnel a bunch of stuff into it and you can use it apply standard structures, annotations, and centralize governance. This won’t be a post describing its basic usage (Vector’s docs do that pretty well! ) but rather a fun exploration of a failure scenario…
One of the earliest documented uses of the check valve is in the myth of Pandora’s box. This is also one of the earliest documented cases of Lorin’s Law due to the check valve being installed backwards. The cat’s out of the bag (or box) as they say, and that brings us to today’s topic: LLMs , a class of things which seems unlikely to flow back into Pandora’s box. A…
It’s been a fun rabbit hole. Maybe I could have saved myself a lot of trouble by paying attention to this throw-away line about the JWT header emitted when using OIDC Authentication on an Application Load Balancer : Standard libraries are not compatible with the padding that is included in the Application Load Balancer authentication token in JWT format. JWTS are neat and this blag post…
It’s been a hot minute. One of my favourite concepts from Normal Accidents is “production pressures.” The gist of it is that safety devices are sometimes counterbalanced by socio-economic pressures to do more or go faster. Charles Perrow describes a literal example of going faster with the introduction of shipboard radar: intended to improve maritime safety, it instead was used…
I think many AWS users first encounter the KMS service as a way to encrypt other AWS resources. It has a really good usability story for things like block storage: here’s a managed encryption service where you enable a checkbox at instance launch and now your disk is encrypted. The resulting encrypted snapshots are no use if exfiltrated without access to the KMS key. That’s why what I…
(Short version: there’s more context below, but for readers who just want to know which AZ doesn’t support Nitro, it’s use1-az3 .) What this is about Link to heading Not all AZs support all instance types Availability zone names aren’t consistent across AWS accounts Availability zone IDs are consistent across AWS accounts! How to find out which zone IDs support a given…
AWS Gateway VPC endpoints are so good that Amazon only ever shipped two. This is a short post so I’m not going to go into exhaustive detail about what they are, but inexhaustively: they help to keep traffic inside a VPC. See, if you can attach an ENI to a resource, then that resource can be inside a VPC, and two ENIs inside the same VPC can chat with each other without ever hitting the…
Stop me if you’ve heard this one before: you have a use case for Postgres logical replication and a few minutes/hours/days after enabling it, PagerDuty is yelling that the database’s disk is dangerously full and would you please do something about it before bits start spilling out onto the datacentre floor? The most common cause for logical replication consuming disk (and which this…
(Author’s note: times sure have changed! 2018 was a different time. At this point you should really be doing some kind of federation/SSO!) Earlier this year I had the opportunity to present a short talk at the AWS Toronto Summit in 2018. While the recording is now available for viewing , I’ve had several requests for a blag version, which is precisely what this post is all about. The…
Caveat: Consul’s ACL system was revamped in November of 2018 so this comparison is only valid for Consul prior to version 1.4.0. In particular, Consul’s token system in >= 1.4.0 has a strong resemblance to Nomad’s. One of the more interesting things about Nomad’s ACL system is the ways in which it differs compared to Consul. ACLs came to Consul in September of 2014 and to…
Wealthsimple’s production environment is powered by hundreds of Docker containers. We have lots of reasons to like containers beyond just their popularity: Containers give us a homogenous way to deploy heterogenous apps Our servers can all be configured uniformly since there are no app dependencies to install We can choose (or even change) our Linux distro without breaking the…
Cloud wrangler, distributed systems-er, finder-of-deadlocks, telemetry amplifier. I lead high-performing, cross-functional engineering teams tackling operations-shaped problems.