Let's Talk: AWS October Outage and AI Observability
The AWS us-east-1 outage taught me something important: with all their AI
investments, humans still had to save the day. Maybe that’s why Andy Jassy
fired 30,000 people.
LinuxCzar
Recent content on LinuxCzar
Latest posts
Resume
Jack Neely 
jjneely at gmail dot com 
 https://linuxczar.net 
 https://github.com/jjneely 
 Summary 
 Founder and Fractional CTO with 25 years of experience building planet scale,
multi-cloud observability platforms and distributed systems. Subject matter
expert in Site Reliability Engineering, Prometheus, Grafana, OpenTelemetry, and
AWS infrastructure. Proven…
Cardinality Cloud and Prometheus SLOs
I’d like to announce my new venture, Cardinality Cloud, LLC . I’ve always
wanted to take what I’ve done for hyper-growth companies like Palo Alto
Networks and Fitbit and make it more available to everyone. Readers of the
site will not be surprised that I’m specializing in Site Reliability
Engineering (SRE) and Observability consulting with a focus on cost…
Links

 Cardinality Cloud, LLC 
 
 Hire me! Or my consulting company at least. We specialize in SRE and
Observability consulting. We also are launching products around running
Prometheus and other Open Source Observability. Check it out! 
 
 Practical Operations Podcast 
 
 I podcast! Find me and friends on the Practical Operations Podcast where
ever you get your…
About

 
 I’ve had some form of website since 1996. Fortunately, most of what has been
on one of those websites has been lost to time. Really, that’s for the better.
Today, LinuxCzar is a place for technical musings, lessons learned, Open Source
projects (old, busted, or new and current), code snippets, and ideas in hopes
that it might be useful. Or that I might write…
Monitorama PDX 2023: Finding Π in Observability
Its been a long time since I’ve done a presentation in front of real live
people. I was honored to be selected to be a Speaker at Monitorama PDX
2023 that was held last week. I put together a talk on some of the
challenges that I, and many others, face in the quest to improve
Observability. I wanted to touch on some mathematical concepts that underline
choices that I…
Logging and Eventing with the SEARCH Method
When working with Software Engineering teams to improve their observability,
I often find that working with a method helps tremendously. A method with a
short and catchy acronym really drives the lessons home. Soon I’ve got
management writing notes about my acronyms and including them in planning
meetings. Methods have a winning madness. Now let’s use them for…
Prometheus Exemplars in Java Spring Boot
Java isn’t my favorite language to work in. However, I realized that to roll
out a successful Observability plan that I needed good examples and most of the
teams I work with create Spring Boot applications. So off I went to create a
simple Java Spring Boot application that demonstrated a structured logging
approach, metrics following the 4 Golden Signals pattern, and…
What Is Observability: A Practitioner's View
Charity Majors, from Honeycomb.io , recently wrote “ Observability: The
5-Year Retrospective .” In it Charity takes a look back, attempts to define
what Observability actually is, and lays out a set of capabilities any
Observability platform must have. Charity’s vision helps all of us understand
what we strive for in creating better software and services. However,…
Writing Change Management Announcements
The most important part of a Change Management process is simply being able to
tell the stakeholders about an upcoming change. Especially in a world where
email is often left unread, there is no one “centralized” group or team making
changes, and instant messenger is possibly the only real means of
communication. Ideally, a Master Station Log (MSL) or Service Status…
Twisted Edwards Curve SSH Keys
If you use SSH keys and haven’t migrated to the newer ED25519 Twisted Edwards
curve key pairs – well you should. It is presently the most recommended
key type. Faster and possibly more secure than RSA key types. Even though
this type has been supported by OpenSSH for a number of years now, there are
still some tricks to have up your sleeve.
Helm Chart Prometheus Rules
The best thing about using the Prometheus Operator to manage Prometheus in
Kubernetes is the CRDs. Alert rules can be managed directly by the application
Helm Charts, FluxCD, or your Cloud Native pipeline of choice. For me, for now,
that is Helm Charts. The only problem is that both Helm and the Prometheus
Alert Rules use Golang Text Templates . The question is: how does one…
Thinking About Keyboards, Part The Second
Its been quiet around LinuxCzar. Mostly because I’ve been thinking about
keyboards again. This line of thought closely follows career changes.
When opportunity knocks – as they say. 
 I first learned how to program in a computer lab full of IBM PS/2 Model 25
machines, each with its own SSK form factor Model M buckling spring keyboard.
I’m amazed at my memory…
Rules of a System Administrator
Over the years, I’ve developed some rules of thumb for systems administration
and operations. They serve me well and might serve others too.
Here they are in no particular order: 
 1. Invest In Openness 
 Most of my solutions involve Open Source Software or Open Standards. They
build the most scalable and rigorous infrastructure. Participating in the Open
Source…
AWS Kinesis Outage
On Wednesday, November 25, 2020, AWS suffered an outage of their Kinesis
service in the us-east-1 region. These RCA (Root Cause Analysis) write
ups are always illuminating. Not just as a peek inside how AWS services
works, but as a chance for the industry as a whole to peer review best
practices. I had a few key takeaways from my experience in Observability
that I want to point…
Finding the Golden Signals with Prometheus
I was honored to be selected to speak at All Things Open 2020 . I wanted
to tell the story of architecting Fitbit’s Prometheus and Thanos solution
for metrics and alerting. Including the many things I learned and that I
think are important to consider as a company scales out their observability
platform. Oddly enough, some of this also applies to handling logs and
events…
42 Lines Site Reliability Engineering
In October I had my first foray into webinars. The consulting
company I work with, 42 Lines, Inc. , gave me the opportunity to help put
together some marketing material. We’ve been providing services that we call
“DevOps Consulting” for a long time. However, I wanted to push forward with
the idea that what we’ve been doing may be best described today with…
Quick and Dirty Sockets
An axiom of programming usually found in the Operational, DevOps, and SRE
spaces is this: 
 
 Don’t write a socket server. 
 There’s good reason. There are a lot of edge cases to handle and, for lack
of a better phrase, magic sauce to be efficient and secure. Not to mention
there are a lot of libraries already written to do this for you. 
 However, this…
Calculating the Error of Quantile Estimation with Histograms
I’ve posted about Histograms before. They fascinate me with the prowess
they bring to recording telemetry about high volume events. The
representations on disk can be quite small, but they can be used to produce a
slew of highly accurate summary statistics. Histograms are both Robust 
and Aggregatable data structures and its rare to have a data type that combines
both of…
SRE: How to Count With Logs
Before an SRE can get into advanced anomaly detection or statistical models
this SRE must first bring the skills of recording events and being able to
count those events. If you are just starting this journey focus first on
gathering events from a small service. Gather these in the form of logs. 
 
 A log message is an ordered record of a unique event. 
 Of course, ordered…
What is a Site Reliability Engineer?
A Site Reliability Engineer (SRE) is a role within your company. Many of the
concepts this role embodies existed before it was popularized by Google.
However, Google did write the books and provide the methods we currently think
of for an SRE. Site Reliability Engineers work across many teams and
disciplines to bring about the DevOps culture we’ve been talking about for…
Open Observability: SRE Prometheus Tips
I was honored to be asked to help the Open Observability folks kick off
the first episode of their vidcast! This talk is an adaptation of the lighting
talk I did at Monitorama PDX 2019 with some expanded material for alerting
on Service Level Objectives (SLOs) with Prometheus. 
 
 
 
 Also, the slide deck I used is below. 
 
 
 A blog post featuring the…
A Site Reliability Engineer Series
Are you a Site Reliability Engineer? Perhaps your team is looking to start
an SRE journey or expand their resources on hand. Or maybe you are looking
to improve your skill set. We’re all engineers, and its about time we
actually use mathematics to prove our designs and practices. We didn’t get
to the Moon by winging it after mashing together some components into…
Prometheus Histograms Part 3: Using Something Else
Telemetry and monitoring with a Time Series Database (TSDB) is a set of 3
overlapping problem spaces. Those are: 
 
 Ingestion and Retrieval of Raw Data Points – Operational Intelligence 
 Aggregation and Roll Ups of Data for Long Term Retention and Trending
– Business Intelligence 
 Efficient Storage and Retrieval of High Velocity Event Data as…
TriLUG: Observability Through the Lenses of Metrics and Events
My friend Breandan and I presented at the local Linux Users’ Group 
on Thursday, September 12, 2019. Our presentation is called “Observability
Through the Lenses of Metrics and Events.” In a large part, its an effort
to build conversations around the differences between Metrics, Log Events, and
Tracing and why you would use each. 
 Of course, the subtext is…
Notes on TSDBs
When I was young and learning to program it was a milestone to write a Pizza
Menu and Ordering program. Select your pizza, crust, toppings, and see how
much that pizza is going to cost. Text mode or graphical and in a number of
languages I created this program over and over. 
 Today, its seems that the milestone to achieve is to write a Time Series
Database (TSDB). You…
Prometheus Alertmanager and Incident Keys
I run a centralized Alertmanager service for about 100 individual software
development and operations teams. Normally, each team has their own Prometheus
VM(s) and sometimes a dedicated Prometheus VM or two will be created for a
specific big or busy service. A single pair of Alertmanager instances scales
very nicely at this load with 100s of alerts firing. 
 However, after a…
Monitorama PDX 2019: 5 Neat Tricks with Prometheus
At Monitorama PDX 2019 I was honored to be selected to give a 5 minute
lighting talk. I figured there woud be no better time to show off some
neat things you can do with PromQL. 
 
 
 The talk recordings are up, and here is a direct link to the time offset in
the Lighting Talks where I spoke. Enjoy! 
 
 
 
 Monitorama PDX 2019 - Lightning Talks from…
Thanos and External Labels
I’ve been experimenting with Thanos over the last few months to add
long term storage of Prometheus metric data and a single query endpoint for
a large Prometheus fleet. This looks like a really well designed solution
to end the 30-ish day retention limits and to solve the oft asked question
“What host name is my Prometheus box?” 
 Thanos requires that each…
Generate Test Prometheus TSDB Data
This tool generates Prometheus 2.x TSDB blocks containing test data. If you
have ever wanted to test Prometheus or Thanos behavior on a specific
time series foot print, say a year’s worth of data, then this tool is for you.
Retro Thinking
nos-tal-gia ( noun ) – A sentimental longing or wistful affection for the
past, typically for a period or place with happy personal associations. 
 I’ve been thinking a lot about retro-computing. I am most definitely a child
of the 1980’s – the glory days of the personal computer revolution.
However, I was never a “gamer.” In fact, I used to…
Writing Documentation
Documentation is every IT professional’s job. Yet, it can be one of the most
overlooked areas of IT, DevOps, System Administration, Developer, or whatever
your professional title might be. To make writing (and reading) documentation
successful, make it easy to do. Create a set of stock templates that cover
most of your writing needs. This may be design documents for how a…
Moving to ZFS from Btrfs
Rule #4 states that failure will happen, therefore you should plan for the
eventual reality. The Linux workstations I build and use (if I have any say
about it) use at least 2 hard drives in some mirrored or otherwise redundant
fashion. My current patternis to build workstations with a small (120G or
there about) SSD drive as the boot drive that contains my OS install and…
How FreeBSD Secretly Won Open Source
If a job in Linux System Administration / Operations can teach you one
thing its how to keep up with the ever changing landscape that Open Source
is. I’ve been working with Linux for 20+ years, and with that comes some,
hopefully, wisdom of experience. Linux distributions, and Open Source are
divergent in terms of change. The more things change, the more things there
are…
Interview Questions for System Administrators
I originally wrote this post in October of 2008. Where did those 9 years go?
I think its time for an update. 
 As the Linux Czar, I’m regularly asked to interview folks that are applying to
various jobs that require some Linux skills. Interviewing isn’t really my
strong point and I always struggle to come up with good questions that
will lead the candidate to talk…
Histograms with Prometheus: A Tale of Woe
I’m a big fan of using histograms for
metrics and visibility. Over a StatsD-like approach that offers a series
of summary metrics, histograms give us the ability to: 
 
 Actually visualize the distribution. You can see if your distribution is
multimodal, for example. This is done with a heatmap. 
 Aggregation. You can aggregate histograms (with the same bucket…
Tenets of Microservice Monitoring
I was able to attend Monitorama PDX 2017 this past week and had a blast.
If you are interesting in monitoring, metrics, related data analysis, alerting,
and of course logs then this is the conference for you. It struck me at
Monitorama that many of us came of age in the pre-microservices (Service
Oriented Architecture or SOA) world. But services in a SOA environment…
Algorithms: Conflict Free Replicated Data Types
Designing systems at scale often means building some sort of “cluster.” These
are common and important questions to answer: 
 
 How is the load shared among many systems? A consistent hash ring of
sorts? Weighted round robin? 
 What trade offs make the most since for your problem? Consistency,
Availability, or resistance to Partitioning events ? 
 How is…
Prometheus and Histograms
One of the killer app features of Prometheus is its native support of
histograms. The move toward supporting and using histograms in metrics
and data based monitoring communities has been, frankly, revolutionary. I
know I don’t want to look back. If you are still relying on somehow
aggregating means and percentiles (say from StatsD) to visualize information
like latencies…
Bandwidth Delay Product and TCP Tuning
There are many factors that limit the available bandwidth of a network
link from point A to point B. Knowing and expecting something reasonably
close to the theoretical maximum bandwidth is one thing. However, the
latency of the link can vastly affect available throughput. This is
called the Bandwidth Delay Product. It can be thought of as the
“memory” of the link…
Installing Cyanite: A Scalable Graphite Storage Backend
I’ve been experimenting with Cyanite to make my Graphite cluster
more reliable. The main problem I face is when a data node goes down
the Graphite web app, more or less, stops responding to requests.
Cyanite is a daemon written in Clojure that runs on the JVM. The daemon
is stateless and stores timeseries data in Cassandra . 
 I found the documentation a bit lacking, so…
Buckytools Updates: Sparse Files, Restorations
I’ve updated Buckytools , my suite for managing at scale consistent
hashing Graphite clusters, with a few minor changes.
Go Vendoring with Git Subtree
In Go 1.5 we have the beginings of vendoring support . The easiest
way to incorperate other projects into your Git repo is by using the
following command. Short and dirty, but it works: 
 $ git subtree add --prefix vendor/gopkg.in/check.v1 \
 https://gopkg.in/check.v1 master --squash

Algorithms: Flap Detection
I’ve been researching quite a few algorithms for my client, Bruce, as I
continue to scale out certain systems. I thought that getting them on my
blog would be very useful for a future version of myself and many others. I
suspect, and hope, that most folks that work in Systems Administration /
Operations will find these at least familiar. 
 Flap Detection 
 Flap…
Analyzing or Replaying UDP Statsd Data
This is a update of an old post so its back at the top of the blog.
Original posting was 2015-08-19. 
 I’m considering swapping out Statsd with Bitly’s statsdaemon for
better performance. But, because Bitly’s version only accepts integer
data I wanted to analyze our Statsd traffic. I figured I’d use my friend
 tcpdump to capture some trafic samples and…
2015 Holiday Updates
Life is busy around the holidays with much family and new family additions.
If there is one thing I wish I could remind the world it is that Christmas
does last 12 days from Christmas Day until January 6th. So, Merry Christmas
and a Happy 2016 to all! Here are some updates in no particular order. 
 Google Jump Hash Support in Buckytools 
 It looks like my Graphite cluster will…
Prometheus and Graphite With a Dash of InfluxDB
I’ve been thinking about the “future” and how I can move my metrics system from
what I have now to what I’d like to have. I run a large Graphite cluster.
In the 26 million metrics per minute range with a quarter petabyte of
provisioned storage. I integrate with a Naemon (Nagios
fork) and Merlin setup for alerting. 
 I’ve been following Prometheus…
Scaling Graphite 3: Whisper Bugs
If you run Graphite at scale you are interested in applying
 this patch . 
 I was tracing performance issues in my Graphite cluster and saw that for some
queries the backend storage nodes were sending abnormally large pickle
objects back to the Graphite web frontends. Python’s httplib was taking
several minutes to download the pickle objects causing query times…
Scaling Graphite Part 2: Merging TimeSeries in the WebApp
The most difficult bit about running a Graphite cluster is handling queries or
graph rendering during a cluster rebalance. Or after a partitioning event when
you use replication in your consistent hashing cluster. Suddenly, graphs under
report, have partial data, or might even be completely different when you
reload the graph. Generally, your Graphite cluster becomes useless…
Scaling Graphite
This was written on request of one of my clients in June. 
 We use Graphite , Grafana , and Statsd extensively.
We monitor everything from basic server health, detailed MySQL debugging
information, web application transactions, and statistics about the
communication of users’ devices. The number of metrics we generate has grown
exponentially – and sometimes beyond…