Recent content on Andy Dote
Earlier in my career I was working as a C# developer, and had read a lot about testing and testability from the Alt.Net crowd, a lose group of people with ideas outside of how Microsoft did things in C#. When I came to needing to write an API, the impossibility of testing in ASP.NET made me look elsewhere. ASP.NET made heavy usage of Annotations, where you add decorators to methods which at…
Back in a previous company (the same place as where Debugging GDI Handle Leaks happened), there was an interesting convention in the database: all tables had a unique 3-letter prefix assigned to them, and all columns in that table must start with the prefix, which I’ve written about before. For example, a person table would have the prefix PEO, and the columns would be PEO_PersonID,…
A system I am working on at the moment started giving errors occasionally, say 5 times out of 10,000 messages or so. The error was pretty straightforward: json: cannot unmarshal array into Go struct field Thing.Parts of type Parts The data structure it is referring to looks like this: type Thing struct { Parts Parts } type Parts struct { Part []Part } Which represents the (slightly weird) json…
Many years ago, I was working on a dotnet Windows Forms application. The application had many issues overall: memory leaks, random crashes, data loss, and in this case, the “red x” problem. The problem showed up at random, and instead of a window, dialogue, or control being rendered, it would be replaced with a white box with a red outline and red diagonal cross, and I think some error…
I use opentelemetry extensively to trace my applications, and one thing I keep running into is when writing a long running process, I want to handle OS signals and still send the telemetry on shutdown. Typically, my application startup looks something like this: func main() { ctx, cancel := context.WithCancel(context.Background()) handleSignals(cancel) tracerProvider := configureTelemetry(ctx)…
Following on from my previous post about not having too much configuration, I want to talk about how I design software. I try and follow what I call “outside in design”; I try and make something that requires the bare minimum amount of configuration to cover the most common of use-cases. Once this functionality is working, further configuration can be added to cover the next most…
This is my thoughts after reading Don’t Look Down on Print Debugging. TLDR Print debugging is a tool, and it has its uses - however, if there are better tools available, maybe use those instead. For me, a better tool is OpenTelemetry tracing; it gives me high granularity, parent-child relationships between operations, timings, and is filterable and searchable. I can also use it to debug…
When writing software, you will come across many questions which don’t always have a clear answer, and its tempting to not answer the question, and provide it as a configuration option to the user of your software. You might go as far as setting a default value though. This seems good for everyone; you don’t have to make a decision, users can change their minds whenever they want.…
A question at work came up recently about how to handle multiple errors in a single span. My reaction is that having multiple errors in a span is a design smell, but I had no particular data or spec to back this up, just a gut feeling. I’ve since thought about this more and think that in general, you should only have one error per span, but there are circumstances where multiple indeed makes…
In one of my too many side projects, I am using htmx and go templates to render a somewhat complicated web UI. I much prefer using htmx for this kind of thing rather than react, as react brings in so much more additional complexity than I need or want. However, there is one thing I miss from the React ecosystem, and that is hot reload. Being able to save a file in my editor and see the changes…
It is no secret that I am not a fan of logs; I’ve baited (rapala in work lingo. Rapala is a Finnish brand of fishing lure, and used to mean baiting in this context) discussion in our work chat with things like: If you’re writing log statements, you’re doing it wrong. This is a pretty incendiary statement, and while there has been some good discussion after, I figured it was time…
One of the many reasons given for using microservices rather than a mono repository is that it enforces boundaries between services/modules. However, there are ways to achieve strong boundaries between modules/services in one repository, using tools which are already available: test runners. Given a repository with the following structure: . ├── libraries │ ├── core │ ├── events │ └── ui ├──…
Tracking where the time goes in your CI pipeline is an important step towards being able to make it go even faster. Up until somewhat recently, the only way of tracking how long tasks took in CI was either hoping people had wrapped all their commands in time ..., or by reading a timestamped build log and calculating the difference between numbers. Which isn’t great or fun, if we’re…
I haven’t seen Expand-Contract written about in some years, and I think it is a great way of performing database schema migrations without the need for application downtime. I also realised that it also applies to microservices and service-to-service communication in general. The Easy Example One of the two examples given is wanting to change how an address is stored in a database. The…
Feature flags are a great tool for helping software development; they provide controlled feature rollouts, facilitate A/B testing, and help decouple deployment from release. So when it comes to building our software, why do we treat the CI pipeline without the same level of engineering as the production code? So, why not use feature flags in your CI pipeline? TLDR Reduce the risk of breaking a CI…
Versioning is one of the many hard problems when it comes to writing software. There is no one correct way to do it, and all have various tradeoffs. After reading keep a changelog, I was inspired to implement this into a couple of CLI tools that I am working on at the moment: Flagon (feature flags on the CLI, for CI usage), and Cas (Content Addressable Storage for Make), but I also wanted to solve…
When migrating to a continuous delivery process, it is often the case that a QA team are worried about what their role is going to be, and how the changes will affect the quality of the software in question. While doing continuous delivery does change the QA process, when done well, it improves everyone’s lives and makes the software better quality. Are silver bullets incoming? Not quite,…
Recently we were having a debate about release processes, and I wrote that deployments are not always equal to releases. also deploy != release —Andy, baiting discussion in Slack This turned out to be somewhat controversial until we discussed what I specifically meant by deploy and release. As with all things, agreeing on definitions or understanding what someone means when they use a specific…
On several occasions when building complex projects, I have been tempted to set up Bazel to help speed up the build process; after all, it has a lot to offer: only building what has changed, caching built artifacts, and sharing that cache between machines for even more speed. TLDR We can use Make and a couple of short shell scripts to implement file content-based caching and read/write that cache…
I write a lot of blog posts which contain code snippets. Like most people, I have done this using a fenced codeblock in markdown which is fine for short blocks of code. The problem occours when I am embedding code from another repository; often there will be tweaks or bug fixes, and keeping the code in the blog post in sync with the code in the other repo is annoying and manual.
This is in response to the recent posts about the death of UML; while I think some parts of UML have fallen ill, the remaining parts are still alive, and useful to this day. TLDR Out of 14 types of diagram there are 3 that I use on a regular basis: Activity Diagram, State Machine Diagram, and Sequence Diagram. I think the Timing Diagram is borderline, but I can only think of a couple of occasions…
One of the reasons I prefer Pulumi over Terraform is the additional control I have over my processes due to the fact that it’s a programming language. For example, I have a CLI, that creates a cluster of machines for a user; the machines use IAM Authentication with Vault so that they can request certificates on boot. The trouble with this application is that it is slow; it takes 175 seconds…
I recently encountered a behaviour in Nginx that I didn’t expect and caused a production outage in the process. While I would love to blame DNS for this, as it’s usually the cause of most network-related issues, in this case, the fault lies with Nginx. I was running a very simple Nginx proxy, relaying an internal service to the outside world. The internal service is behind an AWS ALB,…
The Operator Pattern from Kubernetes is an excellent way of handling tasks in a cluster in an automated way, for example, provisioning applications, running backups, requesting certificates, and injecting chaos testing. As a Nomad user, I wanted to do something similar for my clusters, so I set about seeing how it would be possible. It turns out; it is much easier than I expected! While Nomad…
An interesting question came up at work today: how do you tag your Docker images? In previous projects, I’ve always used a short git sha, or sometimes a semver, but with no great consistency. As luck would have it, I had pushed for a change in tagging format at a client not so long ago as the method we were using didn’t make a lot of sense and, worst of all, it was a manual process.
Key Takeaway: os .cpus() returns the number of cores on a Kubernetes host, not the number of cores assigned to a pod. Investigating excessive memory usage Recently, when I was looking through a cluster health dashboard for a Kubernetes cluster, I noticed that one of the applications deployed was using a considerable amount of RAM - way more than I thought could be reasonable. Each instance (pod)…
One of the things I like to do when setting up a Vault cluster is to visualise all the operations Vault is performing, which helps see usage patterns changing, whether there are lots of failed requests coming in, and what endpoints are receiving the most traffic. While Vault has a lot of data available in Prometheus telemetry, the kind of information I am after is best taken from the Audit…
I had the need to get some OpenTelemetry data out of a NodeJS application, and into NewRelic’s distributed tracing service, but found that there is no way to do it directly, and in this use case, adding a separate collector is more hassle than it’s worth. Luckily, there is an NodeJS OpenTelemetry library which can report to Zipkin, and NewRelic can also ingest Zipkin format data. To…
This article was originally published on the Pulumi blog. When using the Pulumi Automation API to create applications which can provision infrastructure, it is very handy to be able to use observability techniques to ensure the application functions correctly and to help see where performance bottlenecks are. One of the applications I work on creates a VPC and Bastion host and then stores the…
Following on from my last post on Isolated Multistage Docker Builds, I thought it would be useful to cover another advantage to splitting your dockerfiles: building different output containers from a common base. The Problem When I have an application which when built, needs to have all assets in one container, and a subset of assets in a second container. For example, writing a node webapp, where…
Often when building applications, I will use a multistage docker build for output container size and efficiency, but will run the build in two halves, to make use of the extra assets in the builder container, something like this: docker build \ --target builder \ -t builder:$GIT_COMMIT \ . docker run --rm \ -v '$PWD/artefacts/tests:/artefacts/tests' \ builder:$GIT_COMMIT \ yarn ci:test docker run…
I write a lot of bash scripts for both my day job and my personal projects, and while they are functional, bash scripts always seem to lack that structure that I want, especially when compared to writing something in Go or C#. The main problem I have with bash scripts is that when I use functions, I lose the ability to log things. For example the get_config_path function will print the path to the…
Recently, I noticed that when we pull a new version of our application’s docker container, it fetches all layers, not just the ones that change. The problem is that we use ephemeral build agents, which means that each version of the application is built using a different agent, so Docker doesn’t know how to share the layers used. While we can pull the published container before we run…
When it comes to implementing a new feature in an application’s ecosystem, I don’t like spending my innovation tokens unless I have to, so I try not to add new tools to my infrastructure unless I really need them. This same approach comes when I either want, need, or have been told, to implement a Service Mesh. This means I don’t instantly setup Istio. Not because it’s bad…
Before I start on this, I want to make it clear that if you can buy Honeycomb, you should. Outlined below is how I started to add observability to an existing codebase which already had the ELK stack available, and was unable to use Honeycomb. My hope, in this case, is that I can demonstrate how much value observability gives, and also show how much more value you would get with an excellent tool,…
One of the many features of Nomad that I like is the ability to run things other than Docker containers. It has built-in support for Java, QEMU, and Rkt, although the latter is deprecated. Besides these inbuilt “Task Drivers” there are community maintained ones too, covering Podman, LXC, Firecraker and BSD Jails, amongst others. The one I want to talk about today, however, is called…
I noticed when running an Alpine based virtual machine with Consul DNS forwarding set up, that sometimes the machine couldn’t resolve *.consul domains, but not in a consistent manner. Inspecting the logs looked like the request was being made and responded to successfully, but the result was being ignored. After a lot of googling and frustration, I was able to track down that it’s down…
I use Vagrant when testing new machines and experimenting locally with clusters, and since moving (mostly) to Linux, I have been using the LibVirt Plugin to create the virtual machines. Not only is it significantly faster than Hyper-V was on windows, but it also means I don’t need to use Oracle products, so it’s win-win really. The only configuration challenge I have had with it is…
I will update this post as I learn more (both positive and negative), and is here to be linked to when people ask me why I don’t like Kubernetes, and why I would pick Nomad in most situations if I chose to use an orchestrator at all. TLDR: I don’t like complexity, and Kubernetes has more complexity than benefits. Operational Complexity Operating Nomad is very straight forward. There…
So we want to set up a Vault instance, and have it’s storage be a TLS based Consul cluster. The problem is that the Consul cluster needs Vault to create the certificates for TLS, which is quite the catch-22. Luckily for us, quite easy to solve: Start a temporary Vault instance as an intermediate ca Launch Consul cluster, using Vault to generate certificates Destroy temporary Vault instance…
I was recently using my Hashibox for a test, and I noticed the DNS resolution didn’t seem to work. This was a bit worrying, as I have written about how to do DNS resolution with Consul forwarding in Ubuntu, and apparently something is wrong with how I do it. Interestingly, the Alpine version works fine, so it appears there is something not quite working with how I am configuring…
This post is going to go through how to set up a Consul cluster to communicate over TLS. I will be using Vagrant to create three machines locally, which will form my cluster, and in the provisioning step will use Vault to generate the certificates needed. How to securely communicate with Vault to get the TLS certificates is out of scope for this post. Host Configuration Unless you already have…
Often when developing or testing some code, I need (or want) to use SSL, and one of the easiest ways to do that is to use Vault. However, it gets pretty annoying having to generate a new CA for each project, and import the CA cert into windows (less painful in Linux, but still annoying), especially as I forget which cert is in use, and accidentally clean up the wrong ones.
This is a text version of a short talk (affectionately known as a “Coffee Bag”) I gave at work this week, on Architecture Design Records. You can see the slides here, but there isn’t a recording available, unfortunately. It should be noted; these are not to replace full architecture diagrams; you should definitely still write C4 Models to cover the overall architecture. ADRs are…
I wanted to implement canary routing for some HTTP services deployed via Nomad the other day, but rather than having the traffic split by weighting to the containers, I wanted to direct the traffic based on a header. My first choice of tech was to use Fabio, but it only supports routing by URL prefix, and additionally with a route weight. While I was at JustDevOps in Poland, I heard about another…
One of the points I make in my Feature Toggles talk is that you shouldn’t be querying a toggle’s status all over your codebase. Ideally, each toggle gets checked in as few places as possible - preferably only one place. The advantage of doing this is that very little of your codebase needs to be coupled to the toggles (either the toggle itself or the library/system for managing toggles…
Recently, I was asked if I could provide an example of Branch By Abstraction when dealing with feature toggles. As this has come up a few times, I thought a blog post would be a good idea so I can refer others to it later too. The Context As usual, this is some kind of backend (micro)service, and it will send email messages somehow. We will start with two implementations of message sending: the…
DEPRECATED - This has a race condition! Please see this post for an updated version which works! Following on from the post the other day on setting up DNS forwarding to Consul with SystemD, I wanted also to show how to get Consul up and running under Alpine Linux, as it’s a little more awkward in some respects. To start with, I am going to setup Consul as a service - I didn’t do this…
DEPRECATED - This doesn’t work properly Please see this post for an updated version which works! One of the advantages of using Consul for service discovery is that besides an HTTP API, you can also query it by DNS. The DNS server is listening on port 8600 by default, and you can query both A records or SRV records from it. SRV records are useful as they contain additional properties…
Last time I wrote about running a RabbitMQ cluster in Nomad, one of the main pieces of feedback I received was about the (lack) of security of the setup, so I decided to revisit this, and write about how to launch as secure RabbitMQ node in Nomad. The things I want to cover are: Username and Password for the management UI Secure value for the Erlang Cookie SSL for Management and AMQP As usual, the…