At its core, traditional IT Operations is overwhelmingly reactive - humans respond to incidents and tickets. There is so much noise! Alerts, tickets, audits, chat threads and more. What if ops could catch the small things before they turn into big problems? What if your ops team built tools that allowed them to be proactive? We post a new article every time one is ready. Subscribe to learn about how your team can adopt Proactive Ops!
You use a JetBrains IDE? That's your choice. Maybe you prefer Neovim. OK hacker. Maybe you just kept it simple and went with VS Code. Smart move. While I might judge your choice of IDE, I don't really need to know which one you use. The same goes for your operating system. You just do you, but keep it to yourself. Why does everyone want to include their IDE config in their .gitignore files? Do…
The title of today’s post is how AWS has left me feeling over the last few months. AWS claims the default quota for Anthropic’s Claude Sonnet 4.5 with cross region inference as 1000 requests per minute. I only get 50 RPM. For the newer Sonnet 4.6 model it is 500 and 25 respectively. Getting the advertised quota is challenging. Imagine this, you’re looking for a new car. You find one you like. The…
Triaging support tickets is very error prone work. Back in 2024 I became frustrated when my tickets were sent to the wrong team. This would often add a day to getting my issue resolved. I would later learn that the triage team only averaged around 72% accuracy. I was determined to find a way to automate this. Gata is the result. It is an open source ticket router that runs completely within your…
I love sharing my thoughts here, but sometimes someone else says it better. That's the case this week. I want to share an excellent post from fellow AWS Community Builder , Jakub Gaj . Last week, Jakub delivered a talk called Moving from Serverless to Functionless at AWS Dev Day Copenhagen . The talk wasn’t recorded, but the slides are available on Speakerdeck . The talk is based on a blog post of…
This week’s post is an excuse to write a note for my future self. Using immutable container registries and the docker build and push action always trips me up. My pipelines break because the action likes to push a latest tag. This is a post in 3 parts. In the first part I will try to convince you that immutable tags are a good thing. Then I will show you how to configure popular registries.…
In my recent AWS RDS Data API deep dive post , I looked at different ways the API could be called. One item I omitted was how to use the Data API in Terraform. Today we’re going to look at how we can use the RDS Data API to provision database users. PostgreSQL User Management Let’s start with some basic queries to create a new user in our PostgreSQL database. We don’t want our application to use…
In a recent post I explained why the AWS RDS Data API makes bastion servers redundant . In this post we will explore how to use the RDS Data API. The Data API allows you to run SQL queries against Amazon Aurora databases over HTTPS. No jump box needed. The API is available for both MySQL and PostgreSQL flavours of Aurora. In this post we will be using PostgreSQL. Many of the examples will work…
Bastion servers have been a staple of cloud environments. It’s time to remove them. In late 2025 there is no need to run a bastion server in an AWS environment. Origin of Bastions in Cloud Before we had bastions, teams would open up SSH on most if not all of their servers to the internet. If an engineer's home internet used dynamic IP addresses, IP restrictions were skipped. It wasn't uncommon for…
LLMs are only as good as their training data. When it comes to Terraform, much of the publicly available code is pretty bad. This results in bots generating poor quality Terraform. It gets worse. Researchers have warned about model collapse . This is a very real problem with Terraform. LLMs generate bad Terraform, engineers who don’t know any better publish the code, this becomes training data.…
Some days are rough for customer support agents. A customer opens a support case. The problem described sounds unusual. You’re the unlucky support agent assigned to the ticket. The first action is to search for similar incidents. Your eyes start to glaze over as you read through all the details of half a dozen or so tickets. This is so boring. Internal discussion between teams. The back and forth…
AWS offers 3 ways of controlling network egress - Network ACLs , Security Groups and the Network Firewall . Security groups rely on static IP addresses, but that isn't how the modern web works. Amazon's Network Firewall is powerful, but at 290-1000+USD per month plus traffic, that option is expensive. Modern applications need a middle ground. Without a middle tier startups and smaller business…
My favourite AWS PreInvent 2024 announcement was JSONata support in Step Functions . I quickly converted a lot of my Step Functions to use the new JSONata syntax. I was able to eliminate a lot of data transformation steps. The old JSONPath implementation was frustratingly limited. Everything had to be merged into a global object that was carried from state to state. Extracting data from results…
Building integrations and workflows is often a repetitve and error prone task. You start by identifying the endpoints you need to call. Before writing the client, you have to wade through sometimes poorly written docs. Then it is time to find and fix bugs, before eventually launching it. As we approach the end of 2024, there needs to be an easier way. Code Generation Over the last couple of years…
Managing infrastructure using code is accepted best practice in the cloud. Terraform is the most popular cross cloud framework for infrastructure as code, but it can present challenges when dealing with resources that are updated by external processes. This is particularly true for AWS Systems Manager (SSM) Parameters when parameters are modified by other applications or workflows. In this…
Have you heard about Proactive Ops, but don’t know where to start? Do you know all the tools and services you use across your environment? A Proactive Ops platform needs to collect events from all your tools, listing them is a great place to start. Review all the phases of your software lifecycle, identifying all the tools and services you use. What version control platform do you use? What…
Several times I’ve presented a talk entitled Proactive Ops: Event Driven IT Operations . Ops is already event driven - something goes wrong and a human tries to fix it. Proactive Ops is about moving beyond being reactive. In this post, we will explore why you should run your Proactive Ops platform on a serverless stack. These ideas can be applied on AWS or other cloud platforms. If you really…
Proactive Ops is all about using events to identify issues and prevent incidents. It involves building event driven tooling for IT Operations teams. 2024 should be the year of Proactive Ops for your organisation. If you’re a small team, Proactive Ops helps you do more with less - automate those time sinks! Those few minutes each day add up to a lot of hours over a year. More importantly, the…
Rules are mostly made to be broken and are too often for the lazy to hide behind. ― Douglass Macarthur There is merit in Macarthur’s often butchered quote about rules. Some rules are made to be broken. Other rules are made for good reasons - such as security and privacy policies. For policies to be effective, they need to be enforced. There is no point in having policies that are ignored.…
For many years I used Makefiles as an abstraction layer in my CI/CD tooling. make allows the same sequence of actions to be run both by the CI pipeline and developers locally. This is great for consistency and provides a fast feedback loop for the team. Using Makefiles as an abstraction has saved me a tonne of effort when migrating between CI/CD platforms on short notice. It was as easy as…
This week I’m going to keep things short. Over the last few weeks I’ve been reflecting on Proactive Ops. I am going to move to publishing fortnightly. This gives me more opportunities to spread the word about proactive ops on other platforms. If you want to read an article on something related to Proactive Ops this week, I recommend Chris Allen’s Serverless Toolbox post . This is a good example of…
One of the key activities in Proactive Ops is analysing events to identify potential issues and remediating them without human intervention. These events need to be collected and routed to handlers. If you are building your Proactive Ops platform on AWS, Amazon’s EventBridge (nee Amazon CloudWatch Events) is a foundational component of your tech stack. Rather than bore you with all the detail of…
Traditional thinking in application development is there should always be at least 3 environments - dev, stage and prod(uction). Some teams or organisations use different labels, but the purpose of each environment is the same. For over 5 years, my teams have used two environments with over 100 micro services. These environments are strictly segregated so dev can’t access production data or…
experiment (noun): an operation or procedure carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law Source: https://www.merriam-webster.com/dictionary/experiment Experiments are a great way to try new things. To get the most out of experiments, teams need to apply some rigour to ensure accurate results.…
Many teams aspire to building out a collection of loosely coupled domain driven microservices. Some get there, while others struggle. Engineers spend countless hours defining the boundaries of their contexts, designing models and mapping out their endpoints. One area that often doesn’t get the attention it deserves is event design. Events are how services communicate. We have established standards…
Pipes are a key component of Unix. Pipes pass text streams from one process to the next. This allows users to execute a series of commands where the output of one command becomes the input of the next. Pipes can help you find the 5 processes consuming the most CPU time. They can also fetch a webpage and output all the external links in alphabetical order. The possibilities are almost endless. Back…
This post is the first in a series that will explore the technical foundations of an automation platform that supports Proactive Ops . This series will primarily focus on AWS, but the concepts can be applied to other cloud platforms. Monolithic web apps often feature a REST API bolted onto the side. When first adopting serverless patterns, teams often implement API first micro services. Using this…
At its core, traditional IT Operations is overwhelmingly reactive - humans respond to incidents and tickets. There is so much noise! Alerts, tickets, audits, chat threads and more. What if ops could catch the small things before they turn into big problems? What if your ops team built tools that allowed them to be proactive? A Proactive Ops team is an IT Ops team engaged in software engineering.…