To manage migration processes such as adopting a new data integration tool or rehosting applications at scale from one infrastructure service to another, organizations often turn away from established work planning and management tools such as Azure DevOps Boards . Instead, project managers often develop their own systems based on the trusted spreadsheet, typically Microsoft Excel. While these…
In most systems I come across, private keys are all over the place, made available as secrets to the workloads that need them. The trouble is not just that sometimes, secrets are not really secret , but more fundamentally, that private keys are something that we don't want to handle directly at all, they're basically too hot to handle 🔥. Instead, we can use a remote service to manage our private…
In Why is everyone trying to kill Airflow , Daniel Beach argues that the demise of popular open-source orchestrator Apache Airflow is exaggerated, "calling no-go on the doomsday proclaimers who are out there peddling the end of Airflow as we know it is near." He argues that while there are lots of things to be unhappy about (read the article for a long list of those), it's fundamentally a good…
The ingress controller supported by the Kubernetes project itself is nginx . And while there are recipes for setting up automated issuing of TLS certificates using free CAs such as Let's Encrypt , there are quite a few steps involved and you will need to deploy additional services to your cluster to make it work. Meanwhile, the ingress controller for Caddy does it fully-automated, out-of-the-box.…
Inspired by Mike Cartmell's Budget Kubernetes Hosting for Personal Use blog series, I wanted to set up my own Kubernetes cluster on DigitalOcean on a budget (although I have since moved to Scaleway ). There was one component Mike wasn't able to skimp on – the container registry . DigitalOcean charges $5/month for a basic plan of their container registry product which has a storage of 5 GiB. Not…
Apache Airflow is an open-source platform that allows you to programmatically author, schedule and monitor workflows. It comes with out-of-the-box integration to lots of systems, but the adage that the devil's in the details holds true with integration in general and remote execution is no exception – in particular PowerShell Remoting which comes with Windows as part of WinRM (Windows Remote…