As part of my development flow on Airflow I use virtualenvwrapper and a long time ago I set up the ability to automatically set environment variables whenever the virtualenv is activated, and to unset them when leaving. (I think this trick would probably work with a few small changes for …
I ve done a lot of work with Apache Airflow over the last two years, but my experience of how people use it is limited to the teams I ve worked on and a few glances through talks I ve seen. I have plenty of ideas of things I d …
Last week I went along to Breakfast Ops , a semi-monthly event aimed at CTOs and other technical decision makers, run by The Scale Factory , and held under Chatham House Rule which says When a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use …
A very common task when launching an instance in AWS is wanting to configure the instance in some way - maybe we want to install a few packages, download and launch the latest release of our project, or perhaps register with Chef or Puppet and continue configuring that way. One option …
Terraform is my preferred tool for automating cloud infrastructure deployments. It’s not without it’s problems but in balance I still prefer it over any other tool I’ve tried. When re-useable modules were added fairly early on in the development of Terraform (sometime before 0.5.0 in …