direct from the mysterious land of the sysadmin
Since the first DevOpsDays in Ghent in 2009, the community has organised well over 700 events across every populated continent. Here's what the shape of that history looks like. The long arc, 2009–2026 Events grew steadily from a handful in the early years to a peak of 81 in 2020 (yes, the pandemic year). The virtual format lowered the barrier to entry: no venue costs, no travel logistics, no…
If you install software from a public package registry (npm, PyPI, Crates, etc.), every security decision that registry makes sits inside your trust boundary. You don't run the infrastructure, you don't set the publisher policies, you don't decide what "verified" means, and whatever the registry (fails to) enforce travels straight into your build. This is a well-studied surface.…
(This post originally appeared on dev.to ) Here's how to keep your clone of a Git repository current with upstream quickly and efficiently. Just pop this snippet into the .git/config of your local checkout and you're good to go (some assembly required). [ remote " upstream "] url = < http ur l > fetch = +refs/heads/ * :refs/remotes/upstream/ * [ alias ]…
Hi there! I'm Dan. Over the past few years I've done a fair number of presentations, talks, panels, and other appearances at technical conferences and shows. You can check out the complete list of content—including videos, slide decks, and links to additional resources where available—at https://speaking.dark.ca/ . Thanks, and as always, please enjoy responsibly!
I recently received a really great question from a Datadog customer that I'd like to share concerning Observability as Code—specifically managing dashboards via Terraform in this case, but the question and answer apply more broadly. While we use Terraform for 100% of our Datadog monitors, we've found [that] developers are less likely to update dashboards when they see a mistake in the…
The sixth edition of DevOpsDays Berlin was held over the 12th and 13th of September 2018, and despite the fact that it's one of the most consistent, longest-running Devopsdays events on the calendar, this year was my first ever opportunity to attend. The tl;dr is that this was a very fine event, and the local organising team clearly has their act together (though they are actively recruiting…
I joined Mozilla nearly five years ago, which was an interesting period culturally for the company - we were transitioning from being a scrappy, disorganised group of hackers into a proper corporation with a global reach. One of my first bugs was to audit our load balancers - another was to help fix the tap in the office beer keg. It was an interesting time, fraught with mistakes, but heady with…
Last year, my esteemed colleague JP Schneider and I were invited to keynote at a couple of conferences last year. We gave two variations on the same talk, entitled "Genesis: Terraforming a New Home for Firefox Crash Reporter", at each of Hashiconf 2015 and Velocity EU 2015 respectively. The blurb for these talks is as follows: Everyone loves talking about different stacks used in migrating…
Terraform is a Hashicorp tool which embraces the Infrastructure as Code model to manage a variety of platforms and services in today's modern, cloud-based Internet. It's still in development, but it already provides a wealth of useful functionality, notably with regards to Amazon and Digital Ocean interactions. The one thing it doesn't do, however, is manage pre-existing…
The annual Sysadvent series of blog posts is well under-way and this year features, for the first time ever, an entry authored collaboratively by myself and Jonathan Clarke entitled « What System Administrators Can Learn From Community Management ». This post was borne from notes collected over numerous Devops Paris meet-ups and other local community get-togethers, as well as from the personal…
Devopsdays Belgium 2014 was held over 27 and 28 October in the charming town of Ghent (or Gent, or Gand), Belgium. This event marked the five-year anniversary of the Devopsdays series of conferences - and with such an important milestone in play, expectations were running high. I'm happy to report that those expectations were met. First off, the basic details: The Devopsdays format is…
I recently helped organise the inaugural Paris edition of the popular DevOpsDays series of conferences. It was a great event, and I'd like to share some of my thoughts and observations here. The DevOpsDays series of events is, as the name implies, centred around the "devops" movement, and is intended as a way to introduce people to this style of IT workflow, project, and people management.…
Today roidrage and I hosted the inaugural European edition of the popular Hangops podcast / hangout sessions. Hangops.eu (as we've taking to calling it) is functionally the same its North American counterpart - with the exception that "our" 11:00 is that of Western Europe, not California. ;) If you're not familiar with the Hangops, now's the time for you to get with the…
As part of a big Logstash project at Mozilla (more on that to come), I was looking for an HAProxy module for Puppet, stumbling across the official Puppetlabs module in the process. I'm told that this module works fairly well, with the caveat that it sometimes outputs poorly-formatted configuration files (due to a manifestly buggy implementation of concat ). Furthermore, the module more or…
Linux Conf Australia 2013 , (alias linux.conf.au, alias LCA) was an amazing experience, and it's difficult to summarise it succinctly, but there are definitely some highlights and important take-aways that I'd like to make special note of. My fundamental reason for attending was that Ben Kero and I were invited to do a talk regarding Puppet (an important infrastructural tool that we use…
Hello friends, Recently, Ben Kero (a fellow Mozillian) and I were invited to present a talk at Linux Conf Australia . To say that we were excited about presenting at one of the best Libre / Open Source conferences in the world is an understatement. We knew that we'd have to bring our A-game, and in all modesty, I like to think that we did. If you were there in person, I'd like to…
Hello everybody ! If you're interested in learning a little about Puppet at Mozilla, I highly recommend that you check out Puppet Podcast #7 , where Brandan Burton ( @solarce ) and I ( @phrawzty ) talk with Puppet Labs' Mike Stahnke ( @stahnma ) about just that topic. It was our first podcast together, and frankly, it was more difficult that I thought it'd be. That said, it was a…
Hi everybody ! Here's a quick method for generating encrypted passwords that are suitable for things like /etc/passwd . I realise that this isn't terribly complex, but honestly, I always forget how to do this until I actually need to do it - so here's a reminder for all of us. :) #!/bin/bash if [ " x $1 " == ' x ' ] ; then echo " USAGE: $0 'password' " exit 1 fi…
Update : This is an old blog post and is no longer relevant as of version 1.x of Elasticsearch. Now we can just use the snapshot feature. Hello again! Today we're going to talk about backup strategies for Elasticsearch . One popular way to make backups of ES requires the use of separate ES node, while another relies entirely on the underlying file system of a given set of ES nodes. The…
N.B. This post is from 2011 - the landscape has changed since then... Centralised cloud-based logging. It sounds tasty - and it is - but who should you go with? Well, Loggly and Papertrail are the only games in town when it comes to the aforementioned service; the only other competitor in this space is Splunk Storm , but their offering - well-pedigreed though it may be - is strictly in private…
When it comes to code hosting on The Internets today, GitHub is absolutely the hottest, trendiest service going - but it's not alone. Right now, the primary direct competitor to GitHub is Bitbucket , and choosing the best service for you or your company can be a less than obvious scenario - so let's break it down, shall we? GitHub is generally considered to be the most popular code…
Datastax recently announced the availability of Cassandra 1.0 (stable), and along with that announcement, they made a series of blog posts ( 1 , 2 , 3 , 4 , 5 ) about many of the great new features and improvements that the current version brings to the table. For those of you looking for an executive summary of those posts, you're in luck, cause I've got your back on this one. New…
Update 2015-10-07 : This plugin has evolved - please check the latest README for up to date details. Hello all ! I wrote a plugin for Nagios that will parse JSON from an HTTP response. If that sounds interesting to you, feel free to check out my check_http_json repo on Github. The plugin has been tested with Ruby 1.8.7 and 1.9.3. Pull requests welcome ! Usage: ./check_http_json.rb -u -e -w -c…
Hello all, I wrote a rudimentary RabbitMQ plugin for Collectd. If that sounds interesting to you, feel free to take a look at my GitHub . The plugin itself is written in Python and makes use of the Python plugin for Collectd. It will accept four options from the Collectd plugin configuration : Locations of binaries : RmqcBin = /usr/sbin/rabbitmqctl PmapBin =…
Hello everybody ! Today's post is about the Distributed Numeric Assignment (or « DNA » ) plug-in for the 389 Directory Server (also known as the Fedora, Red Hat, and CentOS Directory Servers). Although this plug-in has existed for quite some, there isn't a whole lot of documentation about how to implement it in a real-world scenario. I recently submitted some documentation to the…
Hello all ! Today we're going to take a look at a somewhat obscure problem that - once encountered - can cause nothing but headaches for a system administrator. The problem relates to conflicts in CPAN RPM packages, and what can be done to work around the issue. If you've made it this far, i'm going to assume a couple of things : you're comfortable with RPMs and repositories,…
Happy 2010 fair readers ! I hope that all is well with you and yours. Let's get right to business : Virtualbox has a feature that allows you to access the host OS's file system from the guest OS ( shared folders ), which is super useful, but not exactly perfectly implemented. In particular, there are known, documented performance issues in certain scenarios, such as when accessing a…
Hello again fair readers ! Today's quick tip concerns the problem with missing time zones when deploying CentOS 5.3 (and some of the more recent Fedoras) in a kickstart environment. It's a known problem , and unfortunately, since the source of the problem (an incomplete time zone data file) lies deep in the heart of the kickstart environment, fixing it directly is a distinct pain in the…
Hello again, everybody ! Today i thought that we'd take a look at a fun and useful topic of interest to many system administrators : load balancing & redundancy. Now, i know, it doesn't sound too exciting - but trust me, once you get your first mini-cluster set up, you'll never look at service management quite the same way again. It's not even that tough to set up, and you can…
Hi all ! I'm currently working on a new Howto regarding Load Balancing. It's gonna be a good one, complete with graphical charts and everything - so stay tuned !
Hi everybody - here's a super-quick update for you concerning « ethtool », and how to use it to set options in Fedora properly. Ethtool is a great little tool that can be used to configure all manner of network interface related settings - notably the speed and duplex of a card - on the fly and in real time. One of the most common situations where ethtool would be used is at boot time,…
UPDATE: This article was written back in 2009. According to a commenter below , Busybox has been replaced by Bash in RHEL 6; perhaps Fedora as well? Bonjour my geeky friends ! :) As you are likely aware, it is now summer-time here in the northern hemisphere, and thus, i've been spending as much time away from the computer as possible. That said, it's been a long time, i shouldn't…
Hello again ! Today's quick tip concerns a software package called Dia , which is an open source tool (available for both Windows and Linux, as it goes) used to make diagrams, flowcharts, network maps, and so forth. It has its own file format (.dia), which is (obviously?) useful for saving the projects you're working on, but less useful if you need to give the diagram to anybody else,…
Hello again fair readers. Today i'm going to re-visit POHMELFS, which i introduced in an earlier blog post . I received a comment on that post which basically asked for more information on some of the more interesting (read : advanced) features of POHMELFS, such as distributed storage, and the like. Well, today is the day ! If you need a refresher, be sure to skim over my previous post, as…
Today i'm going to address a question that comes up again and again on the various Linux help forums and mailing lists (in one form or another) : « How do i set the From address when using mail ? » Of course, « mail » in this case refers to that most basic of all Linux mail applications : GNU mail. Commonly part of the default software on a Linux box, it's not used very often in an…
Hello again ! You may be wondering when the next update in the POHMELFS series is coming - well, rest assured that i'm working on it even as you read this, and that it will be worth the wait. Remember that we're working with Staging-level code, and that sometimes things don't always go as well as one might hope - in this case, there are some discrepancies between the code that the…
Hello again ! This is a quick and dirty update which covers a handy little trick when dealing with writeable removable media - especially USB drives, compact flash cards, and the like. I end up using a lot of USB keys in my environment for a variety of reasons, not the least of which is as handy portable Linux drives that can be stuck into any workstation and booted from directly. They're…
Hello again fair readers ! Today we're going to take a look at POHMELFS , which is a network file system that was just recently integrated into the Linux kernel. This is an excellent exercise for three reasons : we'll learn about some great new concepts related to network file systems, we get to compile and install our own kernel, and we get to play with a great new platform that could,…
One of the wonderful things about Perl is that it enables the busy System Administrator to be lazy - and that's a good thing ! Of course, i don't mean lazy as in unmotivated, or possesed of a poor work ethic, i mean it in the sense that Perl lets us do as little work as possible in a wide variety of situations. Let's examine this idea, shall we ? In the computer world, one often…
Today i thought i'd discuss initrd, how modules live within it, and an give a generic example of how it can be modified on a Fedora-based system. Along the way, i'll provide supplementary information about gzip, tar, and cpio. Let's begin ! initrd The term « initrd » is short for « initial ramdisk ». It refers to a file of the same name used by Linux systems during the boot process…