I still run nomad at home. This surprises some people, who assume that anyone with opinions about production has long since capitulated to kubernetes, but the whole point of the lab is that it's mine and it's small and I get to pick. Nomad is still the right amount of …
Something I've been reminded of the last while: old social media posts are not assets, they're liabilities. Not just because of cringe (though there's plenty of that), but because the actual upside of preserving a bunch of posts from 2018 is approximately zero, and the downside is that they exist …
One of my favourite phrases to remind people of as I'm fumbling about the business of producing reliable systems is "We Should Disappoint People Early" . I've gotten enough funny looks about this that I figure I should explain myself. The 'polite' assumption most folks make is that we should aim …
Asking the Right Questions A while back when I was looking for a fulltime gig (and when I was contracting, of course), I had the opportunity to do a bunch of interviews (including the kind of informal interviews you do to get a new contracting client). One of my boilerplate …
One of the more impactful bits of "DEI work" we (ohai Sara Smollett ) did at the G was helping to re-write our job descriptions for SRE (I later re-wrote the internal job ladder for SRE, but that's probably not as directly related). Circa 2010 or so, our JDs needed a …
Part of my day-job as head of Engineering at Google Ireland was attending talks and events/Q&A for local interest groups like Engineers Ireland. I was reminded today of a panel I was on around diversity in engineering in Ireland , and my allegedly 'fascinating' comments at it. That's not …
Communication Scales with Scale Communicating with your team scales up as your team grows - it's an experience not unlike the individual engineer to senior engineer to staff engineer path. As well as scaling up in terms of impact, the very method of how you do it changes. You very quickly …
Bit of a shuffle in what musical projects I'm working on - two groups I'm with have fizzled (at least, my involvement with them has) and one other is coming up. One of the bits I've signed up for is playing samples on stage through front-of-house, so I've invested in a …
On July 19th 2004, I spent my first day at Google. I showed up at the Datacenter in Dublin, since that's apparently where SREs and production folks were going to sit. There wasn't a corp network connection, because this was Google in 2004. A couple of days later, I moved …
Going to be playing a bit of catch-up, since it's been a few months. Work is still workin' -- I've got some coaching going on, and am still working mainly with one team on consulting. It's been fun times, and the gig is flexible enough that I've gone essentially part time …
A few months further into my "not working full-time and seeing how that goes" journey. I've picked up a contract for the consulting end of my 'job' since January, that'll likely last til the summer or thereabouts -- it'd been fun doing the good and useful parts of the job I …
Recycling old talk proposals for fun and...that's it! Here's an amalgamation of the proposal text and some notes from a talk I proposed for SRECon EMEA last year -- I'll be talking about this to the Dublin SRE meetup in a couple of weeks, so here it is while the …
The last few weeks were a bit of a hive of activity. I went back to work, started a small business , and went from no-busy-at-all levels of work-related concentration to quite-busy-and-not-very-stressful-yet. I'm going to be doing more than just coaching, but that's for another time. Earlier this year, I asked …
Standard disclaimer: Maybe don't do this in production? I hear Hashicorp also does a secret store :-) Nomad 1.4.0 added support for 'variables', which are essentially key/value blobs protected by nomad's ACL system. As someone who had steadfastly avoided either running vault, or doing secrets properly, I knew …
Further to my last post where I went away for a few weeks, that away-time featured a power cut at home. Things....did not come back cleanly, and required intervention to come back -- not least of which was DNS. Because of course. My setup had been running DNS locally for …
A lot has happened over the last few months, so rather than a big catch-all post, I'm going to do a few posts over the next few weeks. The last week or so have been my first 'in the office', so to speak, since March. As is patently obvious to …
The last several years have seen a huge upsurge in the popularity of the DevOps/SRE/Production Engineering model, with companies large and small adopting some of the practices and mindsets. One of the principal lessons many of these organisations (hopefully) learned was that it's close to impossible to adopt …
My last few weeks have ended up with me having about 30% of my regular day-job meeting/stuff-I-gotta-do load; due to a perfect storm of school stuff and other adventures (I'm currently collecting driving license categories! Trailer learner permit coming and sitting some theory tests for Truck license). Other adventures …
I've mostly ported my homelab across, and took the opportunity to add actual monitoring and actual resiliency. I'll do a post on the janky setup. I've also (thanks to some kind mastodaoine) gotten set up with restic to some remote storage. I'm glad software like this is still being made …
One of the most constructive things I've found when facing a difficult work situation is to externalise and write things down -- I do quite well with talking things through with folks, and coaching and mentoring is something I get a lot of energy out of. When it's just me and …
In among posts about fiddling with electronics and occasional life updates, here's a life update! My partner (and now fiancee, I guess?) isn't very present on the internet, and escews social media in fairly impressive ways. But, we've been together for close to 15 years, so we decided to make …
Further adventures in pretending homelab is real work! This week, I've actually been hacking on getting restic running semi-regularly. It turns out (or at least, I've found by experimentation) that the way nomad does scheduled/periodic jobs means that the periodic jobs don't have access to variables the core job …
I've been gradually rolling out more homelab updates, including whats turned into a bit of a rebuild and de-flakiness effort. Nothing like a bit of sunlight to realise that things need a proper looking at :-) It's also been pretty good for my brain, I think -- having the benefit of really …
Now that I've a bit more time on my hands (more on that later) the predictable swing back toward being able to do actual tech fiddling has started (I Hope). I've been running nomad from Hashicorp as the compute coordinator for my home setup for a couple of years now …
It's November! That most wonderful time of the year when your brain and being need a little more maintenance. Part of my maintenance schedule is taking the worsening weather as a prompt to feel less guilty about not going outside and doing outside things -- i.e. getting to coding and …
I spent a number of days at SRECon 2022 in Amsterdam, and gave a talk that I'd had rattling around in my head for a wee while - roughtly based on a paper I left at Google when I left. Laura was good enough to suggest it as a ;login: article …
2021 was a year. I've seen the receipts. In summing it up, I guess I'll preface it by saying that this, more than any previous year, was one where the shared experience of being the connected human was a more closely bonded one than ever. The shared experience definitely had …
I'm allowed one cheat blog post, as a treat. The twitter thread I posted a while back seems to have gotten some attention. It's reproduced on threadreader but that too shall pass (and I auto-delete my tweets after 90 days), so here's a reproduction. The SRE book is a history …
After being asked by a work colleague to say words about OKRs and if they're a good thing to adopt, I realised I had more words than fit in an email or short doc -- so, here's a slightly longer doc on my view of OKRs, tempered by 15 years or …
This past week was my first week at elastic.co , where I am now a Director of Engineering - taking on the SRE group and making Elastic Cloud go (Elastic Cloud is essentially the SaaS version of Elastic Search - i.e. you can of course run it yourself, or you can …
The last few days (among other things) saw a little work on the previous project , as well as taking advantage of having new hardware to mess about with. My desktop OS journey has a 12-year (or so) mac gap in it -- which is to say that I used to use …
Showed up to a few low-key conferences last week (Multitasking between SLOconf and o11yfest for a few days). It felt great to be catching up with some folks I hadn't in a while -- not having the GOOG hat on was a refreshing change. It's like the manager/director/VP/whatever …
Thanks to my accomodating erstwhile colleagues, I'm able to publish the original paper (with minor redactions so it makes sense for an external audience). A version of this paper later became Chapter 29 of the SRE Book . To add a little bit of extra colour, the document came out of …
Let's try every week or so? No excuses about no time :-) The last week or two have been decompressing, switching off. I found/find myself unable to do most of the technical/outside/etc. stuff that would usually count as "Doing stuff I want", in favour of a sort of …
So, last week was my last week at Google. I'm writing this from the weird void of disbelief where I've not quite settled into the 6 or so weeks of time off I have before the new thing (which I'm trying not to talk about, because I do want to …
One of my excuses for not updating this more often is the fiddly nature of making sure I have a working Pelican install. I mean, the real reason is laziness, but that's a handy excuse. "Woudn't it be great", my insomnia-brain said to itself, "If I could just check a …
I'm now most of the way through the ~month or so I took off work. Around the start of it, I made a few plans, but I also resolved to unapologetically spend much of it just relaxing, and feeling non-guilty about that. It's something I've been working on, and I'm …
My last attempt at doing the Home Automation/Domotics/Whatyacallit a few years back mostly involved buying expensive widgets, determining they are shite, or marketed at US power and outlets, and then discovering that setup and upgrades generally involved one ore more of downloading a random .exe from random websites …
So I've set both a really good example (take your vacation!) and a really bad example (Don't leave it all til the last minute!) for work folks, and therefore today was my last day at work in 2020. It's been a year. One of the things I'm especially looking forward …
Short a quick trip to Zurich in January 2020, the trip I was on was the last time I travelled outside Ireland, until now. I also don't plan on going anywhere until mid-2021 or thereabouts. Much as all that's been written about how this year has been, one of the …
It's been over two years, and while I'm still patiently waiting for the demise of Medium , it only took re-running pelican-quickstart to make my makefiles work again. Plus we haven't deprecated serving static content just yet, so there's that. Most of the selling points for blog platforms these days is …
The last few months have been a bit hectic, let's say that. Work In work, a few things happened. I got promoted to 'Director', which is the top end of the management progression, bottom of executive. What either of these mean, I'll probably never know. About a year or so …
This is something I wrote for new Googlers, drinking from the information hose. It's not formally part of the onboarding process, but enough people email me about it that I suspect it's passed around a lot informally. (With Apologies to Mary Schmich) Everybody's Free (To Use Their Best Judgment)
This is a paper I wrote for consumption of people doing conferences and trainings at Google. I figured it was more generally applicable, so here it is: Conferences for Introverts
I was away last week with work to an offsite at the Club Med , which as far as I can gather is one of those soviet-era collective holiday camps for people who weren't in a soviet. I bet the soviets never had nespresso on demand. Bizarrely, the weather in Ireland …
The weather turns and turns. We're well into the season where things spring out, and make you cut them back, or just be happy they're there. Our garden has rhubarb. Enough for a tiny put of jam, or perhaps a Mr. Kipling's style rhubarb tartlet or two. It also is …
Further to the Reddit AMA I was involved in a few years back (I am /u/sre_pointyhair) - "The Guy" showed up at my desk last week. Alas, I was away, and he had to make do with doing the unspeakable things we do to keep servers running from my desk …
I finally got my grubby mitts on an actual dead-tree copy of the SRE book. I've wanted this to exist for many years, and it's finally out there. I wrote a chapter on working with teams with high interrupt loads where the team consists of humans. It's a paraphrase of …
Back from a few days in Prague and beyond, at Border War . This was a very decent one - and I ended up not being as wrecked as I thought (due to the previously mentioned unfitness). Did about 45km of hiking over the 3 days of the event, which is a …
So, case in point (since we love examples). A couple of years ago, when I first started fiddling with Pelican , one of the first things in the docs was 'okay, set up virtualenv and do such a thing'. My initial reaction, as an old-skool sysadmin weenie who hasn't had to …