An occasional blog on Free Software
Contents Flavour of the month What would Bill do? You had one job A tool looking for a developer Where to from here? Flavour of the month For the past few years I've worked for a consultancy primarily focused on Amazon Web Services . AWS has been #1 in public cloud market share since that market existed, and depending on who you believe and how you carve it up, they still hold between 30 and 40…
When I picked the title for my recent series , " What's the time , Mister Cloud ?", I originally had in mind an homage to the children's game "What's the time, Mister Wolf?" 🐺 When reviewing my search analytics, I found that some people are coming to this site via searches for songs with similar names, and so of course I had to check them out. 🎶 I enjoyed this funky reggae version: and also this…
Contents KVM PHC on AWS Test process Measurements Microsecond accurate time for the rest of us Comparing apples with apples We have a winner Applying some taco seasoning So, two winners, actually (Note: Readers who aren't familiar with time sync in public clouds might want to check out my recent series . My previous post about KVM timekeeping is also useful background.) KVM PHC on AWS As I…
Contents Background Performance improvements - chrony Common prerequisite Chronyd configuration Ntpd configuration Performance improvements - ntpd Next steps Thanks Background In my last post I described the setup I use to provide time synchronisation to the hosts I maintain in the NTP pool . I only recently learned about the PTP Hardware Clock (PHC) device driver available in KVM, and started…
Contents Background NTP root servers Bare metal VM hosts NTP pool VMs Monitoring Future plans Background In my recent posts I've been focusing on time synchronisation in virtual machines in public clouds. In my upcoming posts I'm going to cover how to apply the same techniques to improve time sync on generic KVM guests. Before we dive into that, I want to give an overview of the infrastructure I…
Contents The AWS time sync service Microsecond-accurate time announced How it works Prerequisites Chrony configuration Measurements Conclusion Afterthought (Note: Readers who aren't familiar with time sync in public clouds might want to check out my recent series .) The AWS time sync service AWS first introduced their NTP-based time service in 2017 and it has been ticking along ( pun fully…
Contents It's About Time Prerequisites What is time synchronisation? Why should you care? What is good or bad, anyway? Technical terms Looking at time Frequency - an accurate clock Frequency - a clock which runs fast, consistently Frequency - a clock which runs consistently slow Frequency - an inconsistent clock Frequency - an equally inconsistent clock (but can you tell?) Phase - a consistent…
Contents Computer Timekeeping Time zones Types of clocks Hardware clocks Real Time Clock (RTC) System clock Clocks in action Bare metal Virtual machines Containers Time sync standards Network Time Protocol (NTP) NTP implementations NTP in action - ntpd NTP in action - chronyd Wait, wasn't this supposed to be something about Mister Cloud? Computer Timekeeping In part 1 of this series we thought…
Contents The Part about Mister Cloud! AWS Services in scope NTP in AWS: initial configuration NTP in AWS: adjusting the defaults Which NTP servers? When is a second not a second? Leap second smearing Azure Services in scope NTP in Azure Local PTP device Not leap smeared? PTP device: pros & cons Testing the services Experiment parameters Method Goals Caveats ⚠️ Graphs ahead! Offset Frequency Root…
Over the past few weeks I've made some changes to my NTP health monitor NTPmon , and I want to explain them more fully and outline the roadmap from here. 2.0.x series It was pointed out to me that it wasn't actually clear how to run NTPmon for anyone looking at it for the first time. Besides updating the documentation to fix, I decided to brush off my long-neglected Debian packaging skills (which…
An IPv6 addressing detour My next post about The Little Network That Could ( TLNTC ) was going to be about IPv6 addressing plans - and this post is definitely about IPv6 addressing - but before we can lay out a plan, there's something more fundamental that needs to be decided: which IPv6 address range should I be using in my network plan? This seems like a very simple matter, and shouldn't need to…
(Yes, I know this is being published on April Fools' Day. I might be a fool , but this is not an April Fool's joke.) This year's model Platform teams and platform engineering are phenomenons which have taken the cloud native world by storm recently, so much so that it has seemed a bit socially unacceptable to question them. Team Topologies has been every DevOps manager's favourite book for a…
In keeping with my usual tradition of throwing in a random off-topic post every now and then, here is my favourite post-run meal, a banana/peanut butter/vanilla protein shake. When I get it just right, I think it's better than any thickshake I've ever tasted. It's vegetarian, but not vegan (although probably could be made so). Ingredients 1 large banana, optionally frozen 1 tablespoon peanut…
Part 1: Even maildirs need their personal space I did a migration of my mail server over the last weekend and I ran into a strange error when I brought things back online. My system was receiving mail fine, but it was being deferred in the postfix queue with the message: Command output: /usr/bin/maildrop: Unable to open mailbox. This was very odd, because the bulk of the files copied over to my…
After maintaining a WordPress -based site for some years, I've decided it's time to try something that requires a little less maintenance, and hopefully performs much better. After looking at a few options, I landed on Pelican , a python-based static site generator. It's a bit like ghost or hugo (in fact, this theme is ported from one of ghost's themes), but I'm much more comfortable with its…
This morning I walked out of the bedroom to make my morning coffee and was greeted with a tech support problem. My wife had been up for a while already, and said something was wrong with the network. Coffee in hand, I opened the laptop and was greeted with a warning that I had no Internet access. As I started to pull up my usual diagnostics pages, I found that DNS requests were timing out in some…
[For more context: see Tech nibble: KVM host networking topologies ] Traditional VLAN bridging and its problems Linux bridges have been the traditional means of connecting KVM guests, and VLAN support is usually the means by which they are segregated, but it used to be that you had to give guests access to all the VLANs enabled on the host bridge. So to isolate multiple guests from one another, a…
As part of my move to IPv6-only networking , I took some time to re-evaluate the functions and configurations of my VM hosts with the aim of greater segregation between workloads, more predictable failure scenarios, and making sure my IPv6 addressing plan is up to scratch. (I also have one VM host still running Xen that I'm migrating to KVM.) Two distinct use cases In thinking about the network…
I've previously explained the background and motivation of The Little Network That Could , but I realised that there are some implicit values that permeate my choices which would be worth making explicit. Here are a few which come to mind: K.I.S.S. Espousing the Keep It Simple, Stupid principle is well-enough established in IT circles that I don't feel the need to justify my selection of it as a…
It seems that MLD snooping on Linux bridges has broken various parts of IPv6 connectivity over the past 10 years or so. Initial reports of bugs were from 2013, when having MLD snooping enabled on the bridge broke neighbour discovery (breaking IPv6 connectivity altogether), corosync multicast (including non-IPv6 multicast), and other guest-to-host multicast traffic:…
Alongside my tech nibbles posts, I'm going to add another type of short post which chronicles the difficulties I've encountered in trying to achieve my IPv6-only end state. These posts may contain somewhat spicy opinions 🌶️ or ranting 🤬. Mostly I have worked through these problems, but documenting them is valuable for me in clarifying my thinking about these specific problems and problem solving…
Once again it has been too long between blog posts, and there has been quite a lot of activity going on behind the scenes in The Little Network That Could ( or TLNTC, as I call it ). I've built up such a large backlog of topics (mostly from the rabbit-holes that I've gone down in making the transition to IPv6-only) that my list of future blog posts itself was becoming unwieldy. So I decided to…
It has been far too long since my previous post introducing TLNTC . This episode will be a (hopefully!) shorter overview of TLNTC's journey into IPv6 connectivity. Early days I don't know when I started first started to think seriously about learning and implementing IPv6, but probably the earliest defining moment I can remember was Geoff Huston's LCA2011 keynote in which he explained the IPv4…
A few months ago now I got a couple of friendly alerts from OpsGenie telling me that my web server VM was down. I logged in and poked around, but things seemed to be pretty normal. The OpsGenie alert self-resolved shortly afterwards, and I thought nothing more of it. Then I got this little message via the comments: Link for part 4 will not load. Server timed out. Tried 4x After a bit more…
(Credit: Flickr ) Gather 'round, folks - Grandpa has a story to tell. (OK, I'm not a grandpa and I don't expect to be one any time soon, but as I've journeyed back in my memory to write this post, I sure feel old...) For as long as it has been possible to have a full-time Internet in a residential home, I've been running my own home network, and I want to share its story: it's The Little Network…
[I originally started this post several years ago, but never got around to finishing it at the time. I think this instalment is more than overdue, and I hope to finish a few more over the coming weeks.] A big change in my running came around October 2015, when my doctor did a routine blood test, and diagnosed fatty liver (like I didn't know that already from the shape of my waistline?). He told me…
This morning before work I sat for (and passed ) my attempt at the newly-minted VyOS Certified Network Engineer certification. Mostly this post is just to let folks know that the certification is out there and encourage them to take it, but also I want to compare it to another certification I recently passed, the AWS Certified Solutions Architect Associate . I've liked VyOS (and its predecessor…
A friend asked me today: what are the pros and cons of chronyd and ntpd? I've used both for a while, but never actually sat down to think about this question. So here are some initial thoughts: ntpd (the older of the two implementations): has had more time to mature has had more time for the codebase to become more kludgy and less maintainable supports a wider range of hardware supports more…
NTP: a behind-the-scenes protocol As I've mentioned before , Network Time Protocol is one of those oft-ignored-but-nonetheless-essential subsystems which is largely unknown, except to a select few. Those who know it well generally fall into the following categories: time geeks who work on protocol standardisation and implementation, enthusiasts who tinker with GPS receivers or run servers in the…
Last week I was privileged to be sponsored by the Network Time Foundation to give a talk at AusNOG 2017 . The slides are now available on the conference site (and here , too). The talk covers some of the operational issues which NTP has faced over the past 30 years, RFC drafts for future changes to the protocol, and Best Current Practices.
I recently wanted to look at some packet captures on my NTP pool servers and find out if any NTP clients hitting my servers use extension fields or legacy MACs. Because the overall number of NTP packets is quite large, I didn't want to spool all NTP packets to disk then later filter with a Wireshark display filter - I wanted to filter at the capture stage. I started searching and found that not…
This week I gave a talk at SecTalks BNE0x18 about how I solved last month's boot-to-root CTF. I've put the slides up now (you'll need to enable JavaScript for the remarkjs slides to work).
Background This will be a shorter, less-polished post than usual. It's really just a way to start to bring a bit of structure to my thoughts. Feel free to add some comments or weigh in on the linked Twitter thread. I came away from a recent PacketPushers episode, " Don’t Believe The Programming Hype " a bit disappointed that the discussion didn't get to the heart of the matter. Then there was this…
Yesterday I updated NTPmon to include telegraf support; if you are interested in the code changes see github compare for details. It should be considered alpha-quality at this stage; it communicates with telegraf using the TCP or socket listener, and a systemd service config which will restart it on failure is included. Here are some graph samples - firstly in LibreNMS via collectd : And in…
I presented my talk based on the blog posts here at Linux.conf.au 2017 ; thanks to everyone who responded - your feedback is appreciated. Here are some links for anyone who's interested: Talk slides Youtube video Downloadable video There won't be much new in there for anyone who has been following along with the blog series, but it's a little more succinct. Note that there are some "Deleted…
In this post, I'm going to address some of the more common myths about NTP and how to avoid the mistakes which they produce. Some of these myths are grounded in fact, and in many cases it's fine to accept them if you don't need highly accurate time and you know the consequences, but they are usually based on misconceptions about how NTP works which can lead to greater errors later. Advance…
(See the menu for other posts in this series.) Am I in sync? So now that we've configured NTP, how do we know it's working? As Limoncelli et. al. have said , "If you aren't monitoring it, you aren't managing it." There are several tools which can be used to monitor and troubleshoot your NTP service. ntpq ntpq is part of the NTP distribution, and is the most important monitoring and troubleshooting…
( Part 1 one of this series gave the background and rationale, and part 2 covered the basics of how NTP works.) Getting NTP Depending on how you install Linux, you might already have a pretty workable NTP client, but for the tools we'll be working with, we'll need the NTP reference implementation server. On Ubuntu (these examples use 16.04 "xenial xerus", but the instructions should work on all…
( Part 1 covered the background and rationale. Part 3 is about installation and configuration.) What is NTP? NTP (Network Time Protocol) is an Internet standard for time synchronisation covered by multiple RFCs . "NTP is [arguably] the longest running, continuously operating, ubiquitously available protocol in the Internet" [ Mills ]. It has been operating since 1985, which is several years before…
(With apologies to Derek Zoolander and Justin Steven . And to whoever had to touch the HP-UX NTP setup at Queensland Police after I left. And to anyone who prefers the American spelling "synchronization".) (This is the first of a series on NTP. Part 2 is an overview of how NTP works.) The problem with NTP In my experience, Network Time Protocol (NTP) is one of the least well-understood of the…
(You can read part 1 first, if you want.) When I first started running, I naturally fell into what I considered to be a proper running pace, but I found that I was not able to run as far as I wanted, and it wasn't until I made a conscious effort to slow down and concentrate on distance that I was able to achieve 5 km without stopping (the goal of Couch to 5K ). They advise that to start you should…
(I'm trying to get motivated to write a little more. This post is a little off-topic from my usual fare; hopefully it will be of interest to some. There will probably be 3 or 4 parts.) Disclaimer This is my story about running. I don't claim it should be the norm for anyone. If I lapse into 2nd person as I write, please be assured that it's not intentional, and is not intended to be advice to…
I'm no longer working for myself, so I've moved most of my old content to this new blog. The same entries should generally be available, although there may be some issues with paths not matching and images not showing by default. If there's something you're particularly interested in, use the search to see if it's there, otherwise let me know and I'll try to dig it up (or you could just use the…
(Warning: Contains political content about the upcoming Australian federal election - although not the type you might expect...) Annabel Crabb hit a nerve with this article opening recently: You're one week into a federal election campaign that is going to sprawl malevolently over one sixth of this calendar year. So I imagine you may be reading this in the comfort of your own home panic room. Or…
I blogged previously about the podcasts I regularly listen to, and the topic came up again this week on the SAGE-AU mailing list, so I thought I'd update it with my current thoughts. Regular listens from the previous list SANS Internet Storm Centre daily podcast [ feed ] - Still my "must listen" podcast. Risky Business [ feeds ] - They have done a little cleaning up on the NSFW content, but they…
The other day I got a bug report about check_ntpmon , which was reporting UNKNOWN status back to Nagios even though everything seemed to be working fine. A bit of debugging revealed that it was receiving the message on standard error: ntpq: write to ::1 failed: Operation not permitted This was a bit strange, because various links I found indicated that this message is usually due to firewalls:…
I've been asked a few different times to advise young (and sometimes not so young) people trying to get a start in IT. Every story is different, but there are a few commonalities. So here's my generic IT career/job seeking advice in rough order. Note that some of it is a bit Australia-centric. Prove to potential employers that you are a lifelong learner: If you don't have a tertiary qualification,…
I was stoked when Patrick Gray took up my suggestion to ask Marcus Ranum to reflect on " The Six Dumbest Ideas in Computer Security ". I encourage you to listen to the interview for yourself, but my summary of it is that Marcus was mostly discouraged that very little progress has been made in computer security, while Patrick was of the opinion that a number of good lessons had been learned in…
Vyatta Core (VC) is one of my top fanboy loves. It provides a firewall/router based on Debian Linux but with an elegant configuration system modeled on Junos. Vyatta previously offered VC as a community edition of their commercial router (with somewhat reduced features and a lag time). Their strategy seems to have changed when Brocade took them over, and unfortunately, the project has withered on…
Tonight i was working on getting a client's Zimbra SSL configuration up to scratch, and found it somewhat difficult to get our server to make Qualys' SSL Labs scanner happy. I was working from the following Zimbra wiki pages: http://wiki.zimbra.com/wiki/Weak_Cipher_Suites_Appear_in_Security_Scans http://wiki.zimbra.com/wiki/Postfix_PCI_Compliance_in_ZCS http://wiki.zimbra.com/wiki/Cipher_suites It…