Recent blog posts on Odds and ends
## Introduction Data bombs are maliciously crafted files designed to exploit vulnerabilities in data processing software, causing them to consume excessive resources or crash altogether. “Lossless” compression is a powerful tool. It enables us to take a large file and represent it with much less information, saving disk space and network bandwidth. That can result in files being read…
## Introduction For a particular experiment at work a few years ago, I found myself needing to make about 10TiB of hard to compress data. /dev/urandom is “slow” at 400ish MB/s on machines I have access to. I put that in quotes because that still seems fast to me, but 10TiB would take about 7 hours at that speed. Unfortunately I don’t remember where I came across this trick, but…
Every now and then, I stumble across videos on YouTube, showing visualisations of different sort algorithms. It’s strangely pleasing watching order emerge from chaos in various ways. Here’s a great example of one: While I don’t have them around any more, I’ve made a few attempts at making my own in the past, and having seen a sorting video recently, was inspired earlier in…
## Introduction Years ago, I needed to track down a specific kernel regression, and I worked out an approach to automatically bisect the kernel, leveraging a combination of git and qemu. A few days ago, someone posted on reddit that they’d seen a massive BTRFS performance regression in the 5.10 kernel under certain circumstances, and they’d managed to track it down to a commit.…
On Wednesday early evening Kari and I welcomed our second daughter to the world, Alia. How do you deal with two kids?! Well.. three I guess if you include our dog Oakey. So far Naia is being a good “big sister” and helping to make sure everything is good with Alia. Oakey is sitting on guard (in a good way), making sure that no one bothers the baby unncesssarily, and every time she…
The Go language from Google has been picking up lots of interest over the past few years. Speaking as someone with a background almost entirely in dynamic languages, Go seems to strike a really nice balance between static dynamic languages and statically compiled ones. It comes without the hassles of memory management, compiles quickly (and easily cross-platform), and doesn’t seem to have a…
A tweet by Kelly Sommers earlier today (well worth a follow if you’re a techie type) got me reminiscing. At a previous job my primary role was to be a consultant sysadmin for customers. They could pay for my time to help resolve their problems. It proved to be an interesting challenge at times, though I seemed to spend an awfully large amount of time dealing with compromised servers.
Is Comcast really throttling Netflix? Short answer: Given the presented evidence so far, we don’t really know. It’s complicated. About the only ones who could really tell us is Comcast, and of course there is little incentive for them to tell you if they are. What we do know is that they’ve got some kind of problem somewhere as people are able to get better performance by using…
The last month has seen a fair bit of disruption in our lives. A few months ago a friend at Amazon put my resume forward in Amazon’s employee referral program. After a fairly straightforward but complete interview process with them I was offered a job in the Glacier team. It’s been our plan for a while to leave Hawaii for Seattle, but there’s a whole world of difference between…
Today saw the culmination of a lot of hard work from our content management team, design team and the sysadmins. The totally overhauled state portal, http://portal.ehawaii.gov, went live today introducing a completely new toolchain to our environment and some innovative concepts for state portals. The new portal switches over to a HTML5 based responsive, mobile first design. It’s touch…
This has been bugging me for most of a year now. As much as I love doing things from the command line, configuring a PIX or ASA Firewall is not one of them. Especially when you get beyond a dozen or so lines in the the access lists. Unfortunately about a year ago I found the ASDM just stopped working from Linux and I couldn’t get it to work again, and neither could my colleague here. He runs…
There’s been a bit of noise recently about websites not producing server-friendly download links, and thought I’d add in my own 2p. There seems to be a mix or reasons, but one of them is due to a future feature in Chromium that should help. As a fairly routine part of my workflow I’ll have to fetch applications or other files from online for use on our servers, and for a good…
In it’s default configuration, Tomcat pays absolutely no attention to any X-Forwarded* headers being provided by load-balancers to the application. If you’re doing SSL Termination at a load-balancer and passing traffic as straight http inside your environment that can present some slight problems. One example is that any redirects carried out by the application will come through as…
This is something I keep finding myself needing to use. We have a series of bash scripts that pull out bits of metrics that are fed to monitoring and other services. The actual result value frequently comes with whitespace all around it. If the scripts were perl based it we could just use s/^\s+// and s/\s+$//, but bash built in substitutions are slightly different.…
When I started working at my current employers monitoring was fairly rudimentary, CPU Load, an inconsistent set of disk space monitoring and so on with nothing being graphed and a decided lack of useful information to aid in troubleshooting. I’m a bit obsessive when it comes to monitoring. In an ideal circumstance I’d monitor absolutely everything, everywhere, provided it doesn’t…
It’s generally advised that when you’re using SSH that you generate an SSH key and change ssh so that you can only use keys for logging in. For security reasons you also ought to consider using separate SSH keys per environment you’re connecting to, so if someone gets hold of one of your keys somehow, they don’t end up with access to everything. However it can get to be a…
So proud to announce the arrival of Naia, born a nice and healthy 8lbs 10oz, and measuring in at 21". Kari was absolutely wonderful throughout the entire process, even at the end of 28 hours of labour.
I tend to hold on to my phones for a good while before upgrading, usually to the point where it gets just too irritating. I’ve had my Nexus One for two years now, and the lack of memory was getting seriously frustrating, even with CyanogenMod on it. A couple of weeks ago I took delivery of a brand new, paid-up-front Samsung Galaxy Nexus which comes with stock Ice-Cream Sandwich version of…
## To flex, or not? Something I find to be a difficult juggling act in my job, and suspect I always will, is juggling the demands of the infrastructure with the requirements of the developers / staff. Ultimately I’m here as a problem solver covering multiple levels like ’this app needs to do foo and bar’ or ’the infrastructure needs to be running, stable, and secure'. That…
# Intro AppFog’s CEO Lucas Carlson about NoOps has stirred up a little brouhaha amongst sysadmins and developers both. I’ve sat and read through both the original infographic and the follow up blog and apart from there being miscommunication in both directions, I still find myself disagreeing with the fundamental premise. He’s not the first one to blog on the subject of NoOps, A…
I don’t like to go in to politics much here, though I try my best to keep up with what is going on, even though as a legally resident alien I have no voting rights. Frankly, with the perspective of a foreigner, American politics disturbs me on many levels. There is such a heavy emphasis on ’left’ vs ‘right’, ‘socialism’ vs ‘conservatism’ that…
I was fiddling about with Project Euler yesterday. Occasionally I like to delve into a bit of maths and programming, just to reassure myself that I’m still totally out of my depth with these things. The puzzle came up that using british coins (1p,2p,5p,10p,20p,50p,£1,£2) how many different ways could you make £2? I was aware of the Greedy Algorithm, it’s what we tend to do…
We’ve been keeping a little something secret for a month or so, and figured we’d give a little extra present to our family today. My dear wife and I are expecting our first child, due the first week of July. We’re just past the 12 week mark and Kari has been dealing with the joys of all-day sickness, long hours at work, school and college; About all I’ve been able to do was…
I got back today from having spent another hour in the data centre. We’ve been setting up 3 new Dell servers to go into production, alongside two new F5 load balancers. I say new, we’ve had them several months but until I had a Jr join me I didn’t have enough time to get them set up and ready for use, given the environment I inherited is rather complicated (80+ apache vhosts with…
# Intro Last week I scheduled to do a Zimbra upgrade on Friday, hoping that a few days off work would have been enough to build up my mental reserves for the chore. Zimbra is great, when it’s running. Very much fire and forget, it just works. It’s not flawless, the Outlook connector still does weird stuff so proves unusable to us, but from a day-to-day perspective I almost never have…
After considering it last year, I’ve decided this year to participate in Movember. What is Movember? Movember is an annual event to raise money for the Prostate Cancer Foundation by cultivating a Moustache for the month. This is going to be very interesting. I consider myself to be extremely lucky to be approaching 31 and only just beginning to reach the stage where I need to shave daily.…
Recently I’ve been thinking about how I use or don’t use my blog. I get fairly steady traffic to a few pages, but nothing particularly large. Wordpress really is overkill. The only thing I even had MySQL and PHP running on the server for was wordpress, and between them they were taking up a lions share of the RAM (I was running Nginx + php-FPM so at least I wasn’t dealing with…
We’re seeing some occasional performance problems with one of the sites we host, that appears to come from the database end. One particular query keeps cropping up in the mysql-slow logs as it typically takes 12+ seconds to execute, even when the server is under light load on a reasonably powerful dedicated database server. mysql> explain select distinct docnumber,doctype from document where…
This is something I’ve been meaning to write for a good while, but never had the time when it would have been useful, and had totally forgotten about when I did have the time. A big thank you to Tony Darnell at Oracle for that. I’ve made just a slight modification to it to make it return a rough progress indicator. Some of the database dumps I’m dealing with are 25Gb+ in size, so…
I’m going to be making a relatively small change to our infrastructure at work soon which involves a DNS change. In theory all of our apps should pick it up within the TTL time-span (15 minutes) and switch to the new address. In reality most of our web-apps are Java apps running on top of Tomcat. Whilst I initially disliked the black-box nature of tomcat/Java I’ve come to like it. It…
Just a quick blog post. One of the earlier tasks with our new junior sysadmin at work was setting up remote syslogging using splunk. It’s something that’s been sitting on my to-do list for a while but wasn’t a large priority. There are a number of reasons to do this. Single point to look at - You don’t have to log in to every box just to try to piece together whats…
With the payment industry’s push on security anyone processing credit card transactions is required to meet something called the Payment Card Industry - Data Security Standards (PCI-DSS). Depending on how much you’re processing and how depends on how much of the security standards you need to match. e.g. Paypal has to do a lot more than your mom & pop store would have to. There are…
As I mentioned in my previous blog post about a bug that caused some interesting behaviour in MySQL, clearing the bug presented us with a new problem. This time the bug isn’t MySQL end but a quirk in the application using the database. Whilst I’m not at liberty to go into the details of the application I can talk a bit about the symptoms and troubleshooting procedure. The day after I…
We have a relatively new web application that has gone live at work. It’s fairly complex and the database is large, or at least large compared to what we usually deal with (~40Gb). During initial setup and testing the decision was made to roll it out using MySQL 5.5 as the back end database, rather than the 5.1 that our linux distribution had packages for. Installing 5.5 is relatively…
Oh dear. Oh dear, oh dear, oh dear. I keep an eye on the field so I knew Unity was coming. I assumed whatever UI quirks it had would have a chance to be polished up over the months. After all it’s been used in Ubuntu Netbook Edition for a while. Today I upgraded my Maverick install to Natty beta. Yes I know it’s beta, and that means test, however we’re mere weeks away from…
Rather inspired by this I thought I’d share a similar nugget of stupidity. It would have been the year 2000, I was in my first full-time job and I was learning a lot about IT, fast. I was a major geek already by then, prettycompetentwith desktops. It was rare to find things that would stump me for any length of a time on a machine, but then I’d been using Windows for many years. It was…
Had one of our Oracle servers start working really hard today. Sometimes that happens when Oracle seems to be throwing some form of a fit, but usually that’s tens of processes all sucking up 2-3% of CPU time, and you can’t connect to sqlplus. This time it was actually just a single PID responsible for all the hard work. Using netstat you could trace it back to a server, but unless…
My folks were over from England to stay with Kari and I for a couple of weeks as we were unable to make it to England this year as originally planned. One of the things my parents expressed a desire to do was go visit Big Island (Hawai’i itself) and amongst those things go to see the volcano. Unfortunately we were unable to go see the lava flowing into the sea due to the levels of sulfur in…
Yesterday saw the primaries and the votes for various local posts here in Hawaii. From what figures I can find online Hawaii has an estimated population of 1.29m. Of that 1.29m, 684,481 (53%) are registered voters, and yesterday just 276,944 (21% of population, 40% of registered voters) bothered to vote. Way to go democracy. It isn’t dying a death to the sound of cheers or clapping,…
After storming through last season with an excellent team, only to fail epically in the final few weeks of the season thanks to clubs resting players for the post-season, and players trying hard not to be injured, I’m up for a challenge again this year. And I’m still rather clueless, not followed football much at all between seasons and mostly forgotten bits I picked up last year.…
Whilst the Choir takes a break for the summer various people have stepped forward to bring something extra musical to the service, generally during the offertory. It’s really shown some of the talent in the church and blessed those present in various ways. A couple of weeks ago was the turn for Kari and myself. We’d ummed and ahhed about what to do and settled on the song “Always…
I was at a Fellowship with church worship team ohana yesterday in lanikai, check out the view! View over Lanikai View 2 over Lanikai
Here’s something I haven’t had to think about much before. Multiple cores and their impact on monitoring. Multiple cores are great. The more I can get in a server for web hosting or web apps, generally the better. One thing that occasionally happens, though, is spinning processes. That’s when a process hits up against some problem, maybe a bug or just random chance, and it sits…
It’s been a while since I had to do this, but I needed to know what perl modules were installed on a system (or at least visible to perl by standard), and quickly knocked up this one liner so that I could dump the contents to a text file.: ```perl` perl -e ‘use ExtUtils::Installed; use Data::Dumper; my ($inst) = ExtUtils::Installed->new(); my (@modules) = $inst->modules(); foreach my…
Just a quick mini-blog on the subject. I’ve long wished for a progress bar when it comes to loading .sql files into MySQL. For the most part this is rather hard to do in a meaningful way. It’s nigh on impossible for a program to guess how long each query in the file is going to take. That said, even something that evaluates the bit or line progress would be good, even if the ETA is…
Last week saw my last day of employment at Superb Internet, a day tinged with sadness and joy both. Sad because I’ve left behind a wonderful bunch of colleagues, but joy because of what I’m moving on to. At Superb I had the distinct pleasure of dealing with a significant number of very capable tech support and sysadmin staff on a wide variety of problems and solutions. I consider…
At Christmas I set myself a deadline of the end of January to make a cat tree / condo. Buying one from a pet store would set us back over $150 for something fairly small, and I really didn’t fancy spending that much on one. During the month I’ve been googling around and looking for ideas, then this weekend, leaving it to the last one of the month naturally, Kari and I set off to the…
Wow that was annoying. I’ve just spent literally hours trying to find out why my usual Plesk hardening steps (based on these) weren’t working and why 8443 was STILL using weak SSL ciphers. I’ve altered files galore, created new ones, googled and all to no avail. Then I started doing what I would have done sooner if I’d quite twigged what changes Plesk have made recently to…
One of the regular things that crop up in my job are requests from customers to help them with their slow performing website. Customers start talking about clustering and load-balancing, and tweaking their server for the best performance. All of those are valid options, and there are advantages for sure, particularly when you start considering High Availability (0% downtime), but it’s not…
A few months back we had a camera crew around taking videos from around the office and interviewing senior staff. The resulting video clip got some air play here in Hawaii on a local station, OC16, enough so that some members of staff were being recognised out on the streets. You can check out the video here Of course, you can only see the back of my head, and from the looks of things I was…