In code we trust.
There's a CVE on GCP that could lead to a privilege escalation (see security bulletin GCP-2023-001 ). This can be mitigated by blocking the affect syscall with a seccomp profile. Unfortunately Kubernetes doesn't make it easy to deploy a profile, you can only reference a file under the kubelet's root directory, and GKE doesn't provide an easy facility to deploy those files to all the nodes in the…
Consider the following excerpt of a Dockerfile building nginx: RUN \ mkdir -p /usr/src/ngx_brotli \ && cd /usr/src/ngx_brotli \ && git init \ && git remote add origin https://github.com/google/ngx_brotli.git \ && git fetch --depth 1 origin $NGX_BROTLI_COMMIT \ && git checkout --recurse-submodules -q FETCH_HEAD \ && git submodule update --init --depth 1 \ && cd .. \ && curl -fSL…
I spent a bunch of time Googling how to do this so I figured it could help someone else if posted the steps to add an admin account on a Kubernetes cluster managed with kops . k8s has service accounts but that's not what you want to create an admin account — equivalent to having root privileges on the cluster. Instead you simply need to create a certificate/key pair for the user and sign it with…
edit: I rejoined Arista Networks in early 2020 ;) 5 years ago, I wrote a blog post on why I joined Arista Networks back in 2012. As I am now suddenly and unexpectedly leaving the company, I figured I'd write a bit of a retrospective and perhaps bring some closure to this otherwise fairly quiet blog. I know that the original blog post has been used by candidates considering to join Arista, and even…
I haven't posted anything here in a while, just been busy with life and hating Blogger's interface (and being too lazy to move to something else). But I wanted to share some of what I've learned recently on the ways one can get liquidity, because I've run into too many people who told me "damn I wish I'd known this earlier!". Disclaimer: This post, or anything else on this blog, is not financial /…
Over the past few months, many people have asked me why I jumped from the "web world" to the "network industry" to work at Arista Networks . I asked myself this question more than once, and it was a bit of a leap of faith, but here's why I did it, and why I'm happy I did it. Choosing a company to work for There is a negative unemployment rate in Silicon Valley provided you know how to type on a…
I just stumbled on one of these for the first time: Here's another one: These were on some Blogger blogs. Looks like Google is using captchas to help improve StreetView's address extraction quality.
If you attempt to use debootstrap with grsec (more specifically with a kernel compiled with CONFIG_GRKERNSEC_CHROOT_MOUNT=y ), you may see it bail out because of this error: W: Failure trying to run: chroot path/to/root mount -t proc proc /proc One way to work around this is to bind-mount procfs into the new chroot. Just apply the following patch before runnning debootstrap : ---…
Just in case this ever helps anyone else, I had a machine where slave lag (as reported by Seconds_Behind_Master in SHOW SLAVE STATUS ) would sometimes suddenly jump to 7 hours and then come back, and jump again, and come back. Turns out, the machine's clock was off by 7 hours and no one had noticed! After fixing NTP synchronization, the issue remained, I suspect that MySQL keeps a base timestamp…
Doing this in Python is bad bad bad: try: # some code except Exception, e: # Bad log.error("Uncaught exception!", e) Yet you need to do something like that, typically in the event loop of an application server, or when one library is calling into another library and needs to make sure that no exception escapes from the call, or that all exceptions are re-packaged in another type of exception. The…
I've used Perforce for 2 years at Google. Google got a lot of things right, but Perforce has always been a pain in the ass to deal with, despite the huge amount of tooling Google built on top. I miss a lot of things from my days at Google, but Perforce is definitely not on the list. Isn't it ironic that for a company that builds large distributed systems on commodity machines, their P4 server had…
Today I'd like to rant on one thing that's been bugging me for last couple years with Apache Hadoop (and all its derived projects). It's a big issue that concerns us all. We have to admit it, each time we write code for the Apache Hadoop stack, we feel bad about it, but we try hard to ignore what's happening right before our eyes. I'm talking, of course, about the constant abuse and molestation of…
If you've read code I wrote, chances are you've seen I'm a strong adept of const correctness ( WP ). Naturally, when I started writing Java code (to my despair), I became equally adept of "final correctness". This is mostly because the keywords const (C/C++) and final (Java/Scala) are truly here to help the compiler help you. Many things aren't supposed to change. References in a given scope are…
Just in case this could be of any use to someone else, I compiled Growl 1.2.2 for Lion with the fix for HardwareGrowler crash on Lion that happens when disconnecting from a wireless network or waking up the Mac. You can download it here . The binary should work on Snow Leopard too. It's only compiled for x86_64 CPUs.
For a lot of people, the conventional wisdom is that XFS outperforms ext4. I'm not sure whether this is just because XFS used to be a lot faster than ext2 or ext3 or what. I don't have anything against XFS, and actually I would like to see it outperform ext4, unfortunately my benchmarks show otherwise. I'm wondering whether I'm doing something wrong. In the benchmark below, the same machine and…
These days, the Hitachi 7K3000 seems like the best bang for your bucks. You can get 2TB disks for around US$100. The 7K3000 isn't an "enterprise disk", so many people wouldn't buy it for their servers. It's not clear what disks sold with the Enterprise™©® label really do to justify the big price difference. Often it seems like the hardware is exactly the same, but the firmware behaves differently,…
XFS has terribly bad performance out of the box, especially on large RAID arrays. Unlike ext4, the filesystem needs to be formatted with the right parameters to perform well. If you don't get the parameters right, you need to reformat the filesystem as they can't be changed later. The 3 main parameters are: agcount : Number of allocation groups sunit : Stripe size (as configured on your RAID…
At StumbleUpon we've had a never ending string of problems with Broadcom's cards that use the bnx2 driver. The machine cannot handle more than 100kpps (packets/s), the driver has bugs that will lock up the NIC until it gets reset manually when you use jumbo frames and/or TSO (TCP Segmentation Offloading). So we switched everything to Intel NICs. Not only they don't have these nasty bugs, but also…
If you ever run into this message while using the Sun JVM / OpenJDK: Java HotSpot(TM) 64-Bit Server VM warning: GC locker is held; pre-dump GC was skipped then I wouldn't worry too much about it as it seems like it's printed when running a jmap -histo:live while the GC is already running or holding a certain lock in the jVM.
After reading the excellent post on The MySQL “swap insanity” problem and the effects of the NUMA architecture , I remembered about the existence of /sys/devices/system/node/node*/numastat and decided to add these numbers to a collector for OpenTSDB . But whenever I add a collector that reads metrics from /proc or /sys , I always need to go read the Linux kernel's source code, because most metrics…
At StumbleUpon we've been tracking down a weird problem with one of our application servers written in Java. We run Sun's jdk1.6.0_24 on Ubuntu Jaunty (9.04 – yes, these servers are old and due for an upgrade) and this application seems to do something that causes the JVM to segfault: [6972247.491417] hbase_regionser[32760]: segfault at 8 ip 00007f26cabd608b sp 00007fffb0798270 error 4 in…
I recently did some work on some of our frontend machines (on which we run Varnish ) at StumbleUpon and decided to track down some of the errors the Linux kernel was regularly throwing in kern.log such as: Feb 25 08:23:42 foo kernel: [3077014.450011] Out of socket memory Before we get started, let me tell you that you should NOT listen to any blog or forum post without doing your homework ,…
I have a fairly simple problem: I want to count the number of lines in a file, then seek back to after the first line, and then read the file line by line. Easy heh? Not in Java. Enter the utterly retarded world of the JDK. So if you're n00b, you'll start with a FileInputStream , but quickly you'll realize that seeking around with it isn't really possible... Indeed, the only way to go back to a…
I will be speaking about OpenTSDB at the Strata conference , Wednesday, February 02, 2011, in Santa Clara, CA. You can sign up with this promo code and get a 25% discount: str11fsd. Strata is a new conference about large scale systems put together by O'Reilly.
That's a interesting question I'm willing to spend some of my time on. Someone at StumbleUpon emitted the hypothesis that with all the improvements in the Nehalem architecture (marketed as Intel i7), context switching would be much faster. How would you devise a test to empirically find an answer to this question? How expensive are context switches anyway? (tl;dr answer: very expensive ) The…