Understanding SMF properties

 Working through an example inspecting SMF properties in various situations. 

Dave Pacheco's Blog
Recent content on Dave Pacheco's Blog
Live Last read · last published · next check
Latest posts
My workflow for stacked PRs on GitHub

 My best attempt so far at an ergonomic workflow that supports incremental review but still results in squashed commits on 'main'. 

Developing for Debugging

 Using DNS propagation in the Oxide control plane as an example of developing for debugging 

Challenges deploying PostgreSQL (9.2) for high availability
A detailed post about experiences trying to keep a fleet of large, high-volume PostgreSQL databases online.
All about swap on illumos systems

 All about swap on illumos systems. Part 2 of 2 in a series on how illumos manages physical memory and swap. 

Physical memory on illumos systems

 An overview of how illumos systems manage physical memory. Part 1 of 2 in a series on how illumos manages physical memory and swap. 

Shell redirection example
Explaining some surprising bash behavior.
Modifying USDT providers with translated arguments
Diving deep on a bug in the Node.js USDT translators.
Performance Puzzler: The Slow Server
What happens when one server in a distributed system gets slow?
Visualizing PostgreSQL Vacuum Progress
As heavy users of PostgreSQL since 2012, we’ve learned quite a bit about operating PostgreSQL at scale. Our Manta object storage system uses a large fleet of sharded, highly-available, replicated PostgreSQL clusters at the heart of the metadata tier. When an end user requests their object, say http://us-east.manta.joyent.com/dap/public/kartlytics/videos/2012-09-06_0000-00.mov, Manta winds up…
TCP puzzlers
It’s been said that we don’t really understand a system until we understand how it fails. Despite having written a (toy) TCP implementation in college and then working for several years in industry, I’m continuing to learn more deeply how TCP works – and how it fails. What’s been most surprising is how basic some of these failures are. They’re not at all…
Programming language debuggability
A few weeks ago I had the fortune to attend a talk by Douglas Crockford on various ideas related to JavaScript past and future and programming language design. The talk was thoughtful and well-received (and thanks to Crockford and Bigcommerce for making it happen!). The main theme, not new to those familiar with Crockford’s work, was that as programmers, we have a responsibility to create…
Debugging enhancements in Node 0.12
(also on the Joyent blog)
Background Node.js provides the richest postmortem debugging facilities of just about any dynamic language. Postmortem debugging refers to debugging programs from a core file, heap dump, or some similar dump of memory. With a core file of a Node program (which is essentially a snapshot of your program’s state), you can:
Find all JavaScript objects and…
Error Handling in Node.js

 Summary of physical memory and swap space on illumos systems. Part 3 in a 3-part series. 

Understanding DTrace ustack helpers
or, everything you ever wanted to know about stack traces I promised this post over a year ago, and now that someone’s actually working on a new ustack helper, I thought it was finally time to write about what ustack helpers are, how they work, and how I went about building one for Node.js. Only a handful of ustack helpers have ever been written: Node, Java, Python, and PHP (the last of…
Tracing Node.js add-on latency
Node.js has some built-in DTrace probes for looking at HTTP request start/done and GC start/done, and I wrote nhttpsnoop to report latency based on these probes. There’s also a future project that will make it easy to trace any asynchronous operations. But in the meantime, there’s one category of latency that’s missing, which is latency of asynchronous operations kicked off by…
Stopping a broken program in its tracks
Last week I debugged a Node issue where under some conditions, fork() failure could result in some file descriptors being closed, leading shortly to bedlam. In the best cases, the program would immediately crash, but often it would just do the wrong thing. This happens a lot, of course, and in many ways it’s worse than a crash because you have no core dump nor even a stack trace to go on.
Node.js in production: runtime log snooping
This post also appeared on the Joyeur blog.
This post is the one of several about how we operate Node.js in production at Joyent. Most of my experience comes from working on our Manta Storage Service, which currently comprises a few thousand Node processes across three datacenters.
Logging is one of the most primitive but most valuable forms of postmortem debugging. Logs let you figure out…
Kartlytics: Applying Big Data Analytics to Mario Kart 64
This post also appears on the Joyeur blog.
If you missed it, Joyent recently launched Manta, a web-facing object store with compute as a first-class operation. Manta makes it easy to crunch on Big Data in the cloud, and we’ve seen it used by both ourselves and others and others to solve real business problems involving Big Data. But it’s not just for user behavior and crash dump…
Fault tolerance in Manta
Since launching Manta last week, we’ve seen a lot of excitement. Thoughtful readers quickly got to burning questions about the system’s fault tolerance: what happens when backend Manta components go down? In this post, I’ll discuss fault tolerance in Manta in more detail. If anything seems left out, please ask: it’s either an oversight or just seemed too arcane to be…
Inside Manta: Distributing the Unix shell
Photo by Jackie Reid, NOAA
Today, Joyent has launched Manta: our internet-facing object store with compute as a first class operation. This is the culmination of over a year’s effort on the part of the whole engineering team, and I’m personally really excited to be able to share this with the world. There’s plenty of documentation on how to use Manta, so in this post I want…
Debugging dynamic library dependencies on illumos
In this short follow-up to my post on illumos process tools, I’ll expand a bit on ldd and pldd, which print the dynamic linking dependencies of binaries and processes, respectively, and crle, which prints out the runtime linker configuration. These tools are available in most illumos distributions including SmartOS.
Understanding builds (and broken builds in particular) can be especially…
illumos tools for observing processes
illumos, with Solaris before it, has a history of delivering rich tools for understanding the system, but discovering these tools can be difficult for new users. Sometimes, tools just have different names than people are used to. In many cases, users don’t even know such tools might exist.
In this post I’ll describe some tools I find most useful, both as a developer and an…
OSCON Slides
Thanks to all who attended my talk at OSCON on Node.js in production: postmortem debugging and performance analysis. Just a reminder: all of the technology I described is open source, most of it part of illumos. For more info, check out the links in slide 22.
For the curious, there are also some slides on implementation details I didn’t have time to cover.
NodeConf slides
NodeConf was a great success this year. Thanks to @mikeal for organizing and everyone who spoke and attended. The slides from my talk on DTrace, Node.js, and Flame Graphs are here.
ACM Turing Centenary Celebration
This past weekend, I was very fortunate to have a chance to attend the ACM’s Turing Centenary Celebration in honor of the 100th anniversary of the birth of Alan Turing. The event brought together nearly every living Turing Award winner for a series of talks and panel discussions on subjects like AI, theory of computation, computer architecture, and the role of computation in other fields. A…
Debugging Node.js in Production (Fluent slides)
Thanks to all who attended my talk at Fluent today on “Debugging Node.js in Production”. The slides are available here. There’s some extra content there that I didn’t have time to cover in just 40 minutes, most notably some implementation notes about mdb_v8 and the DTrace ustack helper. Please leave questions, comments, or feedback below (or @dapsays)!
Debugging RangeError from a core dump
Last week, I tweeted:
I had just run into this nasty Node.js error:
$ node foo.js timers.js:96 if (!process.listeners('uncaughtException').length) throw e; ^ RangeError: Maximum call stack size exceeded What went wrong? It was reasonably obvious from the error message that the program blew its stack, which I assumed was likely the result of some errant recursive function, which was…
Profiling Node.js
(For returning readers, this is basically a “tl;dr” version of my previous post on Node.js performance. The post below also appears on the Node.js blog.)
It’s incredibly easy to visualize where your Node program spends its time using DTrace and node-stackvis (a Node port of Brendan Gregg’s FlameGraph tool):
Run your Node.js program as usual.
In another terminal,…
Managing Node.js dependencies with shrinkwrap
This post also appears on the Node.js blog.
Photo by Luc Viatour (flickr)
Managing dependencies is a fundamental problem in building complex software. The terrific success of github and npm have made code reuse especially easy in the Node world, where packages don’t exist in isolation but rather as nodes in a large graph. The software is constantly changing (releasing new versions),…
Playing with Node/V8 postmortem debugging
2016 Update: The commands and output in this post are pretty dated now, but you can find an up-to-date docs (including a tutorial and instructions for updated binaries) in the mdb_v8 user guide.
“Post Mortem” by C. MacLaurin
Several weeks ago I posted about postmortem debugging for Node.js, a critical technique for understanding fatal software failure (and thereby keeping up…
Where does your Node program spend its time?
Photo by Julian Lim (flickr)
Performance analysis is one of the most difficult challenges in building production software. If a slow application isn’t spending much time on CPU, it could be waiting on filesystem (disk) I/O, network traffic, garbage collection, or many other things. We built the Cloud Analytics tool to help administrators and developers quickly identify these sources of…
USDT Providers Redux
In this post I’m going to review DTrace USDT providers and show a complete working example that I hope will be a useful reference for people interested in building providers for their own applications.
First, the prerequisites:
DTrace is the comprehensive dynamic tracing framework available on Illumos-based, BSD, and MacOS systems. If you’ve never used DTrace, check out…
Node.js/V8 postmortem debugging
2016 Update: The commands and output in this post are pretty dated now, but you can find an up-to-date docs (including a tutorial) in the mdb_v8 user guide.
Photo by Auntie P (flickr)
I recently wrote an article for ACM Queue discussing postmortem debugging for dynamic environments. I argued that while native environments for years have provided rich tools for understanding software…
Surge 2011
We had a great time last week attending Surge in Baltimore. Highlights for me included Baron Schwartz’s visualizations of MySQL execution time (not entirely unlike Brendan’s, but with the addition of modeling), Geir Magnusson’s discussion of scalability at Gilt, and Raymond Blum’s discussion of backup/restore at Google (including recovery after the GMail outage from several…
New metrics on no.de
You may have noticed that the no.de service got a big facelift last week. The new version of the software has a lot of new features, among them some pretty substantial improvements to Analytics. Recall that we already had metrics for Node.js HTTP server and client operations, garbage collection, socket ops, filesystem ops, and CPU executions. We’ve now got a slew more.
First we have…
JavaScript Lint on SmartOS
Photo by Sam Fraser-Smith
Back at Fishworks, we used a tool called JavaScript Lint (JSL) for static code analysis. You may know that lint was originally written to identify potential semantic problems in C code, like use of uninitialized variables or blocks of unreachable code. Lint warnings are usually static checks that could reasonably have been compiler warnings, but for whatever reasons…
Distributed Web Architures @ SF Node.js Meetup
At the Node Meetup here at Joyent’s offices a few weeks ago I gave a brief talk about Cloud Analytics as a distributed web architecture. Matt Ranney of Voxer and Curtis Chambers of Uber also spoke about their companies’ web architectures. Thanks to Jackson for putting the videos together. All around it was a great event!
OSCON Slides
I’ve posted the slides from Brendan’s and my talk yesterday at OSCON Data called “Design and Implementation of a Real-Time Cloud Analytics Platform.” Thanks to everyone who attended. We’d love to get your feedback! And thanks to @adrianco for this photo of our demo.
Heatmap coloring
Brendan has written a great 5-part series on filesystem latency. Towards the end of part 5, he alluded to a lesser-known feature of Cloud Analytics heatmaps, which is the ability to tweak the way heatmaps are colored. In this post I’ll explain heatmap coloring more fully. Keep in mind that this is a pretty advanced topic. That’s not to say that it’s hard to understand, but rather…
Heatmaps and more heatmaps
I’ve talked with a few Cloud Analytics users who weren’t sure how to interpret the latency heatmaps. While we certainly aren’t the first to use heatmaps in performance analysis, they’re not yet common and they can use some explanation the first time you see them. I posted a simple example a few weeks ago, but video’s worth a thousand words so I’ve made a…
Presenting at OSCON Data 2011
Brendan’s and my proposal for OSCON Data 2011 has been accepted! The talk is called “Design and Implementation of a Real-Time Cloud Analytics Platform”. We’ll dive into the implementation of Cloud Analytics with particular focus on the design considerations that allow it to work in real-time on a large production cloud.
We’ll also cover some real-world case…
Example: HTTP request latency and garbage collection
A few weeks ago I posted about Cloud Analytics on no.de. I described the various metrics we included in the initial launch and how they’re relevant for Node applications. Today I’ve got a small example using HTTP request latency and garbage collection. I wrote a small HTTP server in Node.js that simply echoes all requests, but in doing so deliberately creates a bunch of garbage. That…
Welcome to Cloud Analytics
We’ve been talking for several weeks now about our work on Cloud Analytics. Today, we’re showing the world what we’ve put together. Now available on Joyent’s Node.js Service: a first look at Cloud Analytics in action – on your very own Node.js application.
Cloud Analytics (CA for short) is a tool for real-time performance analysis of production systems and…
Tonight at 6: Solving Big Problems (with Cloud Analytics)
Here’s just a quick note to say that Joyent is hosting a talk tonight called “Solving Big Problems with Real-time Visibility into Cloud Performance.” Bryan and Brendan will be showing off our Cloud Analytics tools and discussing how we’ve used them already to analyze actual customer performance problems in real time. If you can’t make the live talk, check out the live…
Joining Joyent
It’s been almost two months since I announced my departure from Oracle, but I didn’t say what I was doing next. After taking several weeks off to decompress, I’ve joined Joyent to work with Bryan and Brendan on tackling the problem of observability in the cloud. We’re working on a project we call Cloud Analytics, which will provide an interface for dynamically instrumenting…
Leaving Oracle
I first came to Sun over 4 years ago for an internship in the Solaris kernel group. I was excited to work with such a highly regarded group of talented engineers, and my experience that summer was no disappointment: I learned a lot and had a blast.
After college, I joined Sun’s Fishworks team. Despite my previous experience at Sun, I didn’t really know what to expect here, and I…
SS7000 Software Updates
In this entry I’ll explain some of the underlying principles around software upgrade for the 7000 series. Keep in mind that nearly all of this information are implementation details of the system and thus subject to change.
Entire system image One of the fundamental design principles about SS7000 software updates is that all releases update the entire system no matter how small the…
Replication for disaster recovery
When designing a disaster recovery solution using remote replication, two important parameters are the recovery time objective (RTO) and the recovery point objective (RPO). For these purposes, a disaster is any event resulting in permanent data loss at the primary site which requires restoring service using data recovered from the disaster recovery (DR) site. The RTO is how soon service must be…
Another detour: short-circuiting cat(1)
What do you think happens when you do this:
# cat vmcore.4 > /dev/null If you’ve used Unix systems before, you might expect this to read vmcore.4 into memory and do nothing with it, since cat(1) reads a file, and “> /dev/null” sends it to the null driver, which accepts data and does nothing. This appears pointless, but can actually be useful to bring a file into memory, for…