For you, I'll make an exception

Looking back at the Exceptional Behavior demo and pairing it with the MIG stuff from even further back gave me an idea. Since the MIG stuff can generate both a client and a server, could we do something … surprising?

In normal circumstances, the exception handler machinery is driven by the kernel. That is to say, the application handling the exception is considered the “server” and the thing making the calls (the kernel in this case) is the “client.” But that’s not the only arrangement. Anyone that has a send or send-once right can drive our interface. And that includes ourselves, if we build both sides of the interface. Once the security implications of this were understood, Audit Trailers (sometimes called Audit Tokens) were tacked on (around 10.3 I think, at least publicly), and those are kernel-mediated only so they can’t be forged, but that’s for yet another article in my infinite regression of article ToDos. Anyway: We Can Send A Message To Us.

Now, that may seem silly, but it allows us to do a few unconventional things. For starters, it lets us send exception types that don’t even exist, which can then drive some fun hallucinatory and self-referential explanations: Aside from humor, this can be useful for fuzzing or driving otherwise-uncommon events manually in a test harness. Since we control all the inputs to mach_exception_raise, we can deliver whatever we want to our server, including normally-invalid exception types and codes.

A more advanced application of this enables answering a problem I’ve been trying to find a reasonable(?) answer to for ages: How Many Instructions Did This Piece Of Code Execute? I’m sure I could learn another programming language and environment to make lldb do something along these lines, but 1) I hate python and 2) I don’t script lldb often enough to remember much of anything, so it burns a lot of time every time I need to do it. Being able to wire up a couple probes makes for a much simpler solution.

The exception handlers are routines in mig, not simpleroutines, so the calling thread is stopped until the exception handler returns. For extra safety with thread state, we can then thread_suspend() it and then read/modify its thread state. Except instead of modifying normal GPRs or NEON registers, we can twiddle debug registers to put a thread into single step mode (Just Like A Real Debugger!), and get an ordinary breakpoint exception (EXC_BREAKPOINT) after that instruction is executed. Single Step is only good for a single instruction, but we can re-arm it in the handler and keep track of how many times our handler has run to get an exact number of instructions. While we’re there, we have the program counter, so we can peek at what the instruction is (macOS doesn’t support XOM, otherwise this gets much trickier) and do some basic classification while we’re in the neighborhood.

All that’s left are the probes themselves to indicate when to start and stop tracking. Those can be “exceptions” with sentinel values, which the handler uses to drive its little state machine and control the counter/classifier.

You can find the full project here. Note that I only did this for ARM64, though it’s possible to do for x86_64 as well. This showcases a few fun tricks in mach, such as atomically swapping exception handler state and later restoring it, handling port notifications, making and managing a port set, and some crazy assembly trampolines to help the instruction counter filter out instructions in the profiler itself. There’s a commented out block where you can print a trace of instructions, if you want to get a lot of terminal output.

It’s ~750 lines in a single source file, and I stupidly thought C++ vectors would make for less array management (that turned into a few hundred lines of move constructors and nonsense), but the core functionality should be almost intelligible. The two main entry points, begin and end showcase some unconventional ways to abuse the data parameters mig transmits to smuggle sentinels, pointers, and flags to and from the exception handler without making additional interfaces or mig routines.

With this, I can finally get interesting answers like the following:

cos(42) takes 61 instructions (25 simd/fp instructions and 8 branches)
sin(42) takes 62 instructions (26 simd/fp instructions and 9 branches)
tan(42) takes 73 instructions (36 simd/fp instructions and 8 branches)

(instruction counts vary a little bit between runs; I haven’t tried with tracing to figure out why that is, and it may indicate a bug in my implementation. There’s definitely room for more advanced classification as well, and maybe even an instruction cache line tabulator using roaring bitmaps or something).

It may be easy to integrate this into a continuous integration suite to look for notable regressions, or even find regressions in platform code.

In the interest of full disclosure, I’ve had a primitive form of this for quite a while (it’s useful for reverse engineering), but not in any form that was suitable for release. The confluence of recent articles and some down time pushed me to clean it all up and make a releasable project out of it. It also pushed me to do it more cleanly (saving and restoring state, not leaking memory, having some unit tests, etc).

By firing breakpoints all the time, you might imagine that attached debuggers get really grumpy about it. In that, you’d be totally correct. Crashing inside a probe might be interesting; I’m not sure if crash reporter would notice properly or not (it might just enter a busy loop, retrying the faulting instruction?). Checking the exception flags to see if it’s a single-step or not might be a prudent improvement, come to think of it.

Also of note, instruction counting isn’t a great proxy for real life performance. Modern reorder buffers, caches, and branch predictors decouple instruction count and wall clock performance. Even so, executing fewer instructions tends to be nicer for the instruction cache, and may use less power or produce less heat. Certain classifications can also help spot if code was vectorized as expected, or otherwise optimized as intended.