Twice the Price

In Exceptional Behavior we got to see a bare-bones mach message to mig server loop in the function exception_server. To summarize the function, mach_msg is used to receive a message, it’s passed along to mach_exc_server for handling, the return value is negligently ignored, and finally the reply is sent out also using mach_msg. Since these interfaces hail from the late 1980’s (and earlier), compute time was at an absolute premium, and syscalls were anything but cheap. This was factored in, and, to paraphrase Kevin Malone, “Why waste time make two syscall when one syscall do trick?”

A common optimization in these sorts of designs is the use of mach_msg_overwrite rather than 2 mach_msgs. mach_msg_overwrite sends a reply message, and then waits until a new message is received (or an optional timeout elapses). In the case of an error on send, the wait isn’t performed and no incoming messages are dequeued so the error can be handled/logged.

Here’s how the modified loop would look:

static void exception_server (mach_port_t exceptionPort)
{
    mach_msg_return_t rt;
    mach_msg_header_t *msg;
    mach_msg_header_t *reply;
    msg = malloc(sizeof(union __RequestUnion__mach_exc_subsystem));
    reply = malloc(sizeof(union __ReplyUnion__catch_mach_exc_subsystem));
    reply->msgh_size = 0;
    printf("server starting...\n");
    bool haveReply = 0;
    while (1) {
        mach_msg_options_t opts = MACH_RCV_MSG;
        if (haveReply)
            opts |= MACH_SEND_MSG;
        rt = mach_msg_overwrite(reply,
                                opts,
                                reply->msgh_size,
                                catch_mach_exc_subsystem.maxsize,
                                exceptionPort,
                                MACH_MSG_TIMEOUT_NONE,
                                MACH_PORT_NULL,
                                msg,
                                catch_mach_exc_subsystem.maxsize);
        assert(rt == MACH_MSG_SUCCESS);
        // Call out to the mach_exc_server generated by mig and mach_exc.defs.
        // This will in turn invoke one of:
        // mach_catch_exception_raise()
        // mach_catch_exception_raise_state()
        // mach_catch_exception_raise_state_identity()
        // .. depending on the behavior specified when registering the Mach exception port.
        if (mach_exc_server(msg, reply))
            haveReply = 1;
        else
            haveReply = 0;
    }
}

We can also remove the extern boolean_t mach_exc_server (mach_msg_header_t *msg, mach_msg_header_t *reply); line, because that’s defined in one of the mig header’s we’re including.

Instead of another costly syscall, we now just track a bool to tell the kernel whether or not our send buffer has anything useful to say. That bool is derived from the return value of the mig server.

In some designs, mostly from days of yore before libdispatch and friends, you could also hijack the reply message’s remote port (msgh_remote_port), where the reply goes, and store that for a later reply for long-running callback handling without stalling the server from handling other messages in the interval. This required special handling, and a fallback path to mach_msg to send the orphaned reply.

Ports and port sets can have multiple threads waiting to dequeue messages from them, and when a message is received, exactly one is woken up with the message. If your server handlers are all thread-safe, you can fire up several threads running your mig server concurrently for even higher message throughput. Sadly, neither libxpc nor libdispatch appear to use mach_msg_overwrite, so perhaps its benefit has faded into the distant past by now. It’s also possible for mach_msg to both send and receive in a single shot, with extra buffer management required to track which buffer holds which state (send or receive); perhaps that’s what they do instead.

You can also design multiple subsystems (different mig files with different message numbers), and you can chain the servers together, passing the message to each one until one of them services it. Or you can peek at the message’smsgh_id yourself and route it to the correct handler directly. This maintains a similar design with mach_msg or mach_msg_overwrite, but with additional server calls for actually handling the message. Regarding message IDs: notice near the top of mach_exc.defs there’s this:

subsystem
...
             mach_exc 2405;

That tells mig that messages in this subsystem start with the ID 2405; by convention, replies are the message ID + 100. You can spot these in the mig-generated headers, with pieces like this:

#ifndef subsystem_to_name_map_mach_exc
#define subsystem_to_name_map_mach_exc \
    { "mach_exception_raise", 2405 },\
    { "mach_exception_raise_state", 2406 },\
    { "mach_exception_raise_state_identity", 2407 }
#endif

To spot the +100 replies, you’ll have to check out the mig-generated C code, which is a little clunkier to read through.

watcher% grep "250" -R *
build/watcher.build/Release/watcher.build/DerivedSources/arm64/mach_excUser.c:  if (Out0P->Head.msgh_id != 2505) {
build/watcher.build/Release/watcher.build/DerivedSources/arm64/mach_excUser.c:  if (Out0P->Head.msgh_id != 2506) {
build/watcher.build/Release/watcher.build/DerivedSources/arm64/mach_excUser.c:  if (Out0P->Head.msgh_id != 2507) {

and where the server accomplishes this mathematical feat:

build/watcher.build/Release/watcher.build/DerivedSources/arm64/mach_excServer.c:    OutHeadP->msgh_id = InHeadP->msgh_id + 100;

This is a lot of moving parts to keep in mind. Nonetheless, it’s fascinating to think about how forward-thinking this design was for the era that produced it. Concepts like ports and messages in this context can trace their lineage through Accent in the late 1970’s, and even into Aleph in 1975. Many of these primitives are over half a century old!