Nexus Shell Engineering Notes

Implementation notes from the developers of Nexus Shell, a native macOS SSH workspace.

An interactive SSH tab can outlive the transport underneath it. The window is still open, the scrollback is still useful, and the user still thinks of it as the same terminal. But the TCP connection may have disappeared during sleep, a network change, a server restart, or an idle timeout.

The obvious recovery behavior is to reconnect when the user presses a key. The less obvious question is what to do with that key.

Replaying it into the new shell feels convenient. It is also surprisingly dangerous.

A tab and a transport are different lifetimes

The terminal tab is a logical session owned by the user interface. The SSH process or in-process protocol session is only the current transport handle feeding that tab.

Keeping those identities separate makes recovery possible. When a transport exits, the tab does not need to disappear. Its scrollback, title, server association, saved dimensions, and session log can remain. Only the mapping from the logical tab to the dead transport is removed.

That distinction also prevents stale callbacks from damaging a newer connection. A disconnect callback must identify the handle that died and clear state only if the tab still points to that handle. Otherwise, a delayed callback from the old process can arrive after reconnection and accidentally tear down the replacement.

Why the triggering key should not be replayed

Suppose a disconnected tab receives Enter. The client notices that no live handle exists and starts a new SSH connection. If it buffers and replays Enter after authentication, the fresh shell receives an empty command immediately after drawing its first prompt.

That is mostly cosmetic. Other keys are not.

A terminal can emit several input events before reconnection finishes. A user may type part of a command while the interface still looks connected. Replaying those bytes into a newly authenticated shell changes their context. The old remote process is gone, the working directory may have changed, and the prompt may now belong to a different login environment.

The conservative rule is simple:

  1. The first input event may trigger reconnection.
  2. Every input event received while that reconnect is in flight is consumed.
  3. Input resumes only after the new transport is established.

This makes the boundary visible. The user gets a new prompt and can decide what to run, instead of having stale input executed automatically.

It also keeps input tracking honest. The triggering Enter is not a command character and should not appear in command history or session metadata.

Coalesce reconnect attempts per logical tab

Input events can arrive faster than an SSH handshake completes. Without an in-flight marker, each event can observe the missing handle and start another connection.

The result is not merely wasted work. Two PTYs may authenticate successfully, both may attach output callbacks, and input can be routed to one while output comes from the other. A multiplexing layer can make the race even harder to notice because both processes may share one underlying TCP connection while still representing different interactive shells.

We keep a set of logical session identifiers currently reconnecting. The marker is installed before the first suspension point. Later input joins the existing attempt and waits for its result instead of starting another one.

There is a second race during initial connection. Opening a tab may launch the handshake in a separate task, so the session can be marked as connecting before its handle is stored. Input arriving in that window waits for the in-flight connection for a bounded period. It does not interpret a temporarily missing handle as permission to create another PTY.

Bind output to a transport epoch

Each successful connection creates a new transport epoch for the logical tab. The output callback is registered against both the tab identifier and the exact transport handle.

When a new epoch replaces an old one, the old output callback is removed and the old handle is disconnected. This prevents bytes from different PTYs from being interleaved in one terminal emulator.

The visual boundary matters too. Shell prompts often have no trailing newline. If retained output from the previous epoch ends mid-line, the new prompt can be drawn over it or trigger shell-specific end-of-line markers. Before reconnecting, the client inserts a line break only when the retained output does not already end with a carriage return or newline.

Session recording should preserve the same boundary. A disconnect marker closes the old epoch, while the next successful registration becomes a reconnect marker rather than pretending the session was uninterrupted.

A local process being alive is not enough

Sleep and network changes create a particularly awkward failure mode on macOS: the local ssh process can still exist after its TCP transport has become unusable.

Checking only the process identifier is insufficient. Even ssh -O check proves mainly that the local multiplexing master is running; it does not require a round trip to the server.

For the direct-download transport, we validate an existing ControlMaster by opening a non-interactive command channel through its socket and running true. Authentication fallback is disabled with batch mode, zero password prompts, and no preferred authentication methods. If the socket is stale, the probe must fail rather than silently opening a new login connection.

The wake path waits briefly before probing because Wi-Fi may not have reassociated when the system wake notification arrives. It then checks sessions serially. Healthy multiplexed connections answer quickly, while dead connections consume the timeout. Serial validation avoids creating a burst of probes at the exact moment the network stack is recovering.

If the probe fails, the handle is invalidated. The next user input follows the normal reconnect path.

Reconnection still has security boundaries

Automatic recovery must not bypass host-key policy. Before opening the replacement transport, the client performs the same host-key trust check used by an initial connection. A changed key remains a decision point; reconnecting is not permission to accept it silently.

Authentication and reachability failures are also not retried forever. A failed reconnect records the error and leaves the session in a failed state. Repeated background retries would create noise, lock accounts, or hide a configuration problem.

A write failure on a supposedly live PTY is treated differently. It is direct evidence that the cached handle is unusable. The client removes that handle and starts one coalesced reconnect attempt immediately, so the user does not need to discover the failure and then press another key.

Protocol-specific state must be cleaned up as well. An active ZMODEM transfer cannot survive the loss of its channel, so its coordinator is stopped when the transport epoch ends. Pretending such a transfer can resume would leave the terminal parser and progress UI in inconsistent states.

What reconnection does not restore

This design preserves the local tab, not the remote process.

It cannot recover an editor, REPL, foreground job, shell variables, or working directory that existed only in the dead remote shell. Users who need process continuity should still use a remote session manager such as tmux or screen.

The client also should not automatically re-run the last command. Whether a command is safe to repeat depends on application semantics that a terminal cannot reliably infer.

The useful promise is narrower: when the transport dies, the tab remains understandable, reconnection does not create duplicate PTYs, stale input is not executed, host-key policy still applies, and the new shell begins at a clear visual and recorded boundary.

That is less magical than replaying everything. It is also much easier to trust.


Developer disclosure: I work on Nexus Shell, a native macOS SSH client. This article describes implementation lessons from that work; it is not a comparison or benchmark against other SSH clients.