Fault Tolerance of Parallel Computations on Heterogeneous Platforms

The moment you spread a computation across a CPU, a GPU, and maybe a smart NIC or FPGA, you have quietly signed up for a much harder reliability problem than anything a single-socket server presents. A homogeneous multi-core box has one failure model: a core either executes correctly or the whole machine typically halts. A heterogeneous platform has several independent failure domains — host DRAM with ECC, device HBM with its own scrubbing engine, an IOMMU translating and policing DMA, a PCIe or NVLink fabric that can drop or corrupt in flight, and a device firmware stack that can wedge without ever generating a CPU exception. Linux does not treat this as one problem solved once. It treats it as a layered set of contracts: the Machine Check Architecture (MCA) contract between silicon and kernel for host-side hardware errors, the EDAC contract for scrubbing and reporting memory errors, the hardlockup/softlockup watchdog contract for detecting a CPU that has stopped making forward progress, and the accelerator-driver fault contract (IOMMU fault queues plus driver-level timeout detection and recovery, or TDR) for devices that are, from the kernel’s point of view, black boxes connected over DMA.

This lesson is about how those contracts fit together, why each one exists, and how you build software on top of them that survives a fault in one compute domain without losing the whole job. That last part matters enormously in practice: a training run spanning 512 GPUs that restarts from scratch because one HBM cell flipped a bit is not just wasteful, it is often the dominant cost of running heterogeneous clusters at scale.

Fault tolerance in Linux did not start with GPUs. It started with mainframes and RAS (Reliability, Availability, Serviceability) engineering in the 1990s, where ECC memory and machine check exceptions were built for single-CPU x86 servers that needed to stay up for banking and telecom workloads. The kernel’s Machine Check Architecture support (arch/x86/kernel/cpu/mce/) dates back to the early 2000s, modeled directly on Intel’s and AMD’s MCA register banks. EDAC (drivers/edac/) followed shortly after as a way to expose memory-controller-level ECC scrubbing statistics to userspace, because MCA alone told you that something went wrong, not where in the DIMM topology.

The softlockup/hardlockup watchdog (kernel/watchdog.c) arrived later, in the mid-2000s, as multi-core systems made “a CPU silently stopped answering” a much more common and much harder to diagnose failure than “the machine rebooted.” It uses a per-CPU hrtimer plus, where available, the NMI-driven hardware performance counter watchdog, precisely because a CPU wedged with interrupts disabled will not respond to an ordinary timer interrupt.

The heterogeneous piece is the newest layer. As GPUs, TPUs, and other accelerators became first-class DMA-capable devices sharing the same physical memory bus and PCIe fabric as the CPU, two things had to be reconciled: the device’s own error-reporting mechanism (which the kernel does not control — it belongs to vendor firmware and the DRM/accelerator driver) and the IOMMU’s fault-reporting path, which is kernel-owned and generic across vendors (drivers/iommu/, struct iommu_fault). DRM’s scheduler TDR mechanism (drivers/gpu/drm/scheduler/) was added so a hung GPU job could be detected and the device reset without taking the whole kernel down — directly analogous to what the hardlockup watchdog does for CPUs, but implemented entirely in software because there is no MCA-equivalent hardware trap for “the GPU firmware stopped answering.”

The core problem is this: a parallel computation on a heterogeneous platform has multiple independent execution units, each with its own failure surface, running asynchronously relative to each other. A CPU thread blocking on a GPU kernel launch does not get a synchronous exception when that GPU kernel corrupts memory or hangs — the CPU thread is just waiting on a fence or a completion queue. This breaks the assumption most fault-handling code is built on: that a fault happens in the context that can be blamed for it and at the instruction that caused it.

Three failure classes matter, and Linux treats each differently:

1. Correctable errors — a single bit flip caught and fixed by ECC. The workload is unaffected, but if you don’t count these, you can’t predict when a DIMM or an HBM stack is about to fail outright. EDAC counts them, rasdaemon logs them, nothing else happens.

2. Uncorrectable but recoverable errors — a multi-bit error the hardware can detect but not fix, localized to a specific physical page. On x86 this is a Software Recoverable Action Optional/Required (SRAO/SRAR) machine check. The kernel’s job is to isolate exactly that page (memory_failure() in mm/memory-failure.c, tagging the struct page with PG_hwpoison) and kill only the process that actually touches it — everything else on the machine keeps running.

3. Fatal errors — the hardware cannot even tell you which page is bad, or the error is in a structure the kernel cannot survive without (kernel text, page tables). The only correct response is panic(), ideally with kdump capturing a crash image for postmortem.

For a device like a GPU, there is a fourth practical category the CPU-centric MCA model doesn’t cover at all: the device stopped responding but issued no error. This is what DRM’s TDR and, more generally, a userspace heartbeat/watchdog pattern exist to catch.

Show Image

Four subsystems cooperate, each owning a distinct failure domain:

• MCA / arch/x86/kernel/cpu/mce/ — owns CPU-detected hardware errors: ECC uncorrectable events on host DRAM, cache errors, interconnect errors. Delivered via the #MC exception (trap vector 18 on x86) or, for correctable errors, via a periodic poll timer (mce_timer) reading the MCA banks through MSRs.

• EDAC / drivers/edac/ — owns memory-controller-level scrubbing and topology (which DIMM, which rank, which channel). It is a reporting layer on top of hardware ECC, not a detection mechanism itself; it exposes counts and locations through /sys/devices/system/edac/ and tracepoints.

• kernel/watchdog.c — owns “a CPU is alive but not scheduling.” It runs a per-CPU kernel thread fed by an hrtimer, and, on hardware that supports it, a hardware-perf-counter-driven NMI watchdog for the hardlockup case (an interrupts-disabled infinite loop that a plain timer interrupt can never preempt).

• The accelerator driver + IOMMU fault path — owns everything the device itself does. The IOMMU (drivers/iommu/) reports faulting DMA transactions through struct iommu_fault and a per-device fault queue (iommu_report_device_fault()); the DRM scheduler reports jobs that exceeded a timeout through drm_sched_job_timedout(), which triggers device-specific reset logic.

None of these four talk to each other directly inside the kernel. What unifies them for a parallel workload is userspace: a runtime (CUDA/ROCm-equivalent, or your own orchestration layer) that consumes signals from all four — SIGBUS for hwpoison, netlink/tracepoint events for EDAC and MCE, dmesg/sysfs reset counters for GPU TDR — and decides how to keep the overall job alive.

Start with the CPU-detected case, because it is the most fully specified. Every logical CPU has a set of MCA banks, each covering a hardware unit (L1, L2, memory controller, interconnect). When an error is detected, the bank’s status register is latched with severity and address information, and, for an uncorrected error, the CPU raises #MC. The kernel’s do_machine_check() handler runs at the highest privilege, reads every bank across the affected CPUs (an MCE can be broadcast to all CPUs simultaneously if the error is global, e.g. a bus error), and hands each populated bank to the MCE decode chain — an atomic_notifier_head (x86_mce_decoder_chain) that lets subsystems like EDAC register a callback to translate a raw bank/address pair into “DIMM 2, channel 1, rank 0.” The core MCE code then classifies severity using mce_severity(): is this recoverable (a specific physical address is poisoned and only that page’s memory content is untrustworthy) or fatal (the error is in a location, such as kernel code, the kernel cannot simply route around)?

For a recoverable error, do_machine_check() calls into memory_failure(pfn, flags). This function:

1. Locks the page and determines what maps it (page_mapping, reverse-mapping via the RMAP subsystem).

2. Sets PG_hwpoison on the struct page so no future allocator ever hands this physical page out again.

3. Unmaps it from every process page table that references it, and delivers SIGBUS with si_code = BUS_MCEERR_AR (Action Required, meaning “you touched this page, deal with it now”) to any process actually accessing it at the time.

4. Leaves every other process on the machine completely untouched.

For the CPU-liveness case, kernel/watchdog.c runs a per-CPU thread (watchdog/N) at a very high scheduling priority, fed by an hrtimer that fires roughly every watchdog_thresh / 5 seconds. Each firing bumps a per-CPU counter the thread is expected to reset. If the counter goes stale past a softlockup threshold (default 20s), the kernel prints a stack trace and, depending on kernel.softlockup_panic, may treat it as fatal. If the NMI-driven hardlockup watchdog (built on a perf hardware counter overflowing at a fixed instruction/cycle count) detects that even NMIs aren’t reaching a CPU, that CPU is presumed truly wedged — the strongest signal short of a full ECC-driven panic.

For the device-side case, there is no #MC-equivalent trap, so the kernel relies on two independent software mechanisms. The IOMMU fault queue catches DMA transactions the device attempted that violate the IOMMU’s page tables (e.g., the device tried to write to an address the driver never mapped for it) and reports them through iommu_report_device_fault(), which the accelerator driver’s fault handler consumes to decide whether to reset the device’s translation context. Separately, DRM’s scheduler tracks every submitted job with a timer; if a job doesn’t complete within the configured timeout, drm_sched_job_timedout() fires, and the driver’s .timedout_job callback performs a device-specific reset (often resetting just the faulting engine, not the whole card) and resubmits or fails the remaining queued jobs.

https://github.com/sysdr/howtech-p/tree/main/Fault_tolerance/fault-tolerant-demo

Show Image

1. Hardware detects an anomaly — an ECC syndrome mismatch on a memory read, a parity error on a cache line, a DMA transaction outside a device’s mapped IOMMU range, or a device firmware watchdog timeout.

2. CPU path: the MCA bank latches status + address, #MC is raised, do_machine_check() runs, walks the decode chain, classifies severity. Device path: the IOMMU raises a fault queue entry, or the DRM scheduler’s per-job timer expires.

3. Decision point: correctable (log via EDAC/tracepoint, continue), uncorrectable-recoverable (isolate one page or reset one device queue), or fatal (panic/kdump, or in the device case, a full device reset with all in-flight work lost).

4. Success path: for memory, memory_failure() poisons the page and signals only the affected task; for a device, the driver resets the faulting context and the runtime resubmits the lost work unit from its last checkpoint.

5. Error path: for a fatal CPU error, panic() triggers kdump‘s second kernel to capture a crash image before reboot; for a device that won’t reset cleanly, the driver marks it offline and the orchestration layer (Kubernetes device plugin, Slurm, or your own scheduler) drains the node.

6. Userspace observability layer (rasdaemon, dmesg, sysfs counters) records the event regardless of path, closing the loop for capacity planning (predicting DIMM/HBM failure from correctable-error trend lines).

c

The recurring pattern worth internalizing: every one of these structures separates identity (which bank, which CPU, which device, which DIMM) from classification (correctable vs. uncorrectable, recoverable vs. fatal). Recovery code only ever acts on the classification; the identity fields exist purely for the observability/reporting path.

Machine checks interact with the CPU pipeline in a way that is easy to get wrong intuitively. An MCA bank can report an error that happened speculatively — for instance, a load that was executed out-of-order down a branch that ends up not being taken. If the CPU consumed a poisoned value from that speculative load into architectural state, that’s an SRAR (Software Recoverable — Action Required) event, and #MC fires synchronously at (approximately) the instruction that consumed it. If the CPU merely detected poison in a location during background scrubbing without any instruction actually consuming it yet, that’s SRAO (Action Optional) — the kernel can log it, poison the page proactively, and let execution continue, because nothing has actually read the bad data yet.

This distinction is why memory_failure() takes flags distinguishing MF_ACTION_REQUIRED from a background poisoning call: an AR error must interrupt and signal the current context because it already touched bad data (the instruction pointer at #MC time is meaningful), while an AO error is handled entirely out of band from any specific instruction stream.

Broadcast MCEs complicate this further: some errors (typically bus/interconnect errors visible to the whole system) are signaled to every logical CPU simultaneously via the local APIC’s MCE-broadcast mechanism, precisely because a single core cannot unilaterally decide “this is fatal” for an error that corrupted shared state — all CPUs must agree before panic() is called, which is why mce_start()/mce_end() implement a barrier-based rendezvous across CPUs during machine check handling.

The performance cost of this architecture is dominated by three things, none of them free:

• EDAC scrubbing runs as a background memory-controller activity (hardware-driven, typically configurable in BIOS as a scrub rate). A too-aggressive scrub rate steals memory bandwidth from your workload; a too-passive one lets correctable errors accumulate into uncorrectable ones before you notice.

• The watchdog hrtimer fires on every CPU roughly every 4 seconds by default (watchdog_thresh=20 → sampling period 4s), which is negligible steady-state overhead but is a real consideration on latency-sensitive isolated cores (isolcpus/nohz_full), where you often deliberately disable the watchdog per-core to avoid any scheduled interrupt.

• DRM TDR timeouts trade detection latency against false-positive resets: a timeout too short kills legitimately long-running kernels (common in ML training with large fused ops); too long, and a genuinely hung device sits unusably idle, silently stalling the whole job that’s waiting on its fence, for the full timeout window.

The dominant cost for heterogeneous parallel workloads specifically, though, is checkpoint granularity, not any single kernel mechanism above. If your fault domain is “one GPU out of 512,” but your checkpoint interval is coarse (say, every 30 minutes of training), the expected wasted compute per fault is (fault_rate) × (avg_time_since_last_checkpoint) × (nodes_in_sync_group) — and in synchronous data-parallel training, the whole sync group stalls waiting for the one lost worker to be replaced, not just that worker.

bash

The single most useful triage habit: when a heterogeneous job dies mysteriously, check EDAC counts before assuming it was a software bug. A steadily climbing ce_count on one DIMM over days, followed by a job crash, is a hardware failure wearing a software costume.

Watchdog starvation under RT priority inversion. The hardlockup watchdog’s NMI path is immune to scheduling, but the softlockup thread is an ordinary (if high-priority) kernel thread. On systems running SCHED_FIFO/SCHED_DEADLINE workloads at very high priority without proper RLIMIT/cgroup bandwidth throttling, the watchdog thread can be starved long enough to itself trigger a false softlockup report — the fix is either sched_rt_runtime_us throttling or moving the watchdog off cores dedicated to RT work via isolcpus.

ECC scrubbing silently disabled. A depressingly common one: a BIOS update or a “performance” BIOS profile disables background patrol scrubbing to shave memory latency. Correctable errors then accumulate undetected until they become uncorrectable, at which point you get a fatal MCE with no warning history — because EDAC only reports what the hardware scrubber actually found.

IOMMU fault storms. A buggy device driver or misconfigured DMA mapping can generate thousands of IOMMU faults per second. Because each fault triggers iommu_report_device_fault() and a queue wakeup, this can itself become a denial-of-service against the host CPU handling the fault queue — the practical mitigation is fault-queue rate limiting in the driver and treating repeated faults from the same device as a signal to proactively reset rather than log-and-continue.

Checkpoint granularity mismatch. Treating a recoverable single-page hwpoison event (which by design only affects one process) as if it required restarting an entire synchronized parallel job. The correct response, at the orchestration layer, is to catch SIGBUS with si_code == BUS_MCEERR_AR, and restart only the affected rank from its last checkpoint while the rest of the synchronous group waits, rather than tearing down the whole job.

Hyperscale ML training clusters are the sharpest example of this stack in production: with thousands of GPUs running for weeks, the expected number of correctable ECC events, IOMMU faults, and individual GPU resets across the fleet during a single training run is not zero — it is a predictable statistical rate, and the training framework’s checkpoint/elastic-restart logic (rank replacement without restarting the whole job) is built directly on top of exactly the SIGBUS/reset-notification primitives described above. Cloud providers run EDAC + rasdaemon fleet-wide specifically to predict DIMM failure from correctable-error trend lines and proactively drain a node before it produces an uncorrectable, job-killing error — turning a reactive fault-tolerance mechanism into a proactive maintenance signal.

We cannot safely load real EDAC/MCE kernel code or trigger genuine IOMMU faults inside a generic sandbox — those require specific hardware and root-level kernel module access. Instead, this lab builds a faithful userspace model of the exact same architecture, so every concept above maps 1:1 onto working code you can run right now.

The demo, generated entirely by startup.sh, implements:

• A supervisor process modeling the kernel’s MCE decode chain + watchdog core.

• Several worker processes modeling heterogeneous compute units (standing in for GPUs/accelerators), each periodically “computing” and occasionally injecting one of three fault classes: correctable, uncorrectable-recoverable, and fatal — mirroring EDAC-logged, hwpoison-and-SIGBUS, and panic-and-restart respectively.

• A heartbeat/watchdog mechanism (a SIGALRM-driven liveness check per worker) modeling kernel/watchdog.c‘s hrtimer-driven staleness detection, so a hung worker (no heartbeat, no fault message) is detected and restarted exactly like a hardlockup-then-recovery path.

• A checkpoint/restart mechanism: each worker periodically writes progress to a checkpoint file; on recoverable or fatal fault, the supervisor restarts only that worker from its last checkpoint, demonstrating why granular checkpointing bounds recomputation cost.

Run it with:

bash

chmod +x startup.sh
./startup.sh

Expect to see coloured log lines showing correctable events logged and ignored, recoverable faults triggering a single-worker restart from checkpoint, and (rarely, by design low-probability) a fatal fault draining and restarting the whole demo domain — with a final summary of total compute lost to faults versus total compute completed.

• Always run rasdaemon (or an equivalent MCE/EDAC consumer) in production; without it, correctable-error trend data — your best early-warning signal — is discarded the moment the kernel ring buffer wraps.

• Treat SIGBUS with BUS_MCEERR_AR as a first-class signal in any long-running parallel runtime, not an unhandled crash; catching it and restarting only the affected worker from checkpoint is almost always cheaper than restarting the whole job.

• Checkpoint at a granularity proportional to your observed fault rate, not an arbitrary wall-clock interval — compute expected wasted work as fault_rate × mean_checkpoint_interval and tune accordingly.

• Keep the CPU hardlockup/softlockup watchdog enabled everywhere except cores you have deliberately isolated for interrupt-free RT/HPC work, and disable it explicitly (nohz_full/isolcpus) rather than leaving it silently starved.

• Size DRM/accelerator job timeouts (TDR) against your actual longest legitimate kernel/op duration, not a generic default — false-positive resets are as costly to a training job as genuine hangs.

• Verify BIOS/firmware memory scrub settings explicitly after any firmware update; “performance” profiles disabling background ECC scrubbing is one of the most common silent regressions in fleet reliability.

Fault tolerance on heterogeneous platforms is not one Linux mechanism — it’s four independently evolved subsystems (MCA/MCE, EDAC, the CPU watchdog, and accelerator/IOMMU fault reporting) that share a common shape: detect, classify by severity, and act at the narrowest possible scope, whether that’s one physical page, one CPU, or one device context. The kernel’s entire design philosophy here is isolation of blast radius — an uncorrectable error should cost you one page or one worker, never the whole machine, and a hung device should cost you one reset, never a kernel panic. Building fault-tolerant parallel software on top of this means consuming the right signal at the right layer: SIGBUS/hwpoison for memory, driver reset notifications for devices, and EDAC/rasdaemon trend data for predictive maintenance — and checkpointing at a granularity that makes each of those recoveries cheap.