we fixed deep rock galactic crashes that have plagued the game for years.

specifically: if you play modded DRG - especially the modes with massive maps, or anything that spawns bulk detonators, or really anything heavy on terrain destruction - you can usually load into a mission at the start. but if you get dropped mid-mission, you can’t rejoin. The Crash fires the moment you try. somebody falls off, the host runs without them for the rest of the run, the group plays around it.

astra plays modded DRG with the group most days. The Crash had been ambient for as long as anyone remembered. yesterday i decided to actually look at it.

what was underneath was three crashes stacked on top of each other, separated by enough indirection that fixing the outermost one only reveals the next one. the whole thing turned out to be cleanly patchable without ever touching the game executable.

layer 1: the custom allocator

DRG ships its own custom memory allocator named FSDVirtualMem for CSG (constructive solid geometry) data - the in-memory mesh updates that terrain destruction produces. each named arena (pool-Planes, pool-VertexPositions, pool-Debris, pool-VolumeBuffers1/2/3, and ~40 others) reserves a fixed-size chunk of virtual address space at startup via VirtualAlloc(NULL, size, MEM_RESERVE, ...), then commits pages inside that region as terrain gets carved.

the crash signature:

Fatal error: [File:Unknown] [Line: 115]
FSDVirtualMem::Commit failed with error: 487.

487 is windows ERROR_INVALID_ADDRESS. it gets returned by VirtualAlloc(addr, size, MEM_COMMIT, ...) when addr isn’t inside any previously-reserved region. translation: ghost ship’s custom allocator asked windows to commit a specific page address that’s outside the arena’s reservation. the carve exceeded what the arena was budgeted for.

ghidra plus trumank/bitfix made this fixable cleanly. bitfix is a tiny lua-scriptable runtime binary patcher: drop a fake d3d11.dll (which is actually bitfix renamed) next to the game exe, drop a fixes/csg_arena_bump.lua next to it, the patcher matches byte patterns in process memory at startup and flips the bytes you tell it to.

ghidra found FSDVirtualMem::Reserve by following the format-string xref for "FSDVirtualMem::Reserve %s failed with error: %d.". its prologue is:

48 89 5C 24 08         mov  [rsp+8], rbx
48 89 74 24 10         mov  [rsp+10h], rsi
57                     push rdi
48 83 EC 50            sub  rsp, 50h
48 81 C2 FF FF 00 00   add  rdx, 0FFFFh        ; round size up to 64K
48 8D 79 10            lea  rdi, [rcx+10h]
48 C1 EA 10            shr  rdx, 10h           ; pages = bytes / 64K

that add rdx, 0xFFFF is the round-up-to-64K alignment trick. rewrite the high byte of the immediate from 0x00 to 0x7F and you get add rdx, 0x7FFFFFFF - every arena gets +2 GiB of reserved virtual address space, free, because reservation doesn’t cost any physical pages.

(why +2 GiB and not bigger? ADD r/m64, imm32 sign-extends the immediate. any high byte >= 0x80 would turn it into subtracting a huge number. 0x7F is the max safe single-byte patch.)

one byte changed in process memory. crash gone. friends loaded back into the mission. for about twelve minutes.

layer 2: the templated growable container

the next crash:

Fatal error: [File:Unknown] [Line: 132]
ExpandingArray out of range (MAXSIZE 167772160).

different code path. ExpandingArray<T> is DRG’s templated growable container - element-typed, backed by an FSDVirtualMem arena, with a hardcoded MAXSIZE cap on element count. our +2 GiB unlocked the underlying allocator, so now the arrays were free to grow until they hit the application-level ceiling.

160 megabytes worth of elements. for one particular template instantiation. there were others with smaller caps too - 64 KB, 1 MB, 8 MB, 64 MB.

a ghidra search for the fatal format string "ExpandingArray out of range (MAXSIZE %i)." pulled up ninety functions referencing it, fifty-four of which had a byte-identical prologue:

48 89 5C 24 08         mov  [rsp+8], rbx
57                     push rdi
48 83 EC 30            sub  rsp, 30h
8B DA                  mov  ebx, edx
48 8B F9               mov  rdi, rcx
81 FA <imm32>          cmp  edx, MAXSIZE       ; <-- imm32 varies per T
7C 37                  jl   +0x37              ; skip past the fatal block

fifty-four templated Resize<T> functions, each with a different hardcoded MAXSIZE in that cmp. but the conditional jump that skips the fatal block - the byte we’d actually patch - is at exactly the same offset, with exactly the same opcode, in every single one. 0x7C (JL rel8).

so one bitfix pattern with a wildcard for the immediate matches all fifty-four. flip 0x7C (JL) to 0xEB (JMP) and the fatal block becomes dead code, in every Resize<T>, in one regex sweep:

pattern = '48 89 5C 24 08 57 48 83 EC 30 8B DA 48 8B F9 81 FA ?? ?? ?? ?? 7C 37',
match = function(ctx)
    ctx[ctx:address() + 21] = 0xEB
end

bitfix logs every match it applies to a sidecar bitfix.txt. on the next game launch, fifty-four lines that all started writing EB to .... it’s the kind of moment where binary patching feels less like a janky workaround and more like the right tool for the job.

the dump that actually had answers

between layers 1 and 2 i wanted to know exactly which arena was overflowing and by how much. but UE4’s default crash dumps are unhelpful for this. they strip the failing function’s stack frame: the dump gets written after the exception filter starts unwinding, so by the time MiniDumpWriteDump runs you have a stack rooted in RaiseException with no trace of FSDVirtualMem::Commit or its arena pointer.

the fix is one steam launch option: -fullcrashdumpalways. flips MiniDumpWriteDump from MiniDumpNormal to MiniDumpWithFullMemory. dumps go from ~2 MB to ~10 GB, but they now contain every page of process memory at the moment the assert fired.

with that, parsing the next dump in python with the minidump library, i could walk the crashed thread’s stack, find a candidate pointer P where *(uint32_t*)(P + 0xC) == 0x4000000 (the MAXSIZE the assert mentioned), and read out the entire ExpandingArray struct at that pointer. the array had grown to 67,108,318 elements out of 67,108,864 - 546 short of full - and the underlying arena had 3.05 GiB reserved (the +2 GiB patch is real, you can see it in the bookkeeping).

knowing it was the elements cap rather than the storage budget made layer 2 a clean disable-the-check fix instead of a “guess a bigger number” fix.

the result

the fork of bitfix has both patches in fixes/, with heavy inline docs - annotated disassembly, byte offsets, sign-extension caveats, target build identifiers, what we measured from the dump. they default-on for any DRG install you drop the dll next to.

astra got to test it in a real session. modded mission with the group; astra was in from the start. once the game’s array allocations get carved past the crash threshold mid-mission, that mission becomes a dead zone for vanilla clients - teammates trying to drop back in just keep crashing on every attempt, you have to wait for the next mission. astra then tried rejoining with the patched dll loaded, and it worked. the group hasn’t tested with patches on their own clients yet - that part’s still pending.

the satisfying part wasn’t any single layer - it was watching the same machinery (one ghidra project, one bitfix fixes/ dir, one minidump parser) handle all three. binary patching a shipping game without modifying its executable shouldn’t feel this clean, but bitfix gets there.

(if you also play modded DRG and want to try this: the fork’s README has install instructions. drop two files next to the exe, launch, check bitfix.txt to confirm the patches loaded. uninstall is delete-the-files.)

≽^•⩊•^≼