Investigating a possible Linux bug: small-sized sbrk calls unexpectedly succeed despite low RLIMIT_AS

This article documents the investigatory process in detail. Skip to the patch for a summary of the situation. Background § I assume knowledge of the differences between the heap and the stack, along with experience using malloc to allocate memory. Each process on Linux is given its own address space with the layout: +-----------+ <- Largest memory address (0xfffffffff..) | Stack | +-----------+ <-…