How Ubuntu dumps core

When your program tries to do anything nasty, like access memory it is not supposed to, you end up seeing a core dump.

If you ran the program under the Bash shell, you might see this error:

$ ./my_program
Segmentation fault (core dumped)

Under the Fish shell, I see this error:

$ ./my_program
fish: “./my_program” terminated by signal SIGSEGV (Address boundary error)

If I check the directory I ran the program from, I see that there is a core file that has the entire dump of the process memory from when the program did its nasty act.

How did a SIGSEGV end up as a core dump file? This is what I found in Ubuntu:

• On SIGSEGV, Linux kernel writes an entry in /var/log/syslog with some information about what happened. Here is the line on my computer:

Aug 13 17:56:09 my_machine kernel: [442853.259571] my_program[18432]: segfault at 4700000046 ip 00007f7688f6980d sp 00007ffd46b5e3e0 error 4 in libc-2.23.so[7f7688f30000+1c0000]

• The Linux kernel checks what settings are present for writing a core dump. There are 3 settings, which are in the files core_pattern, core_pipe_limit and core_uses_pid in /proc/sys/kernel directory. The documentation of these settings can be found described in kernel.txt.

• core_pattern says what the filename of the core dump file should be. On my Ubuntu, this setting had this value:

$ cat /proc/sys/kernel/core_pattern 
|/usr/share/apport/apport %p %s %c %P

From the kernel.txt documentation, we can see that this means that the kernel has to call the /usr/share/apport/apport executable with four input arguments. The first, second and fourth arguments are documented as PID of offending process, signal number and global PID. We do not know what the %c does, but we will find out soon below. The kernel will pipe the process memory to the standard input of this apport program.

• What is this apport program? It is the crash handler of Ubuntu. We can see that it is a Python script:

$ file /usr/share/apport/apport
/usr/share/apport/apport: Python script, ASCII text executable

That is great, because we can open it and read its code to see what it does!

• Opening up this file in Vim, we see that %c input argument is the ulimit value set in our shell:

Line 393: (pid, signum, core_ulimit) = sys.argv[1:4]

• We can also see what exactly it does to write the core dump in its function write_user_coredump. In short, this function checks the ulimit value of the shell and then decides how many bytes to write to the core file and writes it to the same directory as the program.

• We also see that it called error_log to write some log messages. These can be found listed in the /var/log/apport.log file:

ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: called for pid 18474, signal 11, core limit 102400000
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: executable: /home/joe/my_program (command line "./my_program")
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: executable does not belong to a package, ignoring
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: writing core dump to /home/joe/core (limit: 102400000)

This is how Ubuntu writes the core dump file. You can now open this up in GDB for debugging! 😊

Tried with: Ubuntu 16.04