Magic number (programming) explained

In computer programming, a magic number or file signature is a numeric literal in source code that has a special, particular meaning that is less than clear to the reader. Also in computing, but not limited to programming, the term is used for a number that identifies a particular concept but without additional knowledge its meaning is less than clear. For example, some file formats are identified by an embedded magic number in the file). Also, a number that is relatively uniquely associated with a particular concept, such as a universally unique identifier, might be classified as a magic number.

Numeric literal

A magic number or magic constant is a numeric literal in source code which has a special meaning that is less than clear in context. This is considered an anti-pattern and breaks one of the oldest rules of programming, dating back to the COBOL, FORTRAN and PL/1 manuals of the 1960s.[1]

For example, in the following code that computes a price after tax, 1.05 is a magic number since the value encodes the sales tax rate, 5%, in a way that is less than obvious.

price_after_tax = 1.05 * price

The use of magic numbers in code obscures the developers' intent in choosing that number,[2] increases opportunities for subtle errors, and makes it more difficult for the program to be adapted and extended in the future.[3] As an example, it is difficult to tell whether every digit in [[Pi|3.14159265358979323846]] is correctly typed, or if this constant for pi can be truncated to 3.14159 without affecting the functionality of the program with its reduced precision. Replacing all significant magic numbers with named constants (also called explanatory variables) makes programs easier to read, understand and maintain.[4]

The example above can be improved by adding a descriptively named variable: TAX = 0.05 price_after_tax = (1.0 + TAX) * price

A good name can result in code that is more easily understood by a maintainer who is not the original author and even the original author after a period of time. An example of an uninformatively named constant is int SIXTEEN = 16, while int NUMBER_OF_BITS = 16 might be more useful.

Non-numeric data can have the same magical properties, and therefore, the same issues as magic numbers.[1] Thus, declaring const string testUserName = "John" and using might be better than using the literal "John" directly.

Example

For example, if it is required to randomly shuffle the values in an array representing a standard pack of playing cards, this pseudocode does the job using the Fisher–Yates shuffle algorithm:

for i from 1 to 52 j := i + randomInt(53 - i) - 1 a.swapEntries(i, j)

where a is an array object, the function randomInt(x) chooses a random integer between 1 and x, inclusive, and swapEntries(i, j) swaps the ith and jth entries in the array. In the preceding example, 52 and 53 are magic numbers, also not clearly related to each other. It is considered better programming style to write the following:

int deckSize:= 52 for i from 1 to deckSize j := i + randomInt(deckSize + 1 - i) - 1 a.swapEntries(i, j)

This is preferable for several reasons:

function shuffle (int deckSize) for i from 1 to deckSize j := i + randomInt(deckSize + 1 - i) - 1 a.swapEntries(i, j)

Disadvantages are:

Accepted use

When a numeric literal lacks special meaning, then its use is not classified as magic, although what constitutes special is subjective. Examples of literals that are often not considered magic include:

\sqrt{f(x)2+f(y)2}

Format indicator

Origin

Format indicators were first used in early Version 7 Unix source code.

Unix was ported to one of the first DEC PDP-11/20s, which did not have memory protection. So early versions of Unix used the relocatable memory reference model.[5] Pre-Sixth Edition Unix versions read an executable file into memory and jumped to the first low memory address of the program, relative address zero. With the development of paged versions of Unix, a header was created to describe the executable image components. Also, a branch instruction was inserted as the first word of the header to skip the header and start the program. In this way a program could be run in the older relocatable memory reference (regular) mode or in paged mode. As more executable formats were developed, new constants were added by incrementing the branch offset.[6]

In the Sixth Edition source code of the Unix program loader, the exec function read the executable (binary) image from the file system. The first 8 bytes of the file was a header containing the sizes of the program (text) and initialized (global) data areas. Also, the first 16-bit word of the header was compared to two constants to determine if the executable image contained relocatable memory references (normal), the newly implemented paged read-only executable image, or the separated instruction and data paged image.[7] There was no mention of the dual role of the header constant, but the high order byte of the constant was, in fact, the operation code for the PDP-11 branch instruction (octal 000407 or hex 0107). Adding seven to the program counter showed that if this constant was executed, it would branch the Unix exec service over the executable image eight byte header and start the program.

Since the Sixth and Seventh Editions of Unix employed paging code, the dual role of the header constant was hidden. That is, the exec service read the executable file header (meta) data into a kernel space buffer, but read the executable image into user space, thereby not using the constant's branching feature. Magic number creation was implemented in the Unix linker and loader and magic number branching was probably still used in the suite of stand-alone diagnostic programs that came with the Sixth and Seventh Editions. Thus, the header constant did provide an illusion and met the criteria for magic.

In Version Seven Unix, the header constant was not tested directly, but assigned to a variable labeled ux_mag[8] and subsequently referred to as the magic number. Probably because of its uniqueness, the term magic number came to mean executable format type, then expanded to mean file system type, and expanded again to mean any type of file.

In files

See also: List of file signatures.

Magic numbers are common in programs across many operating systems. Magic numbers implement strongly typed data and are a form of in-band signaling to the controlling program that reads the data type(s) at program run-time. Many files have such constants that identify the contained data. Detecting such constants in files is a simple and effective way of distinguishing between many file formats and can yield further run-time information.

Examples
DetectionThe Unix utility program [[File (command)|file]] can read and interpret magic numbers from files, and the file which is used to parse the information is called magic. The Windows utility TrID has a similar purpose.

In protocols

Examples

In interfaces

Magic numbers are common in API functions and interfaces across many operating systems, including DOS, Windows and NetWare:

Examples

Other uses

Examples

GUID

It is possible to create or alter globally unique identifiers (GUIDs) so that they are memorable, but this is highly discouraged as it compromises their strength as near-unique identifiers.[13] [14] The specifications for generating GUIDs and UUIDs are quite complex, which is what leads to them being virtually unique, if properly implemented.[15]

Microsoft Windows product ID numbers for Microsoft Office products sometimes end with 0000-0000-0000000FF1CE ("OFFICE"), such as 90160000-008C-0000-0000-0000000FF1CE, the product ID for the "Office 16 Click-to-Run Extensibility Component".

Java uses several GUIDs starting with CAFEEFAC.[16]

In the GUID Partition Table of the GPT partitioning scheme, BIOS Boot partitions use the special GUID 21686148-6449-6E6F-744E-656564454649[17] which does not follow the GUID definition; instead, it is formed by using the ASCII codes for the string Hah!IdontNeedEFI partially in little endian order.[18]

Debug value

Magic debug values are specific values written to memory during allocation or deallocation, so that it will later be possible to tell whether or not they have become corrupted, and to make it obvious when values taken from uninitialized memory are being used. Memory is usually viewed in hexadecimal, so memorable repeating or hexspeak values are common. Numerically odd values may be preferred so that processors without byte addressing will fault when attempting to use them as pointers (which must fall at even addresses). Values should be chosen that are away from likely addresses (the program code, static data, heap data, or the stack). Similarly, they may be chosen so that they are not valid codes in the instruction set for the given architecture.

Since it is very unlikely, although possible, that a 32-bit integer would take this specific value, the appearance of such a number in a debugger or memory dump most likely indicates an error such as a buffer overflow or an uninitialized variable.

Famous and common examples include:

CodeDescription
00008123 Used in MS Visual C++. Deleted pointers are set to this value, so they throw an exception, when they are used after; it is a more recognizable alias for the zero address. It is activated with the Security Development Lifecycle (/sdl) option.[19]
..FACADE "Facade", Used by a number of RTOSes.
1BADB002 "1 bad boot", Multiboot header magic number.[20]
8BADF00D "Ate bad food", Indicates that an Apple iOS application has been terminated because a watchdog timeout occurred.[21]
A5A5A5A5 Used in embedded development because the alternating bit pattern (1010 0101) creates an easily recognized pattern on oscilloscopes and logic analyzers.
A5 Used in FreeBSD's PHK malloc(3) for debugging when /etc/malloc.conf is symlinked to "-J" to initialize all newly allocated memory as this value is not a NULL pointer or ASCII NUL character.
ABABABAB Used by Microsoft's debug HeapAlloc to mark "no man's land" guard bytes after allocated heap memory.[22]
ABADBABE "A bad babe", Used by Apple as the "Boot Zero Block" magic number.
ABBABABE "ABBA babe", used by memory heap.
ABADCAFE "A bad cafe", Used to initialize all unallocated memory (Mungwall, AmigaOS).
B16B00B5 "Big Boobs", Formerly required by Microsoft's Hyper-V hypervisor to be used by Linux guests as the upper half of their "guest id".[23]
BAADF00D "Bad food", Used by Microsoft's debug HeapAlloc to mark uninitialized allocated heap memory.
BAAAAAAD "Baaaaaad", Indicates that the Apple iOS log is a stackshot of the entire system, not a crash report.
BAD22222 "Bad too repeatedly", Indicates that an Apple iOS VoIP application has been terminated because it resumed too frequently.
BADBADBADBAD "Bad bad bad bad", Burroughs large systems "uninitialized" memory (48-bit words).
BADC0FFEE0DDF00D "Bad coffee odd food", Used on IBM RS/6000 64-bit systems to indicate uninitialized CPU registers.
BADDCAFE "Bad cafe", On Sun Microsystems' Solaris, marks uninitialized kernel memory (KMEM_UNINITIALIZED_PATTERN).
BBADBEEF "Bad beef", Used in WebKit, for particularly unrecoverable errors.
BEBEBEBE Used by AddressSanitizer to fill allocated but not initialized memory.[24]
BEEFCACE "Beef cake", Used by Microsoft .NET as a magic number in resource files.
C00010FF "Cool off", Indicates Apple iOS app was killed by the operating system in response to a thermal event.
CAFEBABE "Cafe babe", Used by Java for class files.Used in multi-architecture Mach-O binaries.
CAFED00D "Cafe dude", Used by Java for their pack200 compression.
CAFEFEED "Cafe feed", Used by Sun Microsystems' Solaris debugging kernel to mark kmemfree memory.
CCCCCCCC Used by Microsoft's C++ debugging runtime library and many DOS environments to mark uninitialized stack memory. CC is the opcode of the INT 3 debug breakpoint interrupt on x86 processors.[25]
CDCDCDCD Used by Microsoft's C/C++ debug malloc function to mark uninitialized heap memory, usually returned from HeapAlloc.
0D15EA5E "Zero Disease", Used as a flag to indicate regular boot on the GameCube and Wii consoles.
DDDDDDDD Used by MicroQuill's SmartHeap and Microsoft's C/C++ debug free function to mark freed heap memory.
DEAD10CC "Dead lock", Indicates that an Apple iOS application has been terminated because it held on to a system resource while running in the background.
DEADBABE "Dead babe", Used at the start of Silicon Graphics' IRIX arena files.
DEADBEEF "Dead beef", Famously used on IBM systems such as the RS/6000, also used in the classic Mac OS operating systems, OPENSTEP Enterprise, and the Commodore Amiga. On Sun Microsystems' Solaris, marks freed kernel memory (KMEM_FREE_PATTERN).
DEADCAFE "Dead cafe", Used by Microsoft .NET as an error number in DLLs.
DEADC0DE "Dead code", Used as a marker in OpenWRT firmware to signify the beginning of the to-be created jffs2 file system at the end of the static firmware.
DEADFA11 "Dead fail", Indicates that an Apple iOS application has been force quit by the user.
DEADF00D "Dead food", Used by Mungwall on the Commodore Amiga to mark allocated but uninitialized memory.[26]
DEFEC8ED "Defecated", Used for OpenSolaris core dumps.
DEADDEAD "Dead Dead" indicates that the user deliberately initiated a crash dump from either the kernel debugger or the keyboard under Microsoft Windows.[27]
D00D2BAD"Dude, Too Bad", Used by Safari crashes on macOS Big Sur.[28]
D00DF33D"Dude feed", Used by the devicetree to mark the start of headers.[29]
EBEBEBEB From MicroQuill's SmartHeap.
FADEDEAD "Fade dead", Comes at the end to identify every AppleScript script.
FDFDFDFD Used by Microsoft's C/C++ debug malloc function to mark "no man's land" guard bytes before and after allocated heap memory, and some debug Secure C-Runtime functions implemented by Microsoft (e.g. strncat_s).[30]
FEE1DEAD "Feel dead", Used by Linux reboot syscall.
FEEDFACE "Feed face", Seen in Mach-O binaries on Apple Inc.'s Mac OSX platform. On Sun Microsystems' Solaris, marks the red zone (KMEM_REDZONE_PATTERN).Used by VLC player and some IP cameras in RTP/RTCP protocol, VLC player sends four bytes in the order of the endianness of the system. Some IP cameras expect the player to send this magic number and do not start the stream if it is not received.
FEEEFEEE "Fee fee", Used by Microsoft's debug HeapFree to mark freed heap memory. Some nearby internal bookkeeping values may have the high word set to FEEE as well.

Most of these are 32 bits longthe word size of most 32-bit architecture computers.

The prevalence of these values in Microsoft technology is no coincidence; they are discussed in detail in Steve Maguire's book Writing Solid Code from Microsoft Press. He gives a variety of criteria for these values, such as:

Since they were often used to mark areas of memory that were essentially empty, some of these terms came to be used in phrases meaning "gone, aborted, flushed from memory"; e.g. "Your program is DEADBEEF".

Notes and References

  1. Book: Martin, Robert C. . Clean Code - A handbook of agile software craftsmanship . limited . 2009 . Prentice Hall . Boston . 978-0-13-235088-4 . 300 . Chapter 17: Smells and Heuristics - G25 Replace Magic Numbers with Named Constants .
  2. Book: Martin, Robert C. . Clean Code - A handbook of agile software craftsmanship . limited . 2009 . Prentice Hall . Boston . 978-0-13-235088-4 . 295 . Chapter 17: Smells and Heuristics - G16 Obscured Intent.
  3. Web site: Bjarne Stroustrup on Educating Software Developers . James . Maguire . 9 December 2008 . Datamation.com . dead . https://web.archive.org/web/20180623112852/http://www.datamation.com/columns/article.php/3789981/Bjarne-Stroustrup-on-Educating-Software-Developers.htm . 23 June 2018.
  4. Web site: Six ways to write more comprehensible code . Jeff . Vogel . 29 May 2007 . IBM Developer . https://web.archive.org/web/20180926205449/https://www.ibm.com/developerworks/linux/library/l-clear-code/?ca=dgr-FClnxw01linuxcodetips . 26 September 2018 . dead .
  5. Web site: Odd Comments and Strange Doings in Unix . 22 June 2002 . . dead . https://web.archive.org/web/20061104034450/http://cm.bell-labs.com/cm/cs/who/dmr/odd.html . 2006-11-04.
  6. Personal communication with Dennis M. Ritchie.
  7. Web site: The Unix Tree V6/usr/sys/ken/sys1.c . . https://web.archive.org/web/20230326024616/https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/sys/ken/sys1.c . 26 March 2023 . live .
  8. Web site: The Unix Tree V7/usr/sys/sys/sys1.c . . https://web.archive.org/web/20230326024632/https://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/sys1.c . 26 March 2023 . live .
  9. Web site: PNG (Portable Network Graphics) Specification Version 1.0: 12.11. PNG file signature . 1 October 1996 . MIT . https://web.archive.org/web/20230326024630/http://www.libpng.org/pub/png/spec/1.0/PNG-Rationale.html#R.PNG-file-signature . 26 March 2023 . live .
  10. Web site: What's the difference between the COM and EXE extensions? . Raymond . Chen . 24 March 2008 . The Old New Thing . dead . https://web.archive.org/web/20190218083526/https://blogs.msdn.microsoft.com/oldnewthing/20080324-00/?p=23033 . 18 February 2019.
  11. Web site: The BIOS/MBR Boot Process . 2015-01-25 . NeoSmart Knowledgebase . en-US . 2019-02-03 . https://web.archive.org/web/20230326024702/https://neosmart.net/wiki/mbr-boot-process/ . 26 March 2023 . live .
  12. Web site: TI E2E Community: Does anyone know if the following configurations can be done with MCP CLI Tool? . 27 August 2011 . Texas Instruments . https://web.archive.org/web/20221007161243/https://e2e.ti.com/support/processors-group/processors/f/processors-forum/589272/ccs-tms320c5545-c5545-uart_test-to-run-without-msp430 . 7 October 2022 . live .
  13. Web site: Message Management: Guaranteeing uniqueness . Joseph M. . Newcomer . 13 October 2001 . Developer Fusion . 2007-11-16 . https://web.archive.org/web/20050421023819/https://www.developerfusion.com/show/1713/4/ . 21 April 2005 . dead .
  14. Web site: UUIDs are only unique if you generate them... . Larry . Osterman . 21 July 2005 . Larry Osterman's WebLog - Confessions of an Old Fogey . MSDN . 2007-11-16 . https://web.archive.org/web/20230328134453/https://learn.microsoft.com/en-us/archive/blogs/larryosterman/uuids-are-only-unique-if-you-generate-them . 28 March 2023 . live .
  15. Web site: RFC 9562 - Universally Unique IDentifiers (UUIDs) . May 2024 . ietf.org . 9 August 2024 .
  16. Web site: Deploying Java Applets With Family JRE Versions in Java Plug-in for Internet Explorer . . 28 March 2023 . https://web.archive.org/web/20221130180350/https://www.oracle.com/java/technologies/javase/family-clsid.html . 30 November 2022 . live .
  17. Web site: GNU GRUB Installation, Section 3.4: BIOS installation . 2014-06-26 . Gnu.org . https://web.archive.org/web/20230315232257/https://www.gnu.org/software/grub/manual/grub/html_node/BIOS-installation.html . 15 March 2023 . live .
  18. Web site: Magic Numbers: The Secret Codes that Programmers Hide in Your PC . Lowell . Heddings . 3 November 2014 . How-To Geek . 3 October 2017 . https://web.archive.org/web/20230326024750/https://www.howtogeek.com/201059/magic-numbers-the-secret-codes-that-programmers-hide-in-your-pc/ . 26 March 2023 . live .
  19. Web site: Cavit . Doug . 24 April 2012 . Guarding against re-use of stale object references . Microsoft Secure . 26 July 2018 . https://web.archive.org/web/20180726103946/https://cloudblogs.microsoft.com/microsoftsecure/2012/04/24/guarding-against-re-use-of-stale-object-references/ . 26 July 2018 . dead .
  20. Web site: Comments on the 'MultiBoot Standard' proposal . Erich Stefan . Boleyn . 4 April 1995 . Uruk.org . https://web.archive.org/web/20230326024756/http://ftp.lyx.org/pub/mach/mach4/multiboot/multiboot-archive . 26 March 2023 . live .
  21. Web site: Technical Note TN2151: Understanding and Analyzing Application Crash Reports . 29 January 2009 . Apple Developer Documentation . https://web.archive.org/web/20181213234116/https://developer.apple.com/library/archive/technotes/tn2151/_index.html . 13 December 2018 . dead .
  22. Web site: Win32 Debug CRT Heap Internals . Andrew . Birkett . Nobugs.org.
  23. Web site: Microsoft code contains the phrase 'big boobs' ... Yes, really . Paul . McNamara . 19 July 2012 . Network World.
  24. Web site: AddressSanitizer - FAQ . . 2022-05-18.
  25. Web site: INTEL 80386 PROGRAMMER'S REFERENCE MANUAL . MIT.
  26. Web site: Amiga Mail Vol.2 Guide . Carolyn . Scheppner . Cataclysm.cx . 2010-08-20 . dead . https://web.archive.org/web/20110718163417/http://cataclysm.cx/random/amiga/reference/AmigaMail_Vol2_guide/node0053.html . 2011-07-18.
  27. Web site: Bug Check 0xDEADDEAD MANUALLY_INITIATED_CRASH1 . Microsoft Documentation. 19 June 2023 .
  28. Web site: Safari Version 14.0.1 Unexpectedly Quits.
  29. Web site: Device Tree Specification.
  30. Web site: strncat_s, _strncat_s_l, wcsncat_s, _wcsncat_s_l, _mbsncat_s, _mbsncat_s_l . Microsoft Documentation . 16 January 2019 . en-us.