Bug report
Bug description:
In encoder_encode_key_value(), 'key' is a borrowed reference from
PyDict_Next(). If the 'default' callback (invoked during value encoding)
clears the dict, 'key' becomes a dangling pointer. The subsequent
_PyErr_FormatNote("%R", key) calls PyObject_Repr() on freed memory.
Root Cause
// Modules/_json.c — encoder_encode_key_value() // 'key' is a borrowed ref from PyDict_Next() if (encoder_listencode_obj(s, writer, value, ...) < 0) { _PyErr_FormatNote("when serializing %T item %R", dct, key); // ^^^ // key may be freed if default() cleared the dict }
encoder_listencode_obj() can invoke the user-supplied default callable. If default clears or mutates the dict being iterated, the borrowed key reference becomes dangling. The subsequent %R format triggers PyObject_Repr()on freed memory.
import json class Evil: """Value object whose encoding triggers the vulnerability.""" pass class AlsoEvil: """Second object that fails to encode, triggering the error path.""" pass big_key = "A" * 100 # non-interned key so it can be freed target_dict = {big_key: Evil()} del big_key # only the dict holds refs now def evil_default(obj): if isinstance(obj, Evil): target_dict.clear() # frees both key and value return AlsoEvil() raise TypeError("not serializable") json.dumps(target_dict, default=evil_default, check_circular=False)
ASAN Output :
================================================================= ==64296==ERROR: AddressSanitizer: heap-use-after-free on address 0x513000044638 at pc 0x5a84e3c1b8ca bp 0x7ffd761be3f0 sp 0x7ffd761be3e0 READ of size 8 at 0x513000044638 thread T0 #0 0x5a84e3c1b8c9 in _Py_TYPE_impl ../Include/object.h:313 #1 0x5a84e3c1b8c9 in unicode_fromformat_arg ../Objects/unicodeobject.c:2986 #2 0x5a84e3c1bd0a in unicode_from_format ../Objects/unicodeobject.c:3075 #3 0x5a84e3c1beba in PyUnicode_FromFormatV ../Objects/unicodeobject.c:3109 #4 0x5a84e3d858e7 in _PyErr_FormatNote ../Python/errors.c:1259 #5 0x7de06590d67b in encoder_listencode_obj ../Modules/_json.c:1644 #6 0x7de06590e273 in encoder_encode_key_value ../Modules/_json.c:1721 #7 0x7de06590e829 in _encoder_iterate_dict_lock_held ../Modules/_json.c:1782 #8 0x7de06590ee86 in encoder_listencode_dict ../Modules/_json.c:1861 #9 0x7de06590d42a in encoder_listencode_obj ../Modules/_json.c:1605 #10 0x7de06590d870 in encoder_call ../Modules/_json.c:1464 #11 0x5a84e3a6aabe in _PyObject_MakeTpCall ../Objects/call.c:242 #12 0x5a84e3a6ad66 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:134 #13 0x5a84e3a6adbf in PyObject_Vectorcall ../Objects/call.c:327 #14 0x5a84e3ce9cf7 in _Py_VectorCallInstrumentation_StackRefSteal ../Python/ceval.c:769 #15 0x5a84e3cf9cbb in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1817 #16 0x5a84e3d30f30 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:118 #17 0x5a84e3d31296 in _PyEval_Vector ../Python/ceval.c:2132 #18 0x5a84e3d3154c in PyEval_EvalCode ../Python/ceval.c:680 #19 0x5a84e3e34d31 in run_eval_code_obj ../Python/pythonrun.c:1366 #20 0x5a84e3e35077 in run_mod ../Python/pythonrun.c:1469 #21 0x5a84e3e35fac in pyrun_file ../Python/pythonrun.c:1294 #22 0x5a84e3e38de2 in _PyRun_SimpleFileObject ../Python/pythonrun.c:518 #23 0x5a84e3e3908e in _PyRun_AnyFileObject ../Python/pythonrun.c:81 #24 0x5a84e3e8e6b6 in pymain_run_file_obj ../Modules/main.c:410 #25 0x5a84e3e8e923 in pymain_run_file ../Modules/main.c:429 #26 0x5a84e3e90121 in pymain_run_python ../Modules/main.c:691 #27 0x5a84e3e907b7 in Py_RunMain ../Modules/main.c:772 #28 0x5a84e3e909a3 in pymain_main ../Modules/main.c:802 #29 0x5a84e3e90d28 in Py_BytesMain ../Modules/main.c:826 #30 0x5a84e38f5675 in main ../Programs/python.c:15 #31 0x7de06662a3b7 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #32 0x7de06662a47a in __libc_start_main_impl ../csu/libc-start.c:360 #33 0x5a84e38f55a4 in _start (/home/raminfp/Projects/cpython/build-asan/python+0x2ee5a4) (BuildId: b6692a1e94ed1222fac0cb716d806843accd7c52) 0x513000044638 is located 56 bytes inside of 352-byte region [0x513000044600,0x513000044760) freed by thread T0 here: #0 0x7de066afc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 #1 0x5a84e3b38f90 in _PyMem_RawFree ../Objects/obmalloc.c:93 #2 0x5a84e3b3b44a in _PyMem_DebugRawFree ../Objects/obmalloc.c:3092 #3 0x5a84e3b3b48b in _PyMem_DebugFree ../Objects/obmalloc.c:3237 #4 0x5a84e3b63854 in PyObject_Free ../Objects/obmalloc.c:1659 #5 0x5a84e3da44e7 in PyObject_GC_Del ../Python/gc.c:2460 #6 0x5a84e3b7ee36 in object_dealloc ../Objects/typeobject.c:7288 #7 0x5a84e3b9d7b8 in subtype_dealloc ../Objects/typeobject.c:2864 #8 0x5a84e3b2f587 in _Py_Dealloc ../Objects/object.c:3271 #9 0x5a84e3d99cc6 in Py_DECREF_MORTAL ../Include/internal/pycore_object.h:425 #10 0x5a84e3d99d7c in PyStackRef_XCLOSE ../Include/internal/pycore_stackref.h:703 #11 0x5a84e3d9aea5 in _PyFrame_ClearLocals ../Python/frame.c:101 #12 0x5a84e3d9b09d in _PyFrame_ClearExceptCode ../Python/frame.c:126 #13 0x5a84e3ce4e0d in clear_thread_frame ../Python/ceval.c:1939 #14 0x5a84e3ceb21e in _PyEval_FrameClearAndPop ../Python/ceval.c:1976 #15 0x5a84e3d264b0 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:10558 #16 0x5a84e3d30f30 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:118 #17 0x5a84e3d31296 in _PyEval_Vector ../Python/ceval.c:2132 #18 0x5a84e3a6a805 in _PyFunction_Vectorcall ../Objects/call.c:413 #19 0x5a84e3a6accc in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:136 #20 0x5a84e3a6ae8c in PyObject_CallOneArg ../Objects/call.c:395 #21 0x7de06590d4f1 in encoder_listencode_obj ../Modules/_json.c:1628 #22 0x7de06590e273 in encoder_encode_key_value ../Modules/_json.c:1721 #23 0x7de06590e829 in _encoder_iterate_dict_lock_held ../Modules/_json.c:1782 #24 0x7de06590ee86 in encoder_listencode_dict ../Modules/_json.c:1861 #25 0x7de06590d42a in encoder_listencode_obj ../Modules/_json.c:1605 #26 0x7de06590d870 in encoder_call ../Modules/_json.c:1464 #27 0x5a84e3a6aabe in _PyObject_MakeTpCall ../Objects/call.c:242 #28 0x5a84e3a6ad66 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:134 #29 0x5a84e3a6adbf in PyObject_Vectorcall ../Objects/call.c:327 previously allocated by thread T0 here: #0 0x7de066afd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x5a84e3b398a7 in _PyMem_RawMalloc ../Objects/obmalloc.c:65 #2 0x5a84e3b38c78 in _PyMem_DebugRawAlloc ../Objects/obmalloc.c:3024 #3 0x5a84e3b38ce0 in _PyMem_DebugRawMalloc ../Objects/obmalloc.c:3057 #4 0x5a84e3b3a68f in _PyMem_DebugMalloc ../Objects/obmalloc.c:3222 #5 0x5a84e3b63710 in PyObject_Malloc ../Objects/obmalloc.c:1630 #6 0x5a84e3b9574d in _PyObject_MallocWithType ../Include/internal/pycore_object_alloc.h:46 #7 0x5a84e3b9574d in _PyType_AllocNoTrack ../Objects/typeobject.c:2516 #8 0x5a84e3b958d9 in PyType_GenericAlloc ../Objects/typeobject.c:2547 #9 0x5a84e3b9a2a4 in object_new ../Objects/typeobject.c:7278 #10 0x5a84e3b9a6e1 in type_call ../Objects/typeobject.c:2460 #11 0x5a84e3a6aabe in _PyObject_MakeTpCall ../Objects/call.c:242 #12 0x5a84e3a6ad66 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:134 #13 0x5a84e3a6adbf in PyObject_Vectorcall ../Objects/call.c:327 #14 0x5a84e3ce9cf7 in _Py_VectorCallInstrumentation_StackRefSteal ../Python/ceval.c:769 #15 0x5a84e3cf9cbb in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1817 #16 0x5a84e3d30f30 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:118 #17 0x5a84e3d31296 in _PyEval_Vector ../Python/ceval.c:2132 #18 0x5a84e3d3154c in PyEval_EvalCode ../Python/ceval.c:680 #19 0x5a84e3e34d31 in run_eval_code_obj ../Python/pythonrun.c:1366 #20 0x5a84e3e35077 in run_mod ../Python/pythonrun.c:1469 #21 0x5a84e3e35fac in pyrun_file ../Python/pythonrun.c:1294 #22 0x5a84e3e38de2 in _PyRun_SimpleFileObject ../Python/pythonrun.c:518 #23 0x5a84e3e3908e in _PyRun_AnyFileObject ../Python/pythonrun.c:81 #24 0x5a84e3e8e6b6 in pymain_run_file_obj ../Modules/main.c:410 #25 0x5a84e3e8e923 in pymain_run_file ../Modules/main.c:429 #26 0x5a84e3e90121 in pymain_run_python ../Modules/main.c:691 #27 0x5a84e3e907b7 in Py_RunMain ../Modules/main.c:772 #28 0x5a84e3e909a3 in pymain_main ../Modules/main.c:802 #29 0x5a84e3e90d28 in Py_BytesMain ../Modules/main.c:826 #30 0x5a84e38f5675 in main ../Programs/python.c:15 SUMMARY: AddressSanitizer: heap-use-after-free ../Include/object.h:313 in _Py_TYPE_impl Shadow bytes around the buggy address: 0x513000044380: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa 0x513000044400: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 0x513000044480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x513000044500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x513000044580: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa =>0x513000044600: fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd 0x513000044680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x513000044700: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa 0x513000044780: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 0x513000044800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x513000044880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==64296==ABORTING
Suggested Fix
Py_INCREF(key) before calling encoder_listencode_obj() and Py_DECREF(key) afterward to ensure the key stays alive through the error path.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux