Bug report
Bug description:
In the free-threaded build, a whole-buffer BytesIO.read() returns self->buf by reference (Py_NewRef) instead of a copy,
| if (size > 1 && | |
| self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) && | |
| FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) { | |
| self->pos += size; | |
| return Py_NewRef(self->buf); | |
| } |
so resize_buffer_lock_held() can reallocate that same self->buf in place (_PyBytes_Resize)
| if (SHARED_BUF(self)) { | |
| if (unshare_buffer_lock_held(self, alloc) < 0) | |
| return -1; | |
| } | |
| else { | |
| if (_PyBytes_Resize(&self->buf, alloc) < 0) | |
| return -1; | |
| } |
while another thread concurrently decrefs the read-returned reference (_Py_DecRefShared), racing on the same buffer object without synchronization.
Reproducer:
import io from threading import Thread, Barrier shared = io.BytesIO(b'x' * 64) N_W, N_R = 2, 8 barrier = Barrier(N_W + N_R) def writer(): barrier.wait() for i in range(20000): try: shared.seek(0) shared.write(b'a' * (64 + (i & 63))) except Exception: pass def reader(): barrier.wait() for _ in range(20000): try: shared.seek(0) shared.read() except Exception: pass if __name__ == "__main__": threads = [Thread(target=writer) for _ in range(N_W)] threads += [Thread(target=reader) for _ in range(N_R)] for t in threads: t.start() for t in threads: t.join()
TSAN Report:
==================
WARNING: ThreadSanitizer: data race (pid=2292869)
Read of size 8 at 0x7fffbc080190 by thread T2:
#0 _PyObject_MiRealloc /cpython/Objects/obmalloc.c:360:13
#1 PyObject_Realloc /cpython/Objects/obmalloc.c:1731:12
#2 _PyBytes_Resize /cpython/Objects/bytesobject.c:3389:9
#3 resize_buffer_lock_held /cpython/./Modules/_io/bytesio.c:176:13
#4 write_bytes_lock_held /cpython/./Modules/_io/bytesio.c:216:13
#5 _io_BytesIO_write_impl /cpython/./Modules/_io/bytesio.c:782:20
#6 bytesio_setstate_lock_held /cpython/./Modules/_io/bytesio.c:927:14
#7 bytesio_setstate /cpython/./Modules/_io/bytesio.c:980:11
#8 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4261:35
...
Previous atomic write of size 8 at 0x7fffbc080190 by thread T8:
#0 _Py_atomic_compare_exchange_ssize /cpython/./Include/cpython/pyatomic_gcc.h:130:10
#1 _Py_DecRefSharedIsDead /cpython/Objects/object.c:406:15
#2 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9
#3 _Py_DecRefShared /cpython/Objects/object.c:433:5
#4 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/obmalloc.c:360:13 in _PyObject_MiRealloc
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux