Crash report

What happened?

In the free-threaded build, io.StringIO methods are made memory-safe by taking the object's critical section. The Argument Clinic wrappers for write, seek, readline, etc. all run under Py_BEGIN_CRITICAL_SECTION(self) (e.g. _io.StringIO.readline)

	skip_optional:
	Py_BEGIN_CRITICAL_SECTION(self);
	return_value = _io_StringIO_readline_impl((stringio *)self, size);
	Py_END_CRITICAL_SECTION();

But the tp_iternext slot stringio_iternext is not a clinic function and takes no critical section. It advances the stream by calling the internal _stringio_readline helper directly.

	stringio_iternext(PyObject *op)
	{
	PyObject *line;
	stringio *self = stringio_CAST(op);

	CHECK_INITIALIZED(self);
	CHECK_CLOSED(self);
	ENSURE_REALIZED(self);

	if (Py_IS_TYPE(self, self->module_state->PyStringIO_Type)) {
	/* Skip method call overhead for speed */
	line = _stringio_readline(self, -1);
	}

Meanwhile write_str (reached from the locked write) reads self->pos/self->string_size and resizes/reallocates self->buf, so next(sio) / for line in sio running concurrently with sio.write() (or seek) reads the buffer and advances self->pos without the lock that the writer holds.

	if (self->pos + len > self->string_size) {
	if (resize_buffer(self, self->pos + len) < 0)

This fails in two ways: the iterator reads a buffer the writer is reallocating (a use-after-free read, seen in the write_str vs _stringio_readline and mimalloc-heap races below), and the iterator's concurrent bump of self->pos makes write_str compute an out-of-bounds memset (stringio.c:253) that writes past the buffer and segfaults.

Reproducer:

import io
from threading import Thread, Barrier
sio = io.StringIO('line\n' * 200)
b = Barrier(9)
def writer():
    b.wait()
    for i in range(20000):
        try:
            sio.write('x' * ((i % 4000) + 1))
        except Exception:
            pass
def iterer():
    b.wait()
    for _ in range(20000):
        try:
            for _ in sio:
                break
        except Exception:
            pass
        try:
            sio.seek(0)
        except Exception:
            pass
threads  = [Thread(target=writer)]
threads += [Thread(target=iterer) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Report: (Segmentation fault)

==================
WARNING: ThreadSanitizer: data race (pid=43515)
  Write of size 8 at 0x7fffb63421e8 by thread T1:
    #0 write_str /cpython/./Modules/_io/stringio.c:267:15
    #1 _io_StringIO_write_impl /cpython/./Modules/_io/stringio.c:575:21
    #2 _io_StringIO_write /cpython/./Modules/_io/clinic/stringio.c.h:254:20
    #3 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4453:35
...
  Previous read of size 8 at 0x7fffb63421e8 by thread T9:
    #0 _stringio_readline /cpython/./Modules/_io/stringio.c:365:15
    #1 stringio_iternext /cpython/./Modules/_io/stringio.c:421:16
    #2 _PyForIter_VirtualIteratorNext /cpython/Python/ceval.c:3744:22
    #3 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:6441:36
...
SUMMARY: ThreadSanitizer: data race /cpython/./Modules/_io/stringio.c:267:15 in write_str
==================
...
==================
WARNING: ThreadSanitizer: data race (pid=43515)
  Atomic write of size 8 at 0x7fffb8340000 by thread T1:
    #0 mi_block_set_nextx /cpython/./Include/internal/mimalloc/mimalloc/internal.h:652:3
    #1 mi_block_set_next /cpython/./Include/internal/mimalloc/mimalloc/internal.h:678:3
    #2 mi_page_free_list_extend /cpython/Objects/mimalloc/page.c:612:5
    #3 mi_page_extend_free /cpython/Objects/mimalloc/page.c:670:5
    #4 mi_page_init /cpython/Objects/mimalloc/page.c:734:3
    #5 mi_page_fresh_alloc /cpython/Objects/mimalloc/page.c:299:3
    #6 mi_large_huge_page_alloc /cpython/Objects/mimalloc/page.c:874:21
    #7 mi_find_page /cpython/Objects/mimalloc/page.c:916:14
    #8 _mi_malloc_generic /cpython/Objects/mimalloc/page.c:950:21
    #9 _mi_heap_malloc_zero_ex /cpython/Objects/mimalloc/alloc.c:161:21
    #10 _mi_heap_malloc_zero /cpython/Objects/mimalloc/alloc.c:179:10
    #11 mi_heap_malloc /cpython/Objects/mimalloc/alloc.c:183:10
    #12 _mi_heap_realloc_zero /cpython/Objects/mimalloc/alloc.c:732:16
    #13 mi_heap_realloc /cpython/Objects/mimalloc/alloc.c:753:10
    #14 _PyMem_MiRealloc /cpython/Objects/obmalloc.c:289:12
    #15 PyMem_Realloc /cpython/Objects/obmalloc.c:1279:12
    #16 resize_buffer /cpython/./Modules/_io/stringio.c:116:26
    #17 write_str /cpython/./Modules/_io/stringio.c:239:13
    #18 _io_StringIO_write_impl /cpython/./Modules/_io/stringio.c:575:21
    #19 _io_StringIO_write /cpython/./Modules/_io/clinic/stringio.c.h:254:20
    #20 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4453:35
...
  Previous read of size 4 at 0x7fffb8340004 by thread T6:
    #0 ucs4lib_find_max_char /cpython/Objects/stringlib/find_max_char.h:80:31 (python3.16t+0x372ba4)
    #1 _PyUnicode_FromUCS4 /cpython/Objects/unicodeobject.c:2216:16
    #2 PyUnicode_FromKindAndData /cpython/Objects/unicodeobject.c:2290:16
    #3 _stringio_readline /cpython/./Modules/_io/stringio.c:384:12
    #4 stringio_iternext /cpython/./Modules/_io/stringio.c:421:16
    #5 _PyForIter_VirtualIteratorNext /cpython/Python/ceval.c:3744:22
    #6 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:6441:36
...
SUMMARY: ThreadSanitizer: data race /cpython/./Include/internal/mimalloc/mimalloc/internal.h:652:3 in mi_block_set_nextx
==================
...
==================
WARNING: ThreadSanitizer: data race (pid=43515)
  Write of size 4 at 0x7fffb8340018 by thread T1:
    #0 as_ucs4 /cpython/Objects/unicodeobject.c:2480:9
    #1 PyUnicode_AsUCS4 /cpython/Objects/unicodeobject.c:2505:12
    #2 write_str /cpython/./Modules/_io/stringio.c:259:10
    #3 _io_StringIO_write_impl /cpython/./Modules/_io/stringio.c:575:21
    #4 _io_StringIO_write /cpython/./Modules/_io/clinic/stringio.c.h:254:20
    #5 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4453:35
...
  Previous read of size 4 at 0x7fffb8340018 by thread T7:
    #0 _PyUnicode_FromUCS4 /cpython/Objects/unicodeobject.c:2221:9
    #1 PyUnicode_FromKindAndData /cpython/Objects/unicodeobject.c:2290:16
    #2 _stringio_readline /cpython/./Modules/_io/stringio.c:384:12
    #3 stringio_iternext /cpython/./Modules/_io/stringio.c:421:16
    #4 _PyForIter_VirtualIteratorNext /cpython/Python/ceval.c:3744:22
    #5 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:6441:36
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/unicodeobject.c:2480:9 in as_ucs4
==================
...
ThreadSanitizer:DEADLYSIGNAL
==43515==ERROR: ThreadSanitizer: SEGV on unknown address 0x7fffaa100000 (pc 0x7ffff7d8954a bp 0x000000000000 sp 0x7fffb11fc7c8 T43517)
==43515==The signal is caused by a WRITE memory access.
    #0 __memset_avx2_unaligned_erms string/../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:328
    #1 __tsan_memset <null> (python3.16t+0xf52a6)
    #2 write_str /cpython/./Modules/_io/stringio.c:253:9
    #3 _io_StringIO_write_impl /cpython/./Modules/_io/stringio.c:575:21
    #4 _io_StringIO_write /cpython/./Modules/_io/clinic/stringio.c.h:254:20
    #5 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4453:35
    #6 _PyEval_EvalFrame /cpython/./Include/internal/pycore_ceval.h:122:16
    #7 _PyEval_Vector /cpython/Python/ceval.c:2141:12
    #8 _PyFunction_Vectorcall /cpython/Objects/call.c
    #9 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11
    #10 _PyObject_VectorcallPrepend /cpython/Objects/call.c:855:20
    #11 method_vectorcall /cpython/Objects/classobject.c:55:12
    #12 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11
    #13 context_run /cpython/Python/context.c:731:29
    #14 method_vectorcall_FASTCALL_KEYWORDS /cpython/Objects/descrobject.c:421:24
    #15 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11
    #16 PyObject_Vectorcall /cpython/Objects/call.c:327:12
    #17 _Py_VectorCallInstrumentation_StackRefSteal /cpython/Python/ceval.c:768:11
    #18 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:1906:35
    #19 _PyEval_EvalFrame /cpython/./Include/internal/pycore_ceval.h:122:16
    #20 _PyEval_Vector /cpython/Python/ceval.c:2141:12
    #21 _PyFunction_Vectorcall /cpython/Objects/call.c
    #22 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11
    #23 _PyObject_VectorcallPrepend /cpython/Objects/call.c:855:20
    #24 method_vectorcall /cpython/Objects/classobject.c:55:12
    #25 _PyVectorcall_Call /cpython/Objects/call.c:273:16
    #26 _PyObject_Call /cpython/Objects/call.c:348:16
    #27 PyObject_Call /cpython/Objects/call.c:373:12
    #28 thread_run /cpython/./Modules/_threadmodule.c:388:21
    #29 pythread_wrapper /cpython/Python/thread_pthread.h:234:5
    #30 __tsan_thread_start_func <null>
    #31 start_thread nptl/pthread_create.c:447:8
    #32 clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV string/../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:328 in __memset_avx2_unaligned_erms
==43515==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

gh-153296: Fix thread-safety data race in io.StringIO iterator #153368