Bug report
tl;dr; See stack overflow post
When copying a bytes object to a shareable list, the trailing zeros are stripped causing data loss. This doesn't appear in the documentation as far as I can tell, and seems to be unexpected behavior related to the implementation.
Example code:
from multiprocessing import shared_memory as shm shmList = shm.ShareableList([bytes(50)]) testBytes = bytes.fromhex("00112233445566778899aabbccddeeff0000") shmList[0] = testBytes print(testBytes) print(shmList[0]) shmList.shm.close() shmList.shm.unlink()
Output:
b'\x00\x11"3DUfw\x88\x99\xaa\xbb\xcc\xdd\xee\xff\x00\x00'
b'\x00\x11"3DUfw\x88\x99\xaa\xbb\xcc\xdd\xee\xff'
Offending portion of CPython code:
_back_transforms_mapping = { 0: lambda value: value, # int, float, bool 1: lambda value: value.rstrip(b'\x00').decode(_encoding), # str 2: lambda value: value.rstrip(b'\x00'), # bytes 3: lambda _value: None, # None }
Linked PRs
- gh-106939: document ShareableList nul-strip quirk. #107266
- [3.12] gh-106939: document ShareableList nul-strip quirk. (GH-107266) #107269
- [3.11] gh-106939: document ShareableList nul-strip quirk. (GH-107266) #107270
- gh-106939: Keep the trailing null bytes in ShareableList #144559
- gh-106939, gh-145261: Fix ShareableList data corruption #145488