Summary

This fixes a reference leak in the set encoders used by both JSON and MessagePack.

json_encode_set and mpack_encode_set iterate with PyIter_Next, which returns a new reference for each item. Those item references were not decref'd after successful encoding, so set elements could be kept alive longer than intended.

Changes

• add missing Py_DECREF(item) in json_encode_set
• add missing Py_DECREF(item) in mpack_encode_set
• preserve correct cleanup on encode failure
• add a regression test covering both msgspec.json and msgspec.msgpack

Testing

• rebuilt the extension locally
• ran a focused regression script confirming set items are collectable after encoding for both protocols

Notes

This script reproduces the bug. Repeatedly encoding sets causes the memory to quickly climb to several GB.

import msgspec
encoder = msgspec.msgpack.Encoder()
for i in range(50_000):
    encoded = encoder.encode(set(range(10_000)))