jcrist · GitHub

@jcrist

This PR changes the msgpack decoder to avoid preallocating large arrays.
This reduces the ability for malicious messages to allocate large
amounts of memory with only a small payload.
We still preallocate small arrays (current threshold is 16 elements).
This means that a crafted message of N bytes could result in at most
`N * 184` bytes being allocated, with `N` being at most
`sys.getrecursionlimit()`. Dropping to no preallocations would reduce
this to `N * 56` bytes (~1/3 of the above usage), but comes with a ~20%
performance decrease for small arrays. Given common CPython
configurations, this means that a 3000 byte message could result in at
most ~500 KiB being allocated, which is low enough to be acceptable.
With this change, all allocations in the decoder scale only with the
size of the message. Limiting the size of the input message will
effectively limit the size of the decoded message.

Read the original on github.com ↗