Zarrista is a new Python library for working with Zarr data. Implemented in Rust, it delivers significant performance improvements over Zarr-Python, the go-to library for handling Zarr data in Python. Work is currently ongoing to integrate Zarrista into Zarr-Python to bring those benefits to the wider Python community.
to avoid fracturing the ecosystem, we plan to integrate Zarrista as a native backend into Zarr-Python.
We hope that a Zarrista backend for Zarr-Python can deliver these performance improvements without any need for end users to change their code. Follow Zarr-Python PR #4064, which uses Zarrista to prototype generic backends for Zarr-Python.
Under the hood, Zarrista uses the Rust Zarr library Zarrs; Zarrista itself only introduces a thin layer between Zarr-Python and Zarrs, adding almost no Zarr-specific functionality.
Zarrista is built as a “dumb, direct” binding to Zarrs, the Rust Zarr library. Zarrista exposes as many APIs as possible from Zarrs, while avoiding creating its own APIs from scratch.
This means that Zarrista’s source code contains no Zarr-specific logic at all. All Zarr-specific logic is either lower-level in Zarrs or higher-level in Zarr-Python. This keeps Zarrista itself maintainable and means that any improvements can be submitted to Zarrs directly, which benefit both Rust and Python users.
The performance gains are handsome: according to the benchmarks, there’s a 1.9× to 2.7× boost compared to Zarr-Python when reading local data.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.