1st1 · GitHub

I've seen a few discussions around lack of non-blocking file io support in asyncio. Normally, we assume that disk io is fast and doesn't block, but there are use cases where it can block for a long time:

  • HDDs are still very common and widely used. Huge ones can be quite slow at random access.
  • Network filesystems - sshfs, nfs, aws ebs etc.

A workaround is to use a threadpool for file io, but that's suboptimal since it's hard to interact with asyncio code.

libuv library (the one that NodeJS is built on) uses threadpools internally, to provide non-blocking file io.

I see three options for asyncio:

  1. Keep the status quo. Advise users to either assume that disk io never blocks, or use threadpools.
  2. Provide a function that will return a pair of (StreamReader, StreamWriter). Loop will use a threadpool to do the actual IO.
  3. Provide a file object with a similar API to FileIO, but with coroutines for most of its methods. Loop will use a threadpool to do the actual IO.

If (2) or (3) is acceptable, I volunteer to draft the initial implementation.

Read the original on github.com ↗