larkost · GitHub

The Python driver does not seem to be playing nice with threading, with a variety of problems:

  • Connection is broken exceptions on queries (but subsequent calls can work)
  • No JSON object could be decoded exceptions
  • hanging forever on queries, sometimes seen as a hang on threading.join()

I will be opening a branch on this, and in two subsequent checkins add a test that shows this, then a Proof-of-Concept fix. But there should probably be a discussion around what the proper strategy should be. My initial thoughts on the options:

  1. Explicitly declare that connections belong ONLY to the thread that started them and provide a reasonable exception to this end when people try to use the connection from another thread. This would be fairly easy using threading.local(). A way of making this somewhat easier to use would be to provide a method to clone a thread (i.e.: use the same connection information) from a connection object passed from another thread.
  2. Do the work within the driver to handle passing the thread object within threads. My initial Proof-of-Concept shows that at least the first part of the problem can be solved relatively simply by putting the socket connection in a thread-local object. But there is a lot of thinking that is going to been to be done about what parts of the current net.Connection object should be shared between threads and what should be thread-local, and putting appropriate locks around the former.

Note that a similar conversation is probably going to apply to things using the multiprocessing module. The same solutions can probably work for both types of (semi) concurrent methods, but we would need to confirm that both work.

Read the original on github.com ↗