Bug report
Bug description:
Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False) is used
- When max_workers < number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len() - When max_workers > number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len()AND then program hangs
There is no problem when shutdown(wait=True) is used or if the worker processes exit cleanly.
import concurrent.futures import multiprocessing as mp import time def task(n: int) -> int: if n == 2: raise Exception("Not gonna do it") else: time.sleep(n) return n def main() -> None: # max_workers=2 results in raised exception and program exits # max_workers=4 results in raised exception and program hangs # executor = concurrent.futures.ProcessPoolExecutor( max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1 ) f1 = executor.submit(task, 1) f2 = executor.submit(task, 2) f3 = executor.submit(task, 3) result = 0 try: result += f1.result() result += f2.result() result += f3.result() print(f"Result = {result}") except Exception as e: print(f"Exception while getting result : {e}") executor.shutdown(wait=False) if __name__ == "__main__": main()
CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
- [3.12] gh-132969: Fix exception/hang shutdown(wait=False) and a task exited abnormally #133220
- gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222
- [3.13] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135343
- [3.14] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135344
- gh-132969: ACKS file update was missed in gh-133222 #136144