ogbiggles · GitHub

Bug report

Bug description:

Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False) is used

  1. When max_workers < number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len()
  2. When max_workers > number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len() AND then program hangs

There is no problem when shutdown(wait=True) is used or if the worker processes exit cleanly.

import concurrent.futures
import multiprocessing as mp
import time
def task(n: int) -> int:
    if n == 2:
        raise Exception("Not gonna do it")
    else:
        time.sleep(n)
        return n
def main() -> None:
    # max_workers=2 results in raised exception and program exits
    # max_workers=4 results in raised exception and program hangs
    #
    executor = concurrent.futures.ProcessPoolExecutor(
        max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1
    )
    f1 = executor.submit(task, 1)
    f2 = executor.submit(task, 2)
    f3 = executor.submit(task, 3)
    result = 0
    try:
        result += f1.result()
        result += f2.result()
        result += f3.result()
        print(f"Result = {result}")
    except Exception as e:
        print(f"Exception while getting result : {e}")
    executor.shutdown(wait=False)
if __name__ == "__main__":
    main()

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

Read the original on github.com ↗