Skip to content

concurrent.futures.Executor.map with buffersize should yield from buffer and raise after executor shutdown #146392

@ebonnal

Description

@ebonnal

Bug report

Bug description:

As pointed by @salomvary in this discussion, the behavior of concurrent.futures.Executor.map with buffersize is misleading when iterating after the executor's shutdown. There is 2 cases:

  1. Create the generator, shut down the executor, start iterating:
with ThreadPoolExecutor(1) as executor:
    iterator = executor.map(str, range(8), buffersize=2)

# start iterating after the shutdown
assert next(iterator) == "0"
assert next(iterator) == "1"
# raises StopIteration
next(iterator)

In this scenario the elements from the buffer are yielded, then the iteration is stopped.

  1. Create the generator, start the iteration, shut down the executor, continue the iteration
with ThreadPoolExecutor(1) as executor:
    iterator = executor.map(str, range(8), buffersize=2)
    # start iterating before the shutdown
    assert next(iterator) == "0"

# raises "RuntimeError: cannot schedule new futures after shutdown"
next(iterator)

In this scenario a RuntimeError is raised by the first post-shutdown next, leaving not-yet-yielded results in the buffer.


I would vote for a mix of both: yield from the buffer and then raise the exception.

(the fix would be a dozen rows)

CPython versions tested on:

3.14, 3.15

Operating systems tested on:

macOS, Linux, Windows

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions