Skip to content

gh-142837: Use semaphore accounting for multiprocessing.Queue.empty()#144832

Open
iamrajhans wants to merge 7 commits intopython:mainfrom
iamrajhans:gh-142837-mp-queue-empty-semaphore
Open

gh-142837: Use semaphore accounting for multiprocessing.Queue.empty()#144832
iamrajhans wants to merge 7 commits intopython:mainfrom
iamrajhans:gh-142837-mp-queue-empty-semaphore

Conversation

@iamrajhans
Copy link
Copy Markdown
Contributor

@iamrajhans iamrajhans commented Feb 15, 2026

Summary

Fixes gh-142837 by making multiprocessing.Queue.empty() use the queue semaphore count (when available) instead of pipe readiness, so it aligns better with qsize() / full() behavior.

What Changed

  • Lib/multiprocessing/queues.py

    • Updated Queue.empty() to:
      • use self._sem.get_value() == self._maxsize when sem_getvalue() is available
      • fall back to not self._poll() when sem_getvalue() is unavailable (e.g. macOS)
    • Preserved closed-queue behavior where empty() may raise OSError.
  • Lib/test/_test_multiprocessing.py

    • Added test_empty_uses_semaphore_count to verify empty() is driven by semaphore accounting rather than pipe poll state when supported.
  • Doc/library/multiprocessing.rst

    • Clarified queue feeder-thread delay implications for get_nowait().
    • Documented platform-dependent behavior of Queue.empty().
    • Added a versionchanged:: 3.15 note for semaphore-based empty() behavior.

Why

Queue.empty() previously relied on pipe readiness, which could report empty while items were already accounted for in queue size/semaphore state. This caused asymmetric behavior versus qsize()/full() and surprising results under load.


📚 Documentation preview 📚: https://cpython-previews--144832.org.readthedocs.build/

q.put('sentinel')
original_poll = q._poll
q._poll = lambda timeout=0.0: False
try:
Copy link
Copy Markdown
Contributor

@YvesDup YvesDup Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if try and finally are very useful here ?

q._poll = original_poll

self.assertEqual(q.get(timeout=support.SHORT_TIMEOUT), 'sentinel')
self.assertTrue(q.empty())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe adding a test when queue is empty which looks like your first one just above ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

multiprocessing.Queue methods have asymmetric behavior

2 participants