Skip to content

gh-135871: Reload lock internal state while spinning in PyMutex_LockTimed#146064

Merged
colesbury merged 5 commits intopython:mainfrom
dpdani:pymutex-spinning-reload
Mar 23, 2026
Merged

gh-135871: Reload lock internal state while spinning in PyMutex_LockTimed#146064
colesbury merged 5 commits intopython:mainfrom
dpdani:pymutex-spinning-reload

Conversation

@dpdani
Copy link
Contributor

@dpdani dpdani commented Mar 17, 2026

This PR adds atomic loads in the slow path of PyMutex to increase the number of lock acquisitions per second that threads can make on a shared mutex.

The tricky part here is to avoid degrading the performance when the lock is highly contended; that is, when many threads are trying to acquire the mutex at a high frequency. This is because the current strategy of never reloading the mutex's state is, maybe counter-intuitively, the best strategy: avoid disturbing the thread that currently holds the mutex.

I've run the lockbench script to assess the performance using the following two scenarios, which were suggested by @colesbury:

  • low contention:
./python.exe Tools/lockbench/lockbench.py --work-inside 5 --work-outside 50 --num-locks 24 --acquisitions 3 --random-locks
  • high contention:
./python.exe Tools/lockbench/lockbench.py --work-inside 5 --work-outside 5

The results are from my M4 MacBook, and Python was compiled with --disable-gil --enable-optimizations --with-lto.

The different lines below represent different reloading strategies:

  • old: main
  • unconditional: remove the if statement at line 107
  • RELOAD_SPIN_COUNT = ...: tweak the value at line 30

The RELOAD_SPIN_COUNT = ... strategies also pseudo-randomize the reload recurrence by having each thread add its own thread-id to the counter. On my machine this nets a ~15/20% improvement over the same strategy, without this pseudo-randomization.

lockbench_compare

And here's the comparison between main and this PR, which picks the RELOAD_SPIN_COUNT = 3 strategy:

lockbench_old_vs_spin_3

The fact that the high contention case shows as an improvement in this chart is misleading: I'd say it's in the noise range.

Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments, but otherwise LGTM.

dpdani and others added 3 commits March 23, 2026 16:06
Co-authored-by: Sam Gross <colesbury@gmail.com>
Change-Id: I5df04888df81de26a04450a9a1a04d4db5026383
@colesbury colesbury merged commit daa159f into python:main Mar 23, 2026
93 of 94 checks passed
CuriousLearner added a commit to CuriousLearner/cpython that referenced this pull request Mar 24, 2026
…on into fix-pythongh-138577

* 'fix-pythongh-138577' of github.com:CuriousLearner/cpython:
  pythongh-146202: Create tmp_dir in regrtest worker (python#146347)
  pythongh-144319: obtain SeLockMemoryPrivilege on Windows (python#144928)
  pythongh-146199: Fix error handling in `code_richcompare` when `PyObject_RichCompareBool` fails (python#146200)
  pythongh-146197: Include a bit more information in sys._emscripten_info.runtime (python#146346)
  pythongh-135871: Reload lock internal state while spinning in `PyMutex_LockTimed` (pythongh-146064)
  pythongh-145719: Add `.efi` file detection in `mimetypes` (python#145720)
@dpdani dpdani deleted the pymutex-spinning-reload branch March 24, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants