Conversation
|
The tests pass with this, but I don't think they are correct. Cold install doesn't seem to be doing anything https://github.com/lincolnloop/python-package-manager-shootout/actions/runs/18383090552/job/52374495457 |
|
For the benchmarks to be comparable, it seems all package managers should run using the same Python version? (And ideally the newest Python version that supports all of the benchmark's dependencies) Currently I see some inconsistent Python versions, eg: |
|
Yes, the intent is that they're all on the same Python version. Not sure why Poetry is pulling 3.10. Would be happy for a PR to get that on 3.11 like the rest. Also happy for a PR that bumps to a newer Python version. I'm not sure what the max supported by these dependencies is. That may be why we're on 3.11 :) |
It's recommended in their docs and pixi.toml seemed to be jumping to Python 3.14
- Add `rm -rf ~/.cache/uv` to `pixi-clean-cache`: pixi uses uv as its PyPI backend, and `pixi clean cache` only clears the conda package cache (~/.pixi/pkgs), leaving the uv cache (~/.cache/uv) intact. This meant "cold" installs were hitting the warm uv cache, making cold and warm install times identical. - Uncomment pixi job in benchmark.yml and add pixi to gather's needs, now that the Python version and cold install issues are resolved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
7713d4b to
ebc984c
Compare
Changes in this PRThree issues were addressed to get the pixi benchmark working correctly: 1. Python version jumping to 3.14Switched from 2. Cold install not actually being cold
Fix: added 3. Re-enabled pixi in the benchmark workflowUncommented the pixi job in ProofAll six jobs pass in run 22916739188, including pixi. The cold vs warm install times now show a meaningful difference:
That ~6× difference confirms the cache is actually being cleared for cold runs. |
Summary
Fixes two issues that were causing the pixi benchmark to be yanked:
1. Python version jumping to 3.14
Switched from
pixi.tomltopyproject.toml(recommended by pixi docs). The oldpixi.tomlwithpython = "==3.11"wasn't reliably pinning the Python version. The newpyproject.tomluses the standardrequires-python = "~= 3.11.0"constraint.2. Cold install not being cold (
Makefile)pixi clean cacheonly clears the conda package cache (rattler, at~/.cache/rattler/). Pixi uses uv as its PyPI backend, whose cache lives at~/.cache/uv— a separate location that wasn't being cleared. This meant "cold" installs silently hit the warm uv cache, making cold and warm install times nearly identical.Fix: add
rm -rf ~/.cache/uvtopixi-clean-cache.Verified in run 22916739188: cold install ~6.2s vs warm install ~1.1s — a clear difference now that the uv cache is actually cleared.
3. Re-enable pixi in benchmark
Uncommented the pixi job in
benchmark.ymland added pixi to thegatherjob'sneedslist. All jobs pass including pixi.Test plan