PEP 803 – “abi3t”: Stable ABI for Free-Threaded Builds

[encukou]

Please consider PEP 803 – “abi3t”: Stable ABI for Free-Threaded Builds
https://peps.python.org/pep-0803/

This is where we are for 3.15. The concept is stable and proven feasible; naming & packaging compromises are acceptable.

The C API WG voting/commenting is in progress.
Since this is partly a packaging PEP, I've also asked Paul for pronouncement: https://discuss.python.org/t/104976/72

You should be aware of @zooba's competing PEP 809. (Personally, I think it's not feasible in 3.15 but it can be rewritten as a follow-up proposal.)

---

• The PEP has been discussed in threads listed in its Post-History header
• The PEP was announced on Discuss (link in Post-History)
• The PEP includes all relevant Suggested Sections
  (that list changed since the PEP was first written...)
• The PEP includes endorsements from the projects/groups/people it helps
• The PEP has a CODEOWNERS entry

SIG-specific:

• Typing PEPs: link to the Typing Council's recommendation for the PEP:
• Packaging PEPs: don't file the issue here, ask the delegate (Paul Moore) on Packaging Discourse

[eli-schwartz]

You should be aware of @zooba's competing PEP 809. (Personally, I think it's not feasible in 3.15 but it can be rewritten as a follow-up proposal.)

The competing proposal competes on multiple points. Maybe it's not feasible to design and deploy a rolling ABI window, but that shouldn't really impact whether it's feasible to design and deploy "abi4.so" in the filename. It is essentially appending a string to an existing list (of filename patterns that are looked up in turn).

So PEP 809 is not (just) a "followup proposal". It needs to be considered in the context of PEP 803 today in order to determine whether PEP 803 has flaws that require revision and resubmission to go together with "defer PEP 809 as a followup".

(The lack of extension tags is imo a major flaw and will cause a lot of trouble for downstream redistributors of python software. The fact that PEP 803 brushes this aside by essentially saying that it's "an additional technical change and not really necessary, therefore it is rejected in order to keep the PEP smaller" troubles me.)

[ngoldbaum]

In support of this proposal, Quansight has been working on updating packaging libraries, build tools, and bindings generators to work with an opaque PyObject ABI stable ABI. See the tracking issue for the latest: Quansight-Labs/free-threaded-compatibility#310.

This is all based on the _Py_OPAQUE_PYOBJECT macro available in 3.15 alphas.

We've also been working on a set of end-to-end tests for packaging tools and are experimenting with building real-world projects, see: https://github.com/Quansight-Labs/stable-abi-testing/. I'm still in the process of updating the tests to reflect work that's happened in the last couple of weeks, but that should be done by the time the steering council is reviewing the PEP.

So far, we have patched setuptools, packaging, meson-python, CFFI, Cython, and PyO3. We have not encountered any significant issues, although we have raised the need for a PyCriticalSection stable ABI (see discussion here). Separately, the maintainer of nanobind asked for a way to get instance data besides PyObject_GetTypeData(), since that requires knowledge of CPython static types to work correctly. I don't think these are difficult problems to solve, but it does require some action from the C API workgroup. Having some endorsement from the steering council for PEP 803 would probably help drive those discussions forward.

The lack of extension tags is imo a major flaw and will cause a lot of trouble for downstream redistributors of python software.

@eli-schwartz I'm not totally sure what this refers to. Can you elaborate a bit? Or maybe link to where this came up in the discourse discussion ?