CPython weighs race-free objects, export rules, and a one-shot ABI break for free threading

Free-threaded CPython is no longer a distant experiment. As the no-GIL builds mature, the language is being asked to decide what safety, public surface, and binary compatibility actually mean when objects can be shared without the historic lock. Four related proposals now sit on the table at once: Mark Shannon’s PEP 805 for race-free object states, the public/private builtins of PEP 844, Neil Girdhar’s export statement in PEP 843, and Shannon’s separate call for a one-time stable-ABI break after 3.15. Together they form a single argument about how much of today’s Python the community is willing to rewrite in exchange for parallelism that is both fast and maintainable.

PEP 805 is the sharpest statement of the safety side. After a year of intermittent work, Shannon presented it as the unification of PEP 703 and PEP 734: “parallel execution of code is race free by default: objects must be explicitly declared to be safe to be shared between parallel threads, or such sharing is prohibited.” Extra per-object state would let the runtime check, cheaply, whether an operation is legal and raise when it is not. The claim is better safety than free-threading alone, better sharing than subinterpreters, and better performance than either. In the discussion Shannon insisted the migration path is gentle: “With a single ThreadGroup, this PEP degenerates to the behavior of the current with-GIL build, with a few minor exceptions. Nothing should break.”

Thomas Wouters was unconvinced. Posting as himself rather than for the Steering Council, he called out “pretty significant gaps, as well as a couple of fundamental misunderstandings” and said he could not see how the scheme “can be introduced without breaking backward compatibility in extremely wide-spread, significant ways.” He also rejected the premise that PEP 703 invents a new threading model; Python’s model has been the same for decades, GIL or no GIL, and free-threading mainly makes existing races more visible. The concrete counter-example that kept resurfacing was the ujson crash under concurrent mutation: a pure-Python default callback releases the GIL, a second thread clears the list, and the C extension dies. Guido van Rossum pressed for clarity on the claim that “relying purely on the GIL for thread-safety is already provably unsafe.” After the exchange he concluded that C extensions must be “very careful” about every call that might drop the GIL, while ordinary Python never treated the GIL as a thread-safety guarantee in the first place. The += 1 case that looks atomic is riddled with caveats about exact types and the absence of arbitrary destructors. Shannon’s runtime states would make those caveats explicit; critics still see an enforcement and compatibility mountain.

Running in parallel is the public/private trilogy. Barry Warsaw published PEP 844, adding public() and private() builtins that keep all synchronized with the names a module actually intends to expose. The motivation is the decade of experience behind the atpublic library and the chronic drift between documented APIs and what from module import * or static tools actually see. Skeptics asked the obvious question: “what’s wrong with all?” It already works, lives at module scope, and is widely understood. Defenders answered that the status quo is scattered across the language reference, typing docs, and PEP 8, that accidental leakage is common, and that a local decorator avoids both name duplication and the need for every small library to grow an API-hub module. Guido’s contribution was characteristically practical: when Tim Hoffmann observed that the real deficit might simply be coherent documentation, Guido replied “Why don’t you give that a try here,” separating runtime behavior, type-checker rules, and mere convention.

PEP 843 occupies the narrowest niche of the same problem. Girdhar’s from x export y [as z] performs the import and appends the name to all in one step, eliminating the double bookkeeping that hub modules (package init files that re-export a public façade) currently maintain by hand. In the scipy.stats stubs the win would be dramatic: hundreds of names could drop out of an explicit all list. Paul Moore tested the idea against real packaging patterns and found fewer clean wins than advertised; many hubs already compute all dynamically or traffic in submodules rather than individual names. The deeper disagreement is cultural. Some maintainers treat the leading-underscore convention as the existing standard; others note that it cannot express “public to this package, private to outsiders” once vendoring enters the picture, and that tooling therefore cannot reliably warn users away from internals.

All of the above collides with binary reality in Shannon’s ABI proposal. He asks for a single, deliberate break so that extensions compile either for the world up to 3.15 or for 3.16 and later. The prize is freedom to reshape the PyObject header: homogenize 32- and 64-bit layouts, shrink the difference between default and free-threaded builds, and place GC and allocator bits where they can be tested in one place. He estimates 1-2 % from simple refactoring and perhaps 5-10 % from better allocation and collection algorithms. Steve Dower’s objection was immediate: “Don’t break the ABI without creating a new name for the new ABI. We can’t just go breaking that promise on our users like that.” The Steering Council has so far declined to create a versioning scheme for incompatible stable-ABI revisions, leaving the project in a “no breaks” posture. Others, including scoder, argued that an ABI refresh every few years is tolerable provided the C API itself stays stable enough for straightforward recompilation; the cost of an ever-growing matrix of wheels is already real, and a clean cut may be cheaper than perpetual work-arounds. The free-threading ABI (abi3t) will eventually force a transition anyway; the open question is whether other header changes should ride the same train or be forbidden until a proper multi-ABI process exists.

What ties the four threads together is the same trade-off. Free-threading removes the GIL’s accidental serialization; the language must then decide whether objects default to shared or isolated, whether public names are declared or merely conventional, and whether the stable ABI is an absolute vow or a renewable contract. Shannon’s camp offers runtime state, explicit export, and a header redesign as the price of race-free speed and a simpler interpreter. The counter-camp sees wide breakage, redundant mechanisms beside all, and a dangerous erosion of the promise that an abi3 extension keeps working. Guido continues to appear in the threads, clarifying the historical meaning of the GIL, nudging documentation, and keeping the design space honest without yet declaring winners.

Nothing is decided. PEP 805 still needs a convincing implementation sketch and a compatibility story that survives Thomas Wouters’s objections. The public/private and export PEPs must demonstrate they improve on the status quo rather than merely add another spelling. The ABI break depends on whether the project is willing to name a successor ABI or will keep papering over header constraints. The free-threaded builds are already shipping; the object model, the export surface, and the binary contract are now the rate-limiting steps. How much compatibility the community is prepared to spend remains the unresolved variable.