Cross-interpreter memory leak in _xidata_release when originating interpreter is destroyed

[imabd645]

Bug report

Bug description:

Bug report

Bug summary
There is a known memory leak in Python/crossinterp.c related to the teardown of sub-interpreters when they share data.

When a sub-interpreter packages data into _PyXIData_t and passes it to a receiving interpreter, the receiver is expected to call _PyXIData_Release to free it. However, if the originating interpreter is destroyed before the receiver calls release, the original interpreter's memory context is lost.

In _xidata_release (Python/crossinterp.c around line 1049), the code correctly identifies this state:

    PyInterpreterState *interp = _PyInterpreterState_LookUpID(
                                        _PyXIData_INTERPID(xidata));
    if (interp == NULL) {
        // The interpreter was already destroyed.
        // This function shouldn't have been called.
        // XXX Someone leaked some memory...

Because the originating interpreter is gone, Python intentionally leaks xidata->obj to avoid a use-after-free segfault.

Proposed Solution Architecture
To resolve this without crashing, we would need to track cross-interpreter data at the interpreter level:

1. Modify PyInterpreterState to include a thread-safe registry tracking all active _PyXIData_t instances created by that interpreter.
2. Register instances during _PyCode_GetXIData and deregister them during _xidata_release.
3. Modify Py_EndInterpreter teardown to check this registry. If outstanding shared data exists, the shutting-down interpreter must either block until it is released, or forcefully invalidate/free the shared data (using an is_invalidated flag) before tearing down its own obmalloc state.

I'm opening this issue to get feedback on the proposed architecture before starting a Pull Request.

CPython versions tested on:
main branch

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-153713: Fix cross-interpreter memory leak in _xidata_release via atomic claim-and-free #153745

[Builder106]

Hey @imabd645. Nice detailed write-up 👌. Sticking a registry on PyInterpreterState to track active _PyXIData_t instances makes total sense to me. We definitely need a way to keep that memory context from vanishing during teardown.

Just one thing to keep in mind for the actual implementation: since the new free-threading (PEP 703) build is in main now, we'll probably need to make this registry a lock-free linked list or wrap it in a pretty heavy mutex. Otherwise, _PyCode_GetXIData and _xidata_release will be hammered by concurrent threads trying to receive data simultaneously.

Forcefully invalidating the shared data when Py_EndInterpreter runs definitely seems like the safest bet to avoid any gnarly shutdown hangs.

Are you planning to take a stab at a PR for this?

[imabd645]

Thanks @Builder106! That's a great point about the free-threading build. Wrapping the registry in a PyMutex and going with the forceful invalidation route on teardown sounds like the most robust plan.

Yes, I am planning to take a stab at a PR for this! I'll start putting it together soon