Pin PyThreadStates so native extensions can cache them per thread#137
Merged
jhonabreul merged 2 commits intoJul 13, 2026
Merged
Conversation
Py.GIL() acquires the GIL via PyGILState_Ensure/Release; the outermost scope on a .NET thread creates a fresh PyThreadState and deletes it on dispose. pybind11-based extensions (matplotlib >= 3.10 _path/ft2font, scipy, PyTorch) cache the first PyThreadState* they see per OS thread in pybind11 internals TLS and never invalidate it, so once a scope deletes the thread state the next native GIL acquire on that thread restores a dangling pointer: 0xC0000005 access violation or heap corruption. Root cause of QuantConnect/Lean#9203 (ReportChartTests crashing the test host on the first matplotlib render). Pin each thread state with one extra never-released PyGILState_Ensure per thread per runtime run, keeping the gilstate counter >= 1 so the thread state lives until engine shutdown - the same lifetime CPython gives thread states it binds itself. The extra Ensure runs with the GIL already held (a bare counter increment): GIL acquire/release timing is unchanged and empty-scope microbenchmarks are within noise (0.212 -> 0.209 us/scope). Regression test: threading.local values live in the thread state dict, so a value stored in one Py.GIL scope survives a second scope on the same thread only if the thread state was not deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bump package <Version>, AssemblyVersion/AssemblyFileVersion and the perf-test baseline reference to 2.0.62. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jhonabreul
approved these changes
Jul 13, 2026
This was referenced Jul 13, 2026
jhonabreul
added a commit
that referenced
this pull request
Jul 14, 2026
* Clear pending Python error when a Try-style conversion fails PyObject.TryAsManagedObject converted with setError: true and returned false on failure, leaving the Python error indicator set on the thread with no one left to surface it. Callers of TryAs/TryAsManagedObject only observe the boolean, so the stale error survived on the thread state and was thrown by the next unrelated Python call that checks the error indicator. This was previously masked because the outermost Py.GIL scope deleted the thread state (and its pending error) on dispose; since thread states are pinned for the lifetime of the run (#137), the leaked error persists and poisons subsequent calls on the same thread - e.g. Lean's PythonUtilTests failing with "int() argument must be a string, a bytes-like object or a real number, not 'function'" leaked by an earlier test that exercises a failed TryAs<int> on a function. Convert with setError: false in TryAsManagedObject and clear any error a conversion sub-path may have left pending before returning false. AsManagedObject keeps converting with setError: true so the fetched error still becomes the InvalidCastException cause, which also consumes the indicator. * Update version to 2.0.63 Bump package <Version>, AssemblyVersion/AssemblyFileVersion and the perf-test baseline reference to 2.0.63. * Clear leaked Python errors in converter collection and label paths Several Converter failure paths could leave a Python error pending even when called with setError: false, relying on the caller to clean up: - ToList did not handle a failed PyObject_GetIter: the raised TypeError survived the call (and the iterator reference leaked). Handle it like ToArray does, disposing the reference and clearing when not setting errors. - MakeList treated a null PyIter_Next result as normal exhaustion, but null also means the iterator raised: the conversion now fails instead of returning a silently truncated list, clearing the error when not setting errors and leaving it pending for the caller when setting errors. The swallowed CLR-exception catch and the element-conversion failure path also clear when not setting errors, since some element conversion paths (failed implicit operators, deleted types) raise regardless of setError to enrich MethodBinder's no-match message. - The type_error and overflow labels now clear pending errors when not setting errors, mirroring what convert_error already did, so a probing sub-path that raised before jumping cannot leak. The unconditional raises in the implicit-conversion catches are kept: MethodBinder.Invoke deliberately surfaces a pending error as the binding failure reason instead of its generic no-match message, and Converter.TryAsManagedObject clears at the API boundary. * Clear rejected overload probe errors when a method bind succeeds MethodBinder probes each overload candidate with setError: false, but a probe can still raise: the implicit-conversion catches in Converter.ToManagedValue raise unconditionally so that Invoke can surface the specific cause (e.g. a throwing implicit operator) as the bind-failure reason instead of the generic no-match message. When the raising candidate was probed after the candidate that ultimately matched, nothing cleared the indicator before the method ran and the otherwise successful call failed with "SystemError: ... returned a result with an error set". Errors raised by earlier candidates were incidentally wiped by the per-candidate PyObject_Type/Exceptions.Clear type check, which is why the leak needs this specific overload ordering: an exact-class-match overload (precedence 40) binds first, then a string overload (precedence 50) is probed and raises. Clear any pending probe error in Bind once an overload has matched. The bind-failure path is untouched, so a pending error is still surfaced as the failure reason when nothing matches (ImplicitConversionErrorHandling).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Py.GIL()acquires the GIL throughPyGILState_Ensure/PyGILState_Release. For a .NET thread, the outermost scope creates a freshPyThreadStateand deletes it on dispose. Native extensions built with pybind11 — matplotlib >= 3.10_path/ft2font, several scipy submodules, PyTorch — cache the firstPyThreadState*they see for each OS thread in pybind11's internals TLS and never invalidate it (pybind/pybind11#2888). Once aPy.GIL()scope deletes the thread state, that cached pointer dangles, and the next pybind11gil_scoped_acquireon the same thread callsPyEval_RestoreThreadon freed memory:0xC0000005access violation (or0xc0000374heap corruption, depending on heap reuse).This is the root cause of QuantConnect/Lean#9203 (
ReportChartTests.ExposureReportWorksForEverySecurityTypecrashing the test host): theChartReportElementconstructor'sPy.GIL()scope imports matplotlib (pybind11 caches that scope's thread state), the scope's dispose frees it, and the first chart render (matshow→_path.update_path_extents/ ft2font text rendering) restores the dangling pointer. Verified from crash dumps: fault atPyEval_RestoreThread+0x7Areadingtstate->interp->runtimewithinterp == NULL, and the restored pointer matched a thread state created and deleted in an earlier GIL scope whose memory had been reused for string data.Fix
Pin each thread's
PyThreadStateonce created:GILStateissues one extra, never-releasedPyGILState_Ensureper thread per runtime run, keeping the gilstate counter >= 1 soPyGILState_Releasenever deletes the thread state. This matches how CPython itself treats thread states it binds via_PyGILState_NoteThreadState(counter starts at 1, never auto-deleted). The pin is keyed onRuntime.GetRun()so engine Initialize/Shutdown cycles re-pin correctly, andPy_Finalizereclaims all surviving thread states at shutdown.Trade-offs, measured:
Ensureruns while the GIL is already held, so it is a bare counter increment — no locking, no change to when the GIL is taken or released. Microbenchmark (200kPy.GIL()scopes on a worker thread): 0.212 -> 0.209 us/scope, i.e. within noise.PyThreadState(~1 KB) is intentionally kept per .NET thread that ever used Python, until engine shutdown — the same lifetime CPython gives Python-created threads.Regression test
TestGILState.ThreadStateIsPreservedBetweenGILScopes:threading.localvalues live in the thread state dictionary, so a value stored in onePy.GIL()scope survives into a second scope on the same thread only if the thread state was not deleted — fails on master, passes with the fix (no pybind11 dependency needed).Repro validation on Lean master (Windows, Python 3.11.9, matplotlib 3.10.1): the single-test crash reproduced 3/3 before the fix; with it, 5/5 repro runs, the full
ReportChartTestsfixture (66 tests) andPandasConverterTests(1216 tests) all pass.Fixes QuantConnect/Lean#9203
🤖 Generated with Claude Code