Skip to content

Clear pending Python error when a Try-style conversion fails#138

Merged
jhonabreul merged 4 commits into
QuantConnect:masterfrom
jhonabreul:bug-tryas-leaves-pending-python-error
Jul 14, 2026
Merged

Clear pending Python error when a Try-style conversion fails#138
jhonabreul merged 4 commits into
QuantConnect:masterfrom
jhonabreul:bug-tryas-leaves-pending-python-error

Conversation

@jhonabreul

@jhonabreul jhonabreul commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

What does this implement/fix?

PyObject.TryAsManagedObject (and therefore TryAs<T>/TrySafeAs<T>) converted with setError: true and returned false on failure, leaving the Python error indicator set with no one left to surface it. The stale error was then thrown by the next unrelated Python call on that thread. This was masked while the outermost Py.GIL scope deleted the thread state on dispose; since #137 pins thread states, the leaked error persists. It is the root cause of the PythonUtilTests.BothInheritedAndNonInheritedClassesWork(True) CI failure in QuantConnect/Lean#9614: an earlier test's failed TryAs<int> on a Python function left a TypeError pending that the next Python-touching test then threw.

The fix: TryAsManagedObject converts with setError: false and clears anything a conversion sub-path may have left pending before returning false. AsManagedObject keeps setError: true — fetching the error for the InvalidCastException cause consumes the indicator. Also bumps the version to 2.0.63.

Two follow-up commits close the same class of leak elsewhere:

  • Converter collection/label paths: ToList handles a failed PyObject_GetIter like ToArray does (also fixing a leaked iterator reference); MakeList fails the conversion when the iterator raises mid-iteration instead of returning a silently truncated list, and clears swallowed errors when not setting errors; the type_error/overflow labels clear pending errors when not setting errors, mirroring convert_error.
  • MethodBinder: overload probes convert with setError: false but can still raise (see below). When the raising candidate was probed after the candidate that matched, the pending error survived into the successful call, which then failed with SystemError: ... returned a result with an error set. Bind now clears any pending probe error once an overload has matched; the no-match path is untouched.

Why some conversion paths deliberately raise even with setError: false, and why they stay that way

The two implicit-conversion catch blocks in Converter.ToManagedValue call Exceptions.RaiseTypeError("Failed to implicitly convert {source} to {target}") unconditionally, ignoring setError. This is intentional and this PR keeps it:

  • MethodBinder probes every overload with setError: false. When no overload matches, MethodBinder.Invoke checks the error indicator and, if a probe left an error, surfaces that instead of the generic "No method matches given arguments" message. A throwing implicit operator is the canonical case: without the unconditional raise, the actionable cause ("Failed to implicitly convert X to Y") would be silently discarded during probing and the user would only see the generic no-match error. This contract is covered by ImplicitConversionErrorHandling.
  • Gating those raises on setError would therefore degrade diagnostics: probing always runs with setError: false, so the specific cause could never reach the user.

The consequence is that "converted with setError: false" does not guarantee "no error pending", so each API boundary that swallows a failed conversion is responsible for clearing: TryAsManagedObject clears before returning false, MakeList clears its element-conversion failures, and MethodBinder.Bind now clears once an overload has matched (keeping the error only on the no-match path, where it is the diagnostic).

Does this close any currently open issues?

Fixes the PythonUtilTests.BothInheritedAndNonInheritedClassesWork CI failure in QuantConnect/Lean#9614.

Any other comments?

Regression tests added:

  • FailedTryAsDoesNotLeavePythonErrorSet — a failed TryAs<int> leaves no pending error; subsequent PyModule.FromString calls work.
  • FailedAsManagedObjectRaisesWithConversionErrorAsCauseAsManagedObject still raises InvalidCastException with the Python error as its cause.
  • RaisingIteratorFailsArrayConversion / OverflowConversionOnlyLeavesErrorWhenSettingErrors — these conversions fail leaving an error pending only when setError is true.
  • RejectedOverloadProbeErrorDoesNotPoisonSuccessfulBind — a rejected overload's raising probe (throwing implicit operator) does not poison a successful bind of a sibling overload.

Verified end-to-end against Lean: 2.0.62 reproduces the exact CI failure when running ExtensionsTests + PythonUtilTests on a single NUnit worker; with this build all 352 tests pass. Full embedding test suite: 965 passed, 8 skipped. A full Lean test-suite sweep with a per-test PyErr-leak detector found the TryAs path to be the only leak source across 36,316 tests.

Checklist

Check all those that are applicable and complete.

  • Make sure to include one or more tests for your change
  • If an enhancement PR, please create docs and at best an example
  • Ensure you have signed the .NET Foundation CLA
  • Add yourself to AUTHORS
  • Updated the CHANGELOG

PyObject.TryAsManagedObject converted with setError: true and returned
false on failure, leaving the Python error indicator set on the thread
with no one left to surface it. Callers of TryAs/TryAsManagedObject
only observe the boolean, so the stale error survived on the thread
state and was thrown by the next unrelated Python call that checks the
error indicator.

This was previously masked because the outermost Py.GIL scope deleted
the thread state (and its pending error) on dispose; since thread
states are pinned for the lifetime of the run (QuantConnect#137), the leaked error
persists and poisons subsequent calls on the same thread - e.g. Lean's
PythonUtilTests failing with "int() argument must be a string, a
bytes-like object or a real number, not 'function'" leaked by an
earlier test that exercises a failed TryAs<int> on a function.

Convert with setError: false in TryAsManagedObject and clear any error
a conversion sub-path may have left pending before returning false.
AsManagedObject keeps converting with setError: true so the fetched
error still becomes the InvalidCastException cause, which also
consumes the indicator.
Bump package <Version>, AssemblyVersion/AssemblyFileVersion and the
perf-test baseline reference to 2.0.63.
Several Converter failure paths could leave a Python error pending even
when called with setError: false, relying on the caller to clean up:

- ToList did not handle a failed PyObject_GetIter: the raised TypeError
  survived the call (and the iterator reference leaked). Handle it like
  ToArray does, disposing the reference and clearing when not setting
  errors.
- MakeList treated a null PyIter_Next result as normal exhaustion, but
  null also means the iterator raised: the conversion now fails instead
  of returning a silently truncated list, clearing the error when not
  setting errors and leaving it pending for the caller when setting
  errors. The swallowed CLR-exception catch and the element-conversion
  failure path also clear when not setting errors, since some element
  conversion paths (failed implicit operators, deleted types) raise
  regardless of setError to enrich MethodBinder's no-match message.
- The type_error and overflow labels now clear pending errors when not
  setting errors, mirroring what convert_error already did, so a
  probing sub-path that raised before jumping cannot leak.

The unconditional raises in the implicit-conversion catches are kept:
MethodBinder.Invoke deliberately surfaces a pending error as the
binding failure reason instead of its generic no-match message, and
Converter.TryAsManagedObject clears at the API boundary.
MethodBinder probes each overload candidate with setError: false, but a
probe can still raise: the implicit-conversion catches in
Converter.ToManagedValue raise unconditionally so that Invoke can
surface the specific cause (e.g. a throwing implicit operator) as the
bind-failure reason instead of the generic no-match message.

When the raising candidate was probed after the candidate that
ultimately matched, nothing cleared the indicator before the method ran
and the otherwise successful call failed with "SystemError: ...
returned a result with an error set". Errors raised by earlier
candidates were incidentally wiped by the per-candidate
PyObject_Type/Exceptions.Clear type check, which is why the leak needs
this specific overload ordering: an exact-class-match overload
(precedence 40) binds first, then a string overload (precedence 50) is
probed and raises.

Clear any pending probe error in Bind once an overload has matched. The
bind-failure path is untouched, so a pending error is still surfaced as
the failure reason when nothing matches (ImplicitConversionErrorHandling).
@jhonabreul jhonabreul merged commit 9bcc291 into QuantConnect:master Jul 14, 2026
15 of 16 checks passed
@jhonabreul jhonabreul deleted the bug-tryas-leaves-pending-python-error branch July 14, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants