[FLINK-40135][python] Prevent Beam state access after runner shutdown

[bowenli86]

What is the purpose of the change

This change prevents Beam state requests from accessing Flink state after Python runner shutdown has completed. During task cancellation, Beam state callbacks can otherwise overlap keyed state backend disposal and access closed RocksDB native resources.

Brief change log

• Add a lifecycle gate to BeamStateRequestHandler.
• Wait for in-flight state requests during close and reject requests after close.
• Close the state handler after Beam request production stops and before the Python runner returns.
• Add deterministic tests for close draining, post-close rejection, and runner shutdown ordering.

Verifying this change

This change added tests and can be verified as follows:

• ./mvnw -q -pl flink-python -am -Dflink.markBundledAsOptional=false -Dtest=BeamStateRequestHandlerTest,BeamPythonFunctionRunnerTest -Dsurefire.failIfNoSpecifiedTests=false -Dfast -Pskip-webui-build -T1 test
• ./mvnw -f flink-python/pom.xml spotless:check
• ./mvnw -f flink-python/pom.xml checkstyle:check

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): yes
• Anything that affects deployment or recovery: yes
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes (Codex GPT-5)

Generated-by: Codex GPT-5

[flinkbot]

CI report:

• ab4b2c3 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build