test(e2e): event-driven harness with per-controller parking and deterministic cancel test

[sbalabanov]

Stacks on #348 (base branch fix-merge-topic-collision).

What this is

Reworks the SubmitQueue e2e suite into a push-based event plane: test-owned consumer groups subscribe to the log topic and runway's check/signal topics over the published mysql-queue port. Delivery state in the MySQL queue is keyed per consumer group, so the observer receives a copy of every message without stealing from the services and with zero service changes. Tests await pushed events (status transitions, runway's answers on the wire) instead of polling.

There are no hard-coded timeouts: every wait is bounded by a suite context whose deadline derives from t.Deadline() (the Bazel test timeout) minus a teardown margin. The remaining polls (Status RPC and request_log, both eventually consistent behind the gateway log consumer) run at a fixed cadence under that context.

NEW: per-controller parking with StageHold

testutil.StageHold parks any single queue consumer controller by planting a phantom partition lease — a row with a far-future lease_renewed_at — in the MySQL queue's partition-leases table for (consumer_group, topic, partition_key). The queue's expiry-based steal semantics (verified in partition_lease_store.go) mean no consumer in that group can acquire the partition until the row is removed, so the stage is starved deterministically with zero service changes. The plant is a plain INSERT that fails loudly if the partition is already leased, so holds must be created before the stage first consumes (pre-hold contract). Granularity is per controller x partition. Release() deletes the row (idempotent after success, retryable on failure) and consumption resumes on the service's next discovery tick. Held messages are never delivered, so there is no retry-budget or DLQ interaction.

Stage catalog

test/e2e/submitqueue/stage_test.go enumerates every queue consumer controller across the three services — all 27 consumers: gateway (1), orchestrator (12 primary + 12 DLQ), runway (2) — with a holdStage() suite helper, so any controller x partition can be starved from a test.

Lease-semantics contract test

test/integration/extension/messagequeue/mysql/stagehold_test.go pins the guarantees StageHold relies on: a held partition starves the group's consumer while other partitions keep flowing, releasing the hold resumes delivery, and Release is idempotent.

Deterministic cancel test (reimplemented)

TestCancel_BeforeBatch_ReachesCancelled now parks the pipeline with a StageHold on runway's merge-conflict-check consumer (partition = queue name) instead of stopping/starting the runway container: park (confirmed by a pushed event), cancel, await the pushed "cancelled" event, assert the terminal store state and that no batch ever enrolled the request, then release the hold and assert runway's late signal does not resurrect it. The connectRunway re-dial helper is gone.

No on-demand service lifecycle

ComposeStack.StartService is removed — services start once in SetupSuite and stop once in TearDownSuite; there is no start/stop lever mid-suite anymore. StopService is retained for the graceful-shutdown verification in teardown.

Also in the suite: runway is first-class — gRPC ping test, graceful-shutdown exit-code check, and happy-path assertions that runway answered both the merge-conflict check and the merge with SUCCEEDED (the assertion that exposed the merge topic collision fixed in #348).

🤖 Generated with Claude Code

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[behinddwalls]

⚠️ Automatic stack rebase failed

This PR could not be automatically rebased after its base PR was merged. The rebase hit conflicts that need manual resolution.

To fix manually:

git fetch origin
git checkout e2e-event-harness
git rebase --onto origin/main 087124c1ac6613d95786c55c7b12dad0e3509203 e2e-event-harness
# resolve conflicts, then:
git push --force-with-lease

Then update this PR's base branch:

gh pr edit 349 --base main