feat(stovepipe): add DLQ reconciliation for the process stage

[gsturges]

Summary

A request whose process-stage message exhausts retries would previously sit in a non-terminal state forever — indistinguishable from "not yet validated" to callers gating deployments on greenness. Per the fail-closed policy in doc/rfc/stovepipe/workflow.md, such requests must be driven to a conservative not-green terminal state instead.

• Entity: add RequestStateFailed terminal state and IsRequestStateTerminal helper.
• New stovepipe/controller/dlq package: a consumer.Controller that drains the process_dlq topic and CAS-transitions the affected request to failed. If the request had already been admitted (processing), the queue's in_flight_count slot is released first, per the integrity rule in doc/rfc/stovepipe/steps/process.md.
• Server wiring: second consumer using errs.AlwaysRetryableProcessor (a reconciliation failure retries in place — the DLQ subscription is a final destination with DLQ.Enabled = false, so nothing can cascade to a _dlq_dlq); process_dlq registered in the topic registry with raised retry attempts; DLQ consumer starts first and stops last.

Reconciliation is idempotent: already-terminal requests are skipped, and optimistic locking lets a late legitimate pipeline transition win. The design mirrors SubmitQueue's proven submitqueue/orchestrator/controller/dlq pattern, simplified for stovepipe's single payload shape.

Only process is a queue-consuming step today (ingest is synchronous RPC); future stages (build, buildsignal, record) can reuse the same pattern.

Test plan

• New table-driven unit tests in stovepipe/controller/dlq/dlq_test.go: accepted→failed, processing→failed with slot release, already-terminal no-ops, not-found no-op, CAS version-mismatch paths, malformed/empty payloads
• go build ./... and go test ./stovepipe/... ./service/stovepipe/... pass
• make gazelle, make fmt, make lint-license clean

🤖 Generated with Claude Code

[gsturges]

Rebased onto main. Besides the review feedback already discussed (the DLQSubscriptionConfig helper and the corrected failRequest / decode-error comments), the rebase required one substantive adaptation worth calling out:

The reconciler now forces recorded_not_green instead of introducing a separate failed state. While this PR was in review, main landed the terminal Request states (#362): recorded_green / recorded_not_green with an IsTerminal() method, and the RequestStateRecordedNotGreen doc comment (plus workflow.md's fail-closed section) already designates it as the conservative verdict the DLQ reconciler writes. Keeping this PR's RequestStateFailed would have meant two overlapping terminal-not-green states, so this branch now:

• no longer touches stovepipe/entity/request.go (no RequestStateFailed, no IsRequestStateTerminal free function),
• transitions reconciled requests to entity.RequestStateRecordedNotGreen and checks request.State.IsTerminal().

Reconciliation behavior is otherwise unchanged (idempotent terminal-skip, slot release before the terminal CAS, always-retryable pairing). newTopicRegistry keeps main's publish-only build topic alongside the new process_dlq subscription.

🤖 Generated with Claude Code