feat(supervisor-middleware): add network egress middleware

[pimlock]

Summary

Implements the first usable RFC 0009 supervisor middleware slice: proto-backed, host-selected HTTP egress middleware for HttpRequest/pre_credentials, with both in-process built-ins and statically registered operator-run gRPC services.

The implementation covers RFC 0009 Phase 1 and adds basic external-service support from Phase 2. It establishes the contract, policy plumbing, ordered chain execution, built-in secret redaction, static gateway registration, relay integration, validation before policy persistence, body limits, audit events, and user-facing configuration and operations documentation.

Tip

See example middleware implementation: #2169
See high-level overview with interactive walk-through: https://gistpreview.github.io/?1f07f93cab1dab587d114fa6eec59c11/pr-2027-middleware-overview.html#request-lab

Related Issue

Closes #2010
Part of #1733
Design/RFC: #1738

Changes

• Add the supervisor middleware contract and runtime, including the built-in openshell/secrets redactor and statically registered operator-run gRPC services.
• Add ordered, host-selector-based network_middlewares policy configuration and validation, independent of the network policy rule that admits a request.
• Enforce middleware on inspected HTTP requests before credential injection, with transformations, safe header additions, body limits, and explicit fail-open/fail-closed behavior.
• Deliver effective service configuration to supervisors with resilient registry reload and last-known-good behavior.
• Add OCSF observability plus gateway, policy, operations, architecture, and extensibility documentation.

Testing

• mise run pre-commit passes
• Unit and integration-style tests added or updated
• E2E coverage considered
  • No separate gateway E2E was added. The request relay, policy validation, registry, and remote gRPC paths are covered by automated unit and integration-style tests.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated
• Published docs updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-2027.docs.buildwithfern.com/openshell

[pimlock]

/ok to test c4b0dcf

[pimlock]

gator-agent

Re-check After Author Update

Thanks @pimlock. I re-evaluated head a12f91bdfea84dc06b7098db7b0821bcafce22a9 after the merge-main/conflict-resolution update.

What I checked: the current PR diff against main, with focus on the forward-proxy, CONNECT/REST relay, middleware buffering, policy admission, server config delivery, audit, tests, and Fern documentation paths. The previous merge-conflict blocker is resolved (mergeable: true). The prior findings around connection-nominated headers, WebSocket canonicalization, Expect: 100-continue, final success auditing, body/envelope bounds, finding cardinality, docs, and the explicit follow-ups for unauthenticated middleware transport and Helm rendering were not repeated.

Review findings:

• P1 - Malformed HTTP request lines can pass policy and middleware inspection, then reach upstream. crates/openshell-supervisor-network/src/l7/rest.rs:311, crates/openshell-supervisor-network/src/proxy.rs:680, and crates/openshell-supervisor-network/src/proxy.rs:3274: header-field validation does not validate the request line, the forward proxy only takes the first two whitespace tokens, and rewrite can preserve malformed version text upstream. A request such as GET http://api.example/p HTTP/1.1 extra can be admitted and rewritten as GET /p HTTP/1.1 extra, creating an HTTP parser differential after middleware inspection (CWE-444). Add shared raw request-line validation before CONNECT/forward dispatch and buffered middleware construction: require exactly METHOD SP target SP HTTP/1.0|HTTP/1.1, validate the method as an HTTP token, reject controls in the target, and add matching/nonmatching-host regressions proving malformed request lines neither invoke middleware nor reach upstream.
• P2 - Unsupported forward-proxy schemes are forwarded as HTTP while middleware sees the original scheme. crates/openshell-supervisor-network/src/proxy.rs:3165, :3216, :3528, and :4571: parse_proxy_uri accepts any URI scheme, defaults non-https schemes to port 80, rejects only https, then passes the original scheme into middleware. ftp://api.example/p can therefore be forwarded as plain HTTP while external middleware receives scheme = "ftp". Reject every forward-proxy scheme except http before policy or middleware evaluation, and add an unsupported-scheme regression.

Docs: Fern docs and docs/index.yml navigation are present for the direct UX surface.

Checks: Branch Checks and required test:e2e are currently pending for this head. No local tests were run as part of this code-only review.

Next state: gator:in-review pending an author update for these findings.

[pimlock]

Thanks for the re-review. I addressed both current findings in be9e5b7f.

Finding	Disposition
Malformed HTTP request lines can pass inspection	Resolved. The shared raw HTTP/1 validator now requires exactly `METHOD SP target SP HTTP/1.0
Unsupported forward-proxy schemes are forwarded as HTTP	Resolved. Absolute-form forward requests now accept only http. HTTPS retains the existing instruction to use CONNECT, and every other scheme returns 400 unsupported_proxy_scheme before policy lookup, middleware, DNS, or upstream connection.

Proxy hardening trace from supervisor middleware work

Supervisor middleware added an inspect-and-transform boundary to the HTTP proxy. Review of that boundary also exposed pre-existing proxy behavior. The invariant behind the changes below is: policy and middleware must evaluate the same effective request that OpenShell sends upstream.

Boundary	Hardening completed	Why it changed
Raw ingress	Validate UTF-8, CRLF framing, the complete request line, method token, target controls, HTTP version, header names and values, obsolete folding, whitespace before colons, and Connection option tokens before host selection or dispatch. Reject ambiguous Content-Length and Transfer-Encoding framing.	Prevent parser differentials before matching a policy or middleware chain. This now applies to matching and nonmatching destinations.
Request routing	Keep canonical request targets synchronized between L7 evaluation and forwarded bytes. Allow only HTTP absolute-form forwarding; require CONNECT for HTTPS and reject other schemes.	Prevent middleware, policy, and upstream from assigning different semantics to the target or scheme.
Header projection and forwarding	Preserve repeated headers in wire order for middleware. Omit credentials and hop-by-hop fields from external inspection, carry dynamically nominated Connection names as protected metadata, strip nominated fields before forwarding, and rebuild only the canonical Connection: Upgrade plus Upgrade: websocket pair for validated WebSocket requests.	Ensure a field cannot be hidden from inspection and then delivered upstream, while avoiding credential disclosure to middleware.
Body framing and buffering	Reject unsupported transfer-coding sequences and CL/TE combinations; bound decoded and wire bytes, chunk counts, extensions, and trailers; handle Expect: 100-continue locally whenever OpenShell buffers; remove stale framing and emit a normalized Content-Length request. Newly affected REST middleware and credential-rewrite paths reject non-empty trailers.	Ensure buffering cannot silently reinterpret framing or forward headers that describe a body OpenShell already consumed.
Middleware transformations	Feed each stage the output of the previous stage, apply capacity per stage, and reclassify plus re-evaluate L7 policy after every body transformation. Validation errors and unsafe mutations follow the configured fail-open or fail-closed behavior, and denial stops upstream forwarding.	A transformed request must remain policy-compliant before the next stage or origin sees it.
Uninspectable traffic	A matching fail-closed chain denies traffic that cannot be inspected, including unsupported h2c or tunnel protocols, TLS without termination, defense-in-depth tls: skip cases, and unimplemented SQL relay. An all-fail-open chain may bypass with an OCSF finding.	Attaching a fail-closed middleware must make inspection mandatory rather than advisory.
Policy lifetime	Policy and middleware registry updates commit as one runtime generation, and generation guards are checked through final relay.	Prevent a request from being evaluated with one policy or service set and forwarded after another becomes active.
Audit	Final forward-success OCSF and activity records are emitted only after middleware, token grant, credential rewriting, generation checks, and relay succeed.	Denied or failed requests must not produce contradictory success records.

Tracked or accepted follow-ups:

• #2251 tracks pre-existing unevaluated pipelined request bytes.
• #2284 tracks unifying buffered HTTP framing and making trailer handling explicit across REST, JSON-RPC/MCP, GraphQL, and inference.
• #2286 tracks Helm rendering for external middleware registrations.
• Unauthenticated external middleware transport, including configured plaintext HTTP, is an acknowledged limitation accepted for this version. Authenticated service transport will be handled separately.

Validation completed:

• mise run pre-commit passes, including formatting, strict Clippy, workspace checks, and the full Rust test suite.
• All 949 openshell-supervisor-network unit tests pass.
• Focused regressions cover malformed request lines for CONNECT and forward requests, matching and nonmatching hosts, invalid method/target/version forms, and unsupported forward schemes.

[pimlock]

gator-agent

Blocked

Thanks @pimlock. I saw your latest update on head be9e5b7f0246f47d395f16fbeff71fa4063d9baa addressing the malformed request-line and unsupported forward-proxy scheme findings, but gator cannot re-review this head yet because GitHub still reports merge conflicts against main.

Head SHA: be9e5b7f0246f47d395f16fbeff71fa4063d9baa

GitHub currently reports mergeable: false and mergeable_state: dirty. Required checks are still running for this head, but the merge-conflict blocker takes precedence.

Next action: @pimlock needs to rebase or merge main into the PR branch and resolve the conflicts, then push the updated head for re-check.

[pimlock]

Follow-up compatibility refinement in 7339c239:

• Restored the policy.local special-case before the generic forward-proxy scheme check. Non-HTTP and non-port-80 forms therefore retain the existing invalid_policy_local_scheme response and guidance to use http://policy.local.
• Kept the generic non-http rejection for all other absolute-form forward requests, including ws:// and wss://.
• Existing WebSocket support is unchanged: plaintext WebSocket upgrades continue to use an http:// request with Upgrade: websocket, while encrypted proxy tunnels use CONNECT.

Added policy_local_preserves_specific_non_http_scheme_error to cover the preserved response for HTTPS and another unsupported scheme.

Validation:

• cargo test -p openshell-supervisor-network --lib policy_local_preserves_specific_non_http_scheme_error
• mise run pre-commit

[pimlock]

gator-agent

Blocked

Gator is blocked by merge conflicts against main.

Head SHA: 7339c239befdfacdd6e40391aaf311ee0b359915

GitHub currently reports mergeable: false and mergeable_state: dirty. Required checks are still pending for this head, but the merge-conflict blocker takes precedence, so gator cannot run a fresh code review or advance the PR yet.

Next action: @pimlock needs to rebase or merge main into the PR branch and resolve the conflicts, then push the updated head for re-check.

[pimlock]

Merged current main into the PR branch in 83ea6d21.

Conflict resolution:

• Resolved the SDK request-builder conflict by adopting main's shared HashMap import.
• Removed the duplicate annotations initializer produced by the automatic merge in client_mock.rs while retaining main's assertion that create requests send empty annotations.

Validation:

• cargo test -p openshell-sdk: 29 tests passed; 2 doctests ignored.
• cargo test -p openshell-supervisor-process --lib: 164 tests passed; 2 privileged tests ignored.
• The full pre-commit gate was attempted twice. Formatting, linting, SDK, network, Python, Helm, and other completed suites passed, but the workspace test task stalled during supervisor-process tests under concurrent execution; the same process suite passed immediately in isolation.

[pimlock]

gator-agent

Re-check After Author Update

Thanks @pimlock. I re-evaluated head 83ea6d21d5153361427983711418ac626957ad0b after your July 15 update merging current main and resolving the previous conflict blocker.

What I checked: the current PR diff against main, with focus on the merge resolution, forward-proxy authority parsing and rewriting, middleware-visible request construction, policy/middleware host selection, audit timing, registry/config delivery, and Fern documentation. The previous merge-conflict blocker is resolved (mergeable: true). The prior malformed request-line and unsupported forward-proxy scheme findings look addressed, and the policy.local compatibility refinement is preserved.

Review findings:

• P2 - Forward proxy policy/middleware evaluate the absolute URI host, but upstream still receives the caller-controlled Host header. crates/openshell-supervisor-network/src/proxy.rs:3700, crates/openshell-supervisor-network/src/proxy.rs:4567, crates/openshell-supervisor-network/src/proxy.rs:3314, and crates/openshell-supervisor-network/src/l7/middleware.rs:325: handle_forward_proxy selects policy and middleware from the absolute-form URI host, then passes the original buffered request into middleware and rewrite_forward_request copies non-hop-by-hop headers through unchanged. Since Host is filtered from middleware input and protected from mutation, a sandbox can send GET http://allowed.example/path with Host: disallowed.example; OpenShell evaluates allowed.example, but the origin receives Host: disallowed.example. On shared virtual hosts or reverse proxies this can bypass host-scoped policy and send injected credentials to a different HTTP authority (CWE-444). Canonicalize or reject Host before middleware and forwarding; prefer replacing it with the absolute URI authority. Add forward-proxy regressions for mismatched Host, repeated Host, and middleware-selected requests proving the upstream bytes contain only the canonical target authority.

Docs: Fern docs and docs/index.yml navigation are present for the direct UX surface.

Checks: Branch Checks, DCO, and Helm Lint are green. Required test:e2e is still pending for this head. No local tests were run as part of this code-only review.

Next state: gator:in-review pending an author update for this finding.

[pimlock]

Addressed the remaining P2 authority finding in e49d20f1.

Finding	Disposition
Forward proxy policy/middleware evaluate the absolute URI host while upstream receives a caller-controlled Host	Resolved. Absolute-form requests now derive a canonical authority from the request-target, replace every received Host field before L7/middleware/credential processing, and defensively generate the only outbound Host field from that same authority during the final origin-form rewrite.

This follows RFC 9112 section 3.2.2:

When a proxy receives a request with an absolute-form of request-target, the proxy MUST ignore the received Host header field (if any) and instead replace it with the host information of the request-target. A proxy that forwards such a request MUST generate a new Host field value based on the received request-target rather than forward the received Host field value.

Implementation details:

• Canonical authority formatting handles default/non-default ports, lower-cased host names, and bracketed IPv6 literals.
• Missing, mismatched, repeated, and mixed-case Host fields all result in exactly one canonical field.
• Host remains protected and hidden from middleware; middleware continues to receive the canonical destination through its host and port properties.
• The final rewrite independently strips all Host fields and emits the canonical value, preventing later request rebuilding from reintroducing a conflicting authority.

Regression coverage includes mismatch, repeated fields, missing fields, non-default ports, IPv6, body read-ahead preservation, and an active middleware-selected request whose captured outbound bytes contain only the canonical authority.

Validation:

• cargo test -p openshell-supervisor-network --lib: 954 passed.
• Strict crate Clippy with warnings denied passed.
• mise run pre-commit passed in full.

No follow-up is needed to close this finding. A separate follow-up would still be useful for broader structured absolute-URI parsing and auditing CONNECT/TLS-inspected authority consistency; those are beyond this specific absolute-form Host replacement requirement.

[pimlock]

gator-agent

Re-check After Author Update

Thanks @pimlock. I re-evaluated head e49d20f1f089b85251b4c66d425626eb0802eb5e after your July 15 update addressing the forward-proxy authority finding.

What I checked: the current forward-proxy authority canonicalization, L7/middleware-visible request construction, final origin-form rewrite, credential-processing path, and regression coverage for missing, repeated, mismatched, non-default-port, IPv6, and middleware-selected Host handling.

Disposition: resolved.

Review findings:

• No blocking findings remain in this code-only pass. The previous issue where policy and middleware evaluated the absolute URI authority while upstream could receive a caller-controlled Host is resolved by canonicalizing Host from the request target before L7/middleware/credential processing and independently emitting the canonical Host during the final upstream rewrite.

Docs: Fern docs and docs/index.yml navigation are present for the direct UX surface.

Checks: Branch Checks, DCO, and Helm Lint are green. Required test:e2e is still pending for this head. No local tests were run as part of this code-only review.

Next state: gator:watch-pipeline while E2E finishes.

[pimlock]

Review follow-up: middleware attachment identity

Addressed the middleware identity and binding-model follow-up in e79cfaa9:

• Policies now attach built-in middleware by its platform name and external middleware by the operator-owned registration name.
• The same external service can be registered and attached multiple times under different operator-controlled names. The service-reported manifest name remains diagnostic metadata and does not need to match the registration name.
• Attaching a middleware attaches the complete manifest. Runtime selection uses the unique (operation, phase) capability; manifests cannot declare two bindings for the same pair.
• Removed service-owned binding IDs from MiddlewareBinding, ValidateConfigRequest, and HttpRequestEvaluation. V1 has only HTTP_REQUEST/PRE_CREDENTIALS, so an ID did not disambiguate or authorize anything.
• External denial and finding diagnostics now derive their stable namespace from the validated operator registration name rather than service-provided text.
• Updated the built-in implementation, policy admission, registry delivery, tests, protobuf documentation, architecture documentation, and user-facing docs to use the same model.
• Updated #2282 so future binding authorization and per-binding overrides select platform-owned operation/phase pairs instead of service-owned IDs.

Validation:

• mise run pre-commit
• Full workspace Rust tests and server integration tests
• Workspace Clippy with warnings denied
• Markdown, Helm, Python, protobuf, formatting, and license checks

[pimlock]

gator-agent

Re-check After Author Update

Thanks @pimlock. I re-evaluated head e79cfaa99e366ad9848c270ec3174d1e41acff59 after your July 16 update moving supervisor middleware attachment identity to built-in names and operator-owned registration names.

What I checked: the current-head delta from e49d20f1f089b85251b4c66d425626eb0802eb5e through e79cfaa99e366ad9848c270ec3174d1e41acff59, including registry construction, policy admission, required-service delivery to supervisors, runtime chain selection, normalized external diagnostics, protobuf comments, architecture docs, and Fern docs. I also used the independent principal-engineer reviewer result for this head.

Disposition: resolved.

Review findings:

• No blocking findings remain in this code-only pass. The attachment model is now consistently keyed by OpenShell built-in names or operator-owned registration names, external diagnostics derive from the validated registration name, and the user-facing docs describe the new model.
• Non-blocking note: the reviewer noticed that proto/supervisor_middleware.proto reuses field numbers removed from the interim PR-head schema. This is not blocking for the unreleased supervisor middleware contract, but if maintainers need compatibility with middleware services generated from earlier PR heads, reserve the removed field numbers before merge.

Docs: Fern docs and docs/index.yml navigation are present for the direct UX surface.

Checks: Branch Checks, DCO, and Helm Lint are green. Required test:e2e is still pending for this head.

Next state: gator:watch-pipeline while E2E finishes.