feat(workspace): add workspace resource model with scoping, membershi…

[derekwaynecarr]

Summary

Implements Phase 1 of RFC 0011 — the workspace and membership model that provides hard isolation boundaries for multi-player OpenShell deployments.

• Workspace resource: CRUD RPCs (CreateWorkspace, GetWorkspace, ListWorkspaces, DeleteWorkspace) with a default workspace created on gateway startup for backwards compatibility
• Workspace-scoped resources: workspace field on ObjectMeta — sandboxes, providers, service endpoints, SSH sessions, policies, settings, provider refresh state, and inference routes all inherit workspace
  from context
• Membership model: AddWorkspaceMember, RemoveWorkspaceMember, ListWorkspaceMembers RPCs with (workspace, principal_subject) → role records; membership records cleaned up on workspace deletion
• Persistence: Name uniqueness shifts from (object_type, name) to (object_type, workspace, name) via migration 006; existing resources backfilled to default workspace; cross-workspace list_by_type store
  method for infrastructure operations (reconciler, resume, provider refresh)
• Provider profiles: Two-tier scoping with independent scope listing — workspace custom + built-in, or platform custom + built-in (no merged cross-scope view)
• Service routing: Endpoint hostnames include workspace prefix ({workspace}--{sandbox}--{service}.{domain}); default workspace preserves current format
• Inference routes: Rename Cluster* RPCs and messages to Route* (SetInferenceRoute, GetInferenceRoute, InferenceRouteConfig); workspace-scope route storage and lookup; derive workspace from sandbox
  principal for bundle resolution; thread --workspace through CLI inference commands
• ObjectWorkspace trait: requires_workspace() method with debug_assert! validation in store write helpers to catch empty-workspace bugs in debug builds
• CLI: --workspace and --all-workspaces flags on all resource commands; WORKSPACE column as first column in list outputs when --all-workspaces (matching kubectl --all-namespaces convention); workspace
  CRUD and membership subcommands; inference help text updated from "gateway-level" to "workspace-level"
• TUI: Workspace state with [w] key cycling, WORKSPACE column in sandbox/provider tables, workspace indicator in title bar, workspace-scoped gRPC requests

Related Issue

#1977

Changes

• 50 files changed, +7188 / -886 lines
• New files: workspace.rs (gRPC handlers + tests), migration 006 (sqlite + postgres), workspace_lifecycle.rs (e2e test), RFC 0011
• Proto: Workspace, WorkspaceMember messages; workspace CRUD and membership RPCs; workspace field on ObjectMeta; all_workspaces on list requests; inference route rename (Cluster* → Route*) with
  workspace fields on set/get request/response messages

Testing

• 897 server unit tests passing (11 workspace-specific, 2 inference route workspace isolation)
• 24 TUI tests passing
• 7 CLI integration tests passing
• e2e workspace lifecycle test covering create/list/get/delete and cross-workspace isolation
• Inference route isolation tests: verify workspace-scoped route resolution (alpha/beta workspaces with different providers/models) and empty bundle for workspace with no route configured

Known remaining work

• Phase 2: Expanded role model (Platform Admin / Workspace Admin / User) and authorization enforcement — not started, sequential dependency on this PR.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[sjenning]

From CodeRabbit review:
Include SSH sessions in workspace deletion blockers.

ObjectWorkspace for SshSession marks sessions as workspace-scoped, but deletion does not check SshSession::object_type(). A workspace can therefore be removed while session records still reference it.

     for (object_type, label) in [
         (Sandbox::object_type(), "sandbox"),
         (Provider::object_type(), "provider"),
+        (SshSession::object_type(), "SSH session"),
         (ServiceEndpoint::object_type(), "service"),
         (InferenceRoute::object_type(), "inference route"),
     ] {

[russellb]

🤖 Generated with Claude Code (Opus 4.8). Reviewed code + RFC 0011 as source of truth. Findings labeled CONFIRMED (traced) vs PLAUSIBLE (needs author confirmation).

Principal review — actionable findings

Framing: Phase 1 ships the workspace data model + membership records but no per-workspace authorization — resolve_workspace (grpc/workspace.rs:55) checks only that the workspace exists, never that the caller is a member; membership records are written but never read for access decisions. Per the RFC this is intentionally deferred to Phase 2, and it doesn't regress today's two-tier RBAC. The ask: state plainly in the RFC/release notes that Phase 1 is not a tenant security boundary until Phase 2, and consider gating the multi-workspace surface behind a config flag until enforcement lands. The blockers below are independent of Phase 2 and will bite existing deployments on upgrade.

Blockers

B1 — Decoded objects don't hydrate workspace from the DB column → every pre-existing object diverges on upgrade. CONFIRMED (independently found by two audits + verified directly).
decode_record (persistence/mod.rs:146-153) hydrates resource_version from the row but not workspace. Migration 006 backfills the column to default, but the workspace is also serialized in the protobuf payload, which stays "" for pre-PR objects; CAS updates never rewrite the column, so the divergence is permanent. Downstream code scoping by object.object_workspace() then gets "" instead of default:

• compute/mod.rs:~1379 teardown deletes sandbox_settings/ssh_session at "" → rows backfilled to default are orphaned on every pre-existing sandbox delete.
• inference.rs:83 resolves the route at "" → inference route lookup fails for all pre-existing sandboxes.
• Same pattern in policy.rs, service.rs, provider_refresh.rs.

Fix: add a SetWorkspace trait and overwrite the message workspace from record.workspace in decode_record, mirroring resource_version.

B2 — TUI provider credential update targets the active workspace, not the selected row → cross-workspace credential overwrite. CONFIRMED.
spawn_update_provider (tui/src/lib.rs:1718) uses app.current_workspace, while name/type come from the selected row. Siblings spawn_get_provider (1681) and spawn_delete_provider (1767) correctly use selected_provider_workspace(). In the all-workspaces view, editing a credential on a beta row sends workspace:"default", merging the secret into default's same-named provider (silent cross-tenant corruption) or a confusing not_found. Fix: use selected_provider_workspace() (capture into the form at open time so it can't drift).

B3 — Migration doesn't backfill provider_profile; pre-existing custom profiles silently vanish. CONFIRMED omission.
provider_profile existed at merge base but is absent from the 006 backfill list. Reads resolve to default (provider.rs:~1287) while rows stay at '', so imported profiles disappear from list/get/update yet are still consumed by providers whose profile_workspace decodes to "". Decide semantics (backfill to default, or union the global "" namespace into reads) and add a test. Also: the backfill uses wrong type strings 'policy'/'draft_chunk' (actual: sandbox_policy/draft_policy_chunk) — dead no-ops; remove or correct.

Should-fix

S1 — all_workspaces reachable by role = "user", but the RFC intends Platform-Admin-only. CONFIRMED. list_sandboxes/list_providers are role = "user" (mod.rs:240,381) and fan out via list_all_messages* when all_workspaces=true. Gate to admin.

S2 — requires_workspace() enforced only in typed helpers, not at the store boundary. CONFIRMED. Raw put_if/put_scoped (persistence/mod.rs:235,270) skip it; the reconcile create path (compute/mod.rs:~1158) writes incoming.workspace from the driver with no non-empty guard. A driver reporting "" silently persists an un-scoped, tenant-invisible sandbox. Move the guard to the store boundary; emit an OCSF DetectionFinding on empty-workspace reconcile instead of persisting silently.

S3 — Sandbox watch-channel workspace seeded only on the initial poll success → permanent silent telemetry loss. CONFIRMED. In openshell-sandbox/src/lib.rs, workspace_tx.send fires only in the initial-poll Ok arm (~:2007); the steady-state loop never re-sends. If the first poll_settings errors (normal at startup), a non-default-workspace sandbox flushes with "" → server defaults to default → UUID scope check fails → denial/activity proposals dropped for the sandbox's lifetime. default-workspace sandboxes work by coincidence, masking this in tests. Send the learned workspace on every successful poll, or fail the flush loudly on empty workspace.

S4 — handle_get_sandbox_logs resolves the workspace then discards it (let _workspace = …, policy.rs:~2150); logs served by sandbox_id alone. CONFIRMED. Latent IDOR once Phase 2 lands (member of A + a B sandbox UUID → B's logs/prompts). Assert sandbox.object_workspace() == workspace after the by-id fetch, or switch to name-based lookup like its siblings. The wider by-UUID family (exec, forward_tcp, create_ssh_session, watch_sandbox) has no workspace on the request proto — consider adding it now so Phase 2 has a hook.

S5 — SDK list(workspace=None) silently means "all workspaces" while the CLI defaults to default. CONFIRMED. An existing client.list() flips from single-namespace to cross-tenant. Default to default, require explicit all_workspaces=True. (The required workspace on Sandbox() is intentional — not flagged.)

S6 — No upgrade/backfill test. CONFIRMED gap. persistence/tests.rs only migrates a fresh DB. A test seeding pre-006-shape rows (column '', payload metadata.workspace=""), migrating, then asserting visibility + correct dependent-resource cleanup would catch B1 and B3.

Minor

• Workspace deletion is non-atomic (workspace.rs:193-233): TOCTOU between emptiness check and delete (no txn primitive); provider_credential_refresh_state missing from the blocker scan.
• CLI provider list-profiles has no --global flag (main.rs:~3379) — profiles imported with --global (workspace "") can't be listed back.
• --all-workspaces + --workspace X silently ignores X; add a conflicts_with guard.
• StoredProviderProfile::requires_workspace() returns false (metadata.rs:229) while profiles are otherwise workspace-scoped — disables the persistence safety net; flip to true or document the global-"" intent.
• SSH-session cleanup (compute/mod.rs:~1398) ignores the list Err with no log.
• Stale SSH config blocks: host alias moved to openshell-{name}.{workspace}, orphaning old openshell-{name} blocks (ssh.rs:~1458).
• Old 2-segment service hostnames now misparse (workspace=sandbox, sandbox=service) rather than cleanly rejecting — add a comment or explicit legacy rejection.

Note: the inference Cluster*→Route* method-path rename is a control-plane wire break for version-skewed clients — worth a CHANGELOG line (payloads/field numbers are unaffected, so persisted rows are fine).

[russellb]

Generated using Codex and gpt-5.5.

Principal engineer review findings:

1. P1: all_workspaces is exposed to ordinary workspace-role callers.

   The RFC says all_workspaces=true is Platform Admin only, and workspace roles must not be able to set it. The service annotations still allow normal user role callers for the RPCs that honor this flag:

   • crates/openshell-server/src/grpc/mod.rs:240 ListSandboxes requires role = "user".
   • crates/openshell-server/src/grpc/mod.rs:339 ListServices requires role = "user".
   • crates/openshell-server/src/grpc/mod.rs:381 ListProviders requires role = "user".

   The handlers then trust request.all_workspaces directly and call the all-message store path:

   • crates/openshell-server/src/grpc/sandbox.rs:275
   • crates/openshell-server/src/grpc/provider.rs:1265
   • crates/openshell-server/src/grpc/service.rs:163

   This creates a cross-workspace listing bypass for any caller who can invoke the normal workspace-scoped list APIs. Even if full Phase 2 authorization is not complete, the Phase 1 contract in the RFC distinguishes workspace-scoped reads from Platform Admin reads, so the server should reject or ignore all_workspaces unless the caller is platform-scoped/admin.

2. P1: Platform-scoped provider profile management is unreachable; --global maps to the default workspace.

   The proto documents empty workspace as platform-scoped for provider profiles:

   • proto/openshell.proto:947-954 ListProviderProfilesRequest
   • proto/openshell.proto:1196-1200 ImportProviderProfilesRequest
   • proto/openshell.proto:1210-1222 UpdateProviderProfilesRequest
   • proto/openshell.proto:1251-1256 DeleteProviderProfileRequest

   The CLI also appears to intend this behavior: crates/openshell-cli/src/main.rs:3383-3385 maps --global to workspace = "".

   However, the server profile handlers normalize empty workspace through resolve_workspace, which turns it into "default" before reading or writing:

   • crates/openshell-server/src/grpc/provider.rs:1291-1296
   • crates/openshell-server/src/grpc/provider.rs:1312-1317
   • crates/openshell-server/src/grpc/provider.rs:1331-1338
   • crates/openshell-server/src/grpc/provider.rs:1362-1369
   • crates/openshell-server/src/grpc/provider.rs:1397-1405
   • crates/openshell-server/src/grpc/provider.rs:1512-1518
   • crates/openshell-server/src/grpc/provider.rs:1532-1546

   Lower-level provider profile code and tests still treat "" as platform scope, for example get_provider_type_profile and merged_provider_profiles use the exact workspace argument, and provider_with_global_profile_resolves_platform_scoped_profile stores in "" and asserts it is not visible as "default" (crates/openshell-server/src/grpc/provider.rs:7005-7033). As written, users cannot manage true platform-scoped profiles through the new APIs/CLI; --global silently creates default-workspace profiles instead.

3. P2: Service endpoint host validation can emit invalid DNS labels.

   Service routing now emits a single leftmost label as {workspace}--{sandbox} or {workspace}--{sandbox}--{service} (crates/openshell-server/src/service_routing.rs:77-88). The individual validators allow sandbox and service names up to 28 chars each (crates/openshell-server/src/grpc/service.rs:21-22, :31-32, :138-139, :207-208).

   With the default workspace alone, a max-length sandbox and service produce 7 + 2 + 28 + 2 + 28 = 67 chars in one DNS label, exceeding the 63-char DNS label limit. Non-default workspaces make this worse; the CLI help says workspace names are DNS-1123 labels up to 63 chars, while server-side validate_dns1123_label uses the global MAX_NAME_LEN = 253 (crates/openshell-server/src/grpc/mod.rs:108).

   The Kubernetes driver has a combined-name guard for resource names (crates/openshell-driver-kubernetes/src/driver.rs:905-915), but service routing does not appear to have an equivalent combined hostname-label guard. A user can therefore expose a service successfully and receive a URL that is not a valid hostname.

4. P2: GetSandboxLogsRequest.workspace is accepted but not enforced against sandbox_id.

   The proto added workspace to GetSandboxLogsRequest and documents it as the workspace scope (proto/openshell.proto:1465-1478). The handler resolves it but names it _workspace, then never uses it:

   • crates/openshell-server/src/grpc/policy.rs:2150-2157
   • crates/openshell-server/src/grpc/policy.rs:2161-2162

   The log tail is keyed only by sandbox_id, so a request with workspace=beta and a sandbox ID from alpha returns the alpha sandbox logs if the caller knows the ID. This violates the new request semantics and will become an authorization bypass once workspace membership enforcement is layered on top.

5. P3: Policy revisions and draft chunks can be orphaned after workspace deletion.

   The RFC says policy revisions and drafts are workspace-scoped resources. The new write paths do store them with workspace context:

   • crates/openshell-server/src/grpc/policy.rs:1168
   • crates/openshell-server/src/grpc/policy.rs:1592
   • crates/openshell-server/src/grpc/policy.rs:1930
   • crates/openshell-server/src/grpc/policy.rs:2447-2450

   But sandbox cleanup only deletes SSH sessions and sandbox settings (crates/openshell-server/src/compute/mod.rs:1379-1395), and workspace deletion blocks on sandbox/provider/service/route/session/settings object types only (crates/openshell-server/src/grpc/workspace.rs:193-202). It does not check or delete sandbox_policy / draft_policy_chunk rows (crates/openshell-server/src/persistence/mod.rs:23-25).

   After deleting the sandboxes in a workspace, deleting the workspace can therefore succeed while leaving workspace-owned policy revision and draft rows behind. The migration also appears to backfill object types named policy and draft_chunk rather than the actual constants sandbox_policy and draft_policy_chunk, which increases the chance of stale policy rows surviving workspace migration and deletion.

[jhjaggars]

Potential Concerns (automated review)

• No authorization/membership check on any workspace-scoped RPC. resolve_workspace only verifies the workspace exists — it does not check the caller is a member of it. Every handler in grpc/provider.rs, grpc/sandbox.rs, grpc/service.rs, and grpc/workspace.rs (including AddWorkspaceMember/RemoveWorkspaceMember) trusts the client-supplied workspace field with no gate. This matches the PR's stated "Phase 2 not started" scope, but until Phase 2 lands, workspace boundaries are advisory only — any authenticated principal can read/write another workspace's providers, sandboxes, or membership by simply naming it. Worth confirming this interim state is acceptable to merge as-is.
• ListProviders/similar all_workspaces=true escape hatches return cross-tenant data (names/types/config, not credentials) with no additional privilege check visible in the handler.
• Inconsistent backward-compat behavior across compute drivers: Kubernetes hard-errors on sandboxes/pods missing the workspace label (pre-migration objects), while Docker/Podman silently default to workspace "". No migration/backfill step for already-running sandboxes was found in the diff.
• SSH host alias format changed (openshell-{name} → openshell-{name}.{workspace}), which will orphan existing ~/.ssh/config entries generated by older CLI versions.
• TOCTOU on workspace deletion: the "any resources exist" scan and the final delete are not transactional; a resource could be created in the gap.
• Stringly-typed "" sentinel overload in the CLI (profile_workspace: "" = global, workspace: "" = all-workspaces) — functionally fine but fragile if the accompanying bool flags are ever refactored away.
• TUI: selected row index isn't reset after switching workspace (can silently point at an unrelated row), and workspace-list refresh failures are silently logged rather than surfaced.
• E2e coverage gap: workspace_lifecycle.rs covers CRUD + provider isolation + delete-guard, but doesn't exercise the new membership RPCs or any role-based behavior — consistent with Phase 2 being separate work.
• Breaking changes to flag in release notes: inference RPC/message rename (Cluster* → Route*) and the Python SDK's workspace becoming a required kwarg on several methods.

---

Review conducted by Claude (Sonnet 5).

[jhjaggars]

Deep review — verified against PR head 1785722c

A second, deeper pass (full checkout, not just the diff). Findings below were confirmed by reading the actual code; a few first-pass concerns were checked and cleared (noted at the end).

Note: CI hasn't meaningfully run yet — "Branch Checks" / "Helm Lint" are pending on the /ok to test mirror gate, and the E2E checks reported pass only because test:e2e isn't applied.

High severity

1. Cross-workspace sandbox access by ID — Phase 2 will not fix this. The sandbox data-plane RPCs have no workspace dimension. ExecSandbox, ExecSandboxInteractive, WatchSandbox, CreateSshSession, and ForwardTcp carry no workspace field in the proto, and handle_get_sandbox_logs (grpc/policy.rs:2155-2162) resolves the workspace into an unused _workspace, then serves logs purely by sandbox_id. Because ListSandboxes with all_workspaces=true is available at role = "user" and returns sandbox IDs for every workspace, any tenant can enumerate IDs and then exec into / watch / SSH into / read logs of another workspace's sandbox. The planned Phase 2 membership check lives in resolve_workspace, which these handlers either never call or ignore — so closing this later requires adding proto fields. Cheapest to do now, while the proto is already changing. (Everything else follows a sound (workspace, name) keyed-lookup pattern; sandbox-principal RPCs correctly derive workspace server-side.)

2. Upgrades orphan/break existing sandboxes (podman, kubernetes). Sandbox lookup switched to a label selector on openshell.ai/sandbox-id.

• Podman: the label key literally changed from openshell.sandbox-id → openshell.ai/sandbox-id (crates/openshell-driver-podman/src/container.rs). Pre-upgrade containers never match, so stop/delete/get silently fail; the reconciler then deletes the store record after the grace period while the real container keeps running invisibly.
• Kubernetes: sandbox_from_object hard-errors when the workspace annotation/label is absent, and list_sandboxes collects with ? — a single pre-upgrade pod fails the entire list, and the watch loop enters a permanent restart cycle, killing reconciliation gateway-wide.
• Docker is fine (label keys unchanged; missing workspace falls back to "").

No migration, legacy-key fallback, or upgrade note exists. Recommend at minimum a documented "delete sandboxes before upgrading" note, ideally a legacy-label fallback + per-object skip (rather than fail-the-list) in the k8s driver.

3. Migration 006 orphans existing provider profiles. The backfill allowlist omits provider_profile. Because StoredProviderProfile::requires_workspace() is false, it also dodged the write-guard safety net — yet every post-PR profile read/write is workspace-scoped (resolve_workspace → "default"). Existing custom/imported profiles stay at workspace='', so after upgrade they disappear from list/get/update/delete, and re-import creates duplicates under default while the '' rows orphan permanently. (Inference config is safe — already stored as inference_route with a non-null name, and it's in the backfill list.)

4. Proto field 9 reused with a changed type. SandboxSpec field 9 went from bool gpu (varint) to ResourceRequirements resource_requirements (message) in both proto/openshell.proto and proto/compute_driver.proto. This is a silent wire-decode hazard for persisted sandbox specs and older clients sending gpu=true. This appears to be a GPU-refactor change riding along in this branch rather than core to the workspace feature — worth confirming it's intentional and that no persisted specs predate it. (Other removed fields were reserved; 9 was reused.)

Medium severity

5. Global/platform provider-profile scope is unreachable. Proto + CLI treat empty workspace as "platform/global" scope (--global, --global-profile), but every profile RPC handler runs the field through resolve_workspace, which folds "" → "default". So provider profile import --global writes into the default workspace, and provider create --global-profile (which looks up partition "" directly) can never resolve it. The passing unit test masks this by writing the ""-scoped profile straight into the store, bypassing the RPC.

6. Sandbox denial/activity flush can permanently emit workspace="". The sandbox-side workspace watch channel is seeded "" and workspace_tx.send fires only on the initial poll_settings success — the retry loop never re-sends. If the first poll fails, every subsequent flush ships workspace="" forever. Server-side this resolves to default and fails the sandbox-principal scope check, so denials/drafts are silently dropped (warn-only) rather than misfiled into another tenant — data loss, not contamination. A skip-while-empty guard (or resend on the retry path) would fix it.

7. Combined hostname label can exceed 63 chars. endpoint_host emits {workspace}--{sandbox}--{service} as a single DNS label with no combined-length check; individual name caps are generous enough that e.g. a 30-char workspace + 28-char sandbox + service overflows 63, yielding a dead endpoint URL. K8s validates the {workspace}--{name} resource name at 63 but not the derived hostname.

Low / cleanup

• Docs gap (project rules require docs for user-facing changes): 36 docs files changed on the branch, none cover the workspace feature — no --workspace flag, no openshell workspace commands, no multi-tenancy page; the breaking Python SDK change and the new openshell-{name}.{workspace} SSH alias are undocumented.
• e2e Python suite not migrated: e2e/python/conftest.py:58 and several test_*.py call Sandbox(...) / create() without the now-required workspace kwarg; test_inference_routing.py still calls the removed set_cluster()/get_cluster() and imports the renamed ClusterInferenceConfig. These fail at runtime.
• Stale rename straggler: architecture/google-vertex-ai-provider.md:63 (new in this PR) still references SetClusterInferenceRequest.
• TOCTOU on workspace deletion (scan-then-delete not transactional) and the "" sentinel overload carry over from the earlier comment.

Cleared on deeper inspection

-- delimiter safety is enforced and test-pinned end-to-end (validators reject --; parse_host independently fail-closes to 404). CLI provider create --global-profile arg order is correct (no workspace/profile_workspace swap). Migrations run in per-file transactions (sqlx), so there's no window without the uniqueness index. Write guards are real runtime errors, not debug_assert! (PR description is stale on this). TUI cycle_workspace ring logic is correct. all_workspaces mutual-exclusion is checked consistently before workspace resolution.

---

Deep review pass conducted by Claude (Fable 5).