Skip to content

Pathfinder: add support on cutensorMp library loading#2374

Merged
rwgk merged 3 commits into
NVIDIA:mainfrom
yangcal:pathfinder/cutensormp
Jul 15, 2026
Merged

Pathfinder: add support on cutensorMp library loading#2374
rwgk merged 3 commits into
NVIDIA:mainfrom
yangcal:pathfinder/cutensormp

Conversation

@yangcal

@yangcal yangcal commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Description

This PR enables pathfinder to support cutensorMp library loading

@copy-pr-bot

copy-pr-bot Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.pathfinder Everything related to the cuda.pathfinder module label Jul 15, 2026
@yangcal yangcal requested a review from leofang July 15, 2026 17:32
@yangcal

yangcal commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@b-kloss @ChrisPsa for vis, for cutensorMp bindings, we need to first have library loading support from pathfinder

@rwgk rwgk added this to the cuda.pathfinder next milestone Jul 15, 2026
@rwgk rwgk added the P0 High priority - Must do! label Jul 15, 2026
@rwgk rwgk self-requested a review July 15, 2026 17:48
@rwgk

rwgk commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

QQ (I only have a minute right now):

Did you already look at cuda_pathfinder/pyproject.toml?

Do we need to add cutensorMp to the test dependencies? (Do we need to distinguish cu12, cu13?)

@yangcal

yangcal commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@rwgk good question.

cutensorMp is shipped as part of the cutensor-cu12/13 package and the cutensor-cuXX wheels are already included in the cu12/13 dependency groups so there is no action required there. For the so path, there is no difference between cu12/cu13, both under site-packages/cutensor/lib (same as the existing base cutensor library)

Re: tests, I believe the existing parameterized tests automatically include cutensorMp as part of descriptor tests and linux dynamic loading tests. Therefore no action should be needed there either.

@rwgk
Fix cuTENSORMp CUDA runtime preload
cf47da7 
CUDA 12's libcutensorMp.so.2 directly depends on libcudart.so.12, but its wheel RUNPATH does not include the cuda_runtime package. Pathfinder can therefore find and preload cuTENSOR, cuBLASLt, and NCCL yet still fail to dlopen cuTENSORMp in a wheel-only environment unless libcudart is already available from an ambient toolkit.

Add cudart to the descriptor dependency list so Pathfinder resolves the packaged CUDA runtime before opening cuTENSORMp. This makes CUDA 12 wheel-only loading independent of LD_LIBRARY_PATH while preserving CUDA 13 and local-toolkit behavior.
@rwgk

rwgk commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/ok to test cf47da7

rwgk
rwgk approved these changes Jul 15, 2026
View reviewed changes

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex figured out that cudart was missing in the dependencies list. I went ahead an added it, then triggered the CI.

@rwgk rwgk enabled auto-merge (squash) July 15, 2026 21:28
@github-actions

This comment has been minimized.

Sign in to view
@yangcal

yangcal commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Codex figured out that cudart was missing in the dependencies list. I went ahead an added it, then triggered the CI.

I can confirm cudart is indeed a required dependency. Thanks for putting up the fix

@rwgk rwgk merged commit a533b15 into NVIDIA:main Jul 15, 2026
199 of 201 checks passed
@github-actions

This comment has been minimized.

Sign in to view
1 similar comment
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rwgk rwgk rwgk approved these changes
@leofang leofang Awaiting requested review from leofang

Assignees

@yangcal yangcal

Labels

cuda.pathfinder Everything related to the cuda.pathfinder module P0 High priority - Must do!

Projects

None yet

Milestone

cuda.pathfinder next

Development

Successfully merging this pull request may close these issues.

2 participants

@yangcal @rwgk