Add RapidsMPF launcher rrun
#616
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks @madsbk , I think I've addressed/responded to everything. Could you take another look when you have a chance? |
madsbk
reviewed
Nov 4, 2025
| // Write to temporary file | ||
| std::ofstream ofs(tmp_path, std::ios::binary | std::ios::trunc); | ||
| if (!ofs) { | ||
| throw std::runtime_error("Failed to open temporary file: " + tmp_path); |
Comment on lines
+231
to
+233
| if (!ifs) { | ||
| throw std::runtime_error("Failed to open file for reading: " + path); | ||
| } |
cpp/tools/rrun.cpp
Outdated
| suppress_output->store(true, std::memory_order_relaxed); | ||
| // Forward signal to all local children | ||
| for (pid_t pid : pids) { | ||
| (void)kill(pid, sig); |
There was a problem hiding this comment.
Suggested change
| (void)kill(pid, sig); | |
| std::ignore = kill(pid, sig); |
cpp/tools/rrun.cpp
Outdated
Comment on lines
579
to
586
| { | ||
| std::error_code ec; | ||
| std::filesystem::remove_all(cfg.coord_dir, ec); | ||
| if (ec) { | ||
| std::cerr << "Warning: Failed to cleanup directory: " << cfg.coord_dir | ||
| << ": " << ec.message() << std::endl; | ||
| } | ||
| } |
There was a problem hiding this comment.
Suggested change
| { | |
| std::error_code ec; | |
| std::filesystem::remove_all(cfg.coord_dir, ec); | |
| if (ec) { | |
| std::cerr << "Warning: Failed to cleanup directory: " << cfg.coord_dir | |
| << ": " << ec.message() << std::endl; | |
| } | |
| } | |
| std::error_code ec; | |
| std::filesystem::remove_all(cfg.coord_dir, ec); | |
| if (ec) { | |
| std::cerr << "Warning: Failed to cleanup directory: " << cfg.coord_dir | |
| << ": " << ec.message() << std::endl; | |
| } |
Co-authored-by: Mads R. B. Kristensen <[email protected]>
Co-authored-by: Mads R. B. Kristensen <[email protected]>
pentschev
commented
Nov 4, 2025
| // Write to temporary file | ||
| std::ofstream ofs(tmp_path, std::ios::binary | std::ios::trunc); | ||
| if (!ofs) { | ||
| throw std::runtime_error("Failed to open temporary file: " + tmp_path); |
There was a problem hiding this comment.
Sorry, I committed these changes locally yesterday but didn't push them. 🤦
Fixed in db112cb
Comment on lines
+231
to
+233
| if (!ifs) { | ||
| throw std::runtime_error("Failed to open file for reading: " + path); | ||
| } |
madsbk
approved these changes
Nov 4, 2025
cpp/src/bootstrap/file_backend.cpp
Outdated
Comment on lines
210
to
221
| { | ||
| std::error_code ec; | ||
| std::filesystem::rename(tmp_path, path, ec); | ||
| if (ec) { | ||
| std::error_code rm_ec; | ||
| std::filesystem::remove(tmp_path, rm_ec); // Clean up temp file | ||
| RAPIDSMPF_FAIL( | ||
| "Failed to rename " + tmp_path + " to " + path + ": " + ec.message(), | ||
| std::runtime_error | ||
| ); | ||
| } | ||
| } |
There was a problem hiding this comment.
Suggested change
| { | |
| std::error_code ec; | |
| std::filesystem::rename(tmp_path, path, ec); | |
| if (ec) { | |
| std::error_code rm_ec; | |
| std::filesystem::remove(tmp_path, rm_ec); // Clean up temp file | |
| RAPIDSMPF_FAIL( | |
| "Failed to rename " + tmp_path + " to " + path + ": " + ec.message(), | |
| std::runtime_error | |
| ); | |
| } | |
| } | |
| std::error_code ec; | |
| std::filesystem::rename(tmp_path, path, ec); | |
| if (ec) { | |
| std::error_code rm_ec; | |
| std::filesystem::remove(tmp_path, rm_ec); // Clean up temp file | |
| RAPIDSMPF_FAIL( | |
| "Failed to rename " + tmp_path + " to " + path + ": " + ec.message(), | |
| std::runtime_error | |
| ); | |
| } |
cpp/tools/rrun.cpp
Outdated
| std::vector<int> gpus; | ||
| std::stringstream ss(gpu_str); | ||
| std::string item; | ||
|
|
cpp/tools/rrun.cpp
Outdated
| RAPIDSMPF_FAIL("Invalid GPU ID: " + item, std::runtime_error); | ||
| } | ||
| } | ||
|
|
Co-authored-by: Mads R. B. Kristensen <[email protected]>
This reverts commit db112cb.
|
@madsbk as discussed offline, we can't use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce a new launcher for RapidsMPF without dependencies on
mpirun. The new launcher currently namedrrun(as in RapidsMPF run) introduces basic functionality to launch processes locally to run on multiple-GPUs while providing an extension to allow it to support different backends, the only existing backend supports file-based synchronization, in the future we plan to extend it to support other synchronization mechanisms including over-the-network via specialized service, Slurm, as well as Kubernetes and others that may be requested. Additionally, it will support automatic discovery and configuration of system topology per process (i.e., per GPU).This initial implementation only supports local node multi-GPU, the implementation for multi-node multi-GPU already exists using SSH to spawn multiple processes but was split into a different PR to make this shorter.