Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: set sharded true if WORLD_SIZE is set #2062

Closed
wants to merge 1 commit into from

Conversation

drbh
Copy link
Collaborator

@drbh drbh commented Jun 12, 2024

This is a tiny PR that simply set assumes sharded is true if a WORLD_SIZE env is set.

Adding because I wasted time debugging when I just forgot to set --sharded along with the WORLD_SIZE and RANK env vars. This PR allows --sharded to be inferred and is for dev convenience

note** we already do this in the launcher

// Activate tensor parallelism
if world_size > 1 {
shard_args.push("--sharded".to_string());
}
but this avoids the issue if you are running the server/router separately

@drbh drbh force-pushed the derive-sharded-from-args branch from c1e0528 to c4b333f Compare June 14, 2024 13:17
@drbh drbh force-pushed the derive-sharded-from-args branch from c4b333f to 9854f20 Compare June 20, 2024 00:37
@Narsil
Copy link
Collaborator

Narsil commented Jun 24, 2024

Not a huge fan tbh.

All of this is supposed to be handled by the launcher which shouldn't make any mistake.
I understand the inconvenience, but this seems like opening up a new class of bugs for relatively little benefit. (Because WORLD_SIZE could be defined by somehow else than the launcher).

Let's see how the others feel about this.

@drbh
Copy link
Collaborator Author

drbh commented Jul 2, 2024

Closing since this change is unneeded and it does not seem like others are interested at the moment. Will revisit if interest arises

@drbh drbh closed this Jul 2, 2024
@drbh drbh deleted the derive-sharded-from-args branch July 2, 2024 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants