Skip to content

Conversation

@drbh
Copy link
Collaborator

@drbh drbh commented Jun 12, 2024

This is a tiny PR that simply set assumes sharded is true if a WORLD_SIZE env is set.

Adding because I wasted time debugging when I just forgot to set --sharded along with the WORLD_SIZE and RANK env vars. This PR allows --sharded to be inferred and is for dev convenience

note** we already do this in the launcher

// Activate tensor parallelism
if world_size > 1 {
shard_args.push("--sharded".to_string());
}
but this avoids the issue if you are running the server/router separately

@drbh drbh force-pushed the derive-sharded-from-args branch from c1e0528 to c4b333f Compare June 14, 2024 13:17
@drbh drbh force-pushed the derive-sharded-from-args branch from c4b333f to 9854f20 Compare June 20, 2024 00:37
@Narsil
Copy link
Contributor

Narsil commented Jun 24, 2024

Not a huge fan tbh.

All of this is supposed to be handled by the launcher which shouldn't make any mistake.
I understand the inconvenience, but this seems like opening up a new class of bugs for relatively little benefit. (Because WORLD_SIZE could be defined by somehow else than the launcher).

Let's see how the others feel about this.

@drbh
Copy link
Collaborator Author

drbh commented Jul 2, 2024

Closing since this change is unneeded and it does not seem like others are interested at the moment. Will revisit if interest arises

@drbh drbh closed this Jul 2, 2024
@drbh drbh deleted the derive-sharded-from-args branch July 2, 2024 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants