Estimate redistribute_cost by _TransformInfo #230
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Estimate redistribute_cost by _TransformInfo rather than comparing source and destination state
Here are the changes:
a. S(0)S(0) -> S(0)R, need 1 allgather
b. S(0)S(0) -> RS(0), need 2 allgather, which could not be found if only care S(0)->R
comm_bytes_gbis based on tensor shape and number of shards. In case 2.b, the comm_byte for 2 allgather is different.TODO:
There are some compute_cost with comm_bytes_gb, need to verify whether they could return the expected cost.