DTensor support for bfloat16 stochastic rounding #3266
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3266
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 2 New FailuresAs of commit 82a05bd with merge base 1e473ed ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Hi @EduardDurech! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
meta-cla
bot
added
the
CLA Signed
| # LICENSE file in the root directory of this source tree. | ||
| import torch | ||
| from torch import Tensor | ||
| try: |
There was a problem hiding this comment.
is this for different PyTorch versions? If yes, could you clarify which specific versions?
in general we support 3 most recent stable PyTorch releases max, so if it's older than that I'd just leave it out
| try: | ||
| from torch.distributed._tensor import DTensor | ||
| except Exception: | ||
| DTensor = tuple() |
There was a problem hiding this comment.
rethrow this instead returning tuple
| return x_f32_bits.view(torch.float32).bfloat16() | ||
| x_bf16_trunc = x_f32_bits.view(torch.float32).bfloat16() | ||
|
|
||
| return DTensor.from_local( |
There was a problem hiding this comment.
looks reasonable, can we add a test to cover?
2 participants
Resolves #2296
Upstream torch is not taking this seriously pytorch/pytorch#156649