🎀 New defaults: gradient_checkpointing=True
#3510
Conversation
…ingface/trl into gradient_checkpointing_true
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| "help": "If True, use gradient checkpointing to save memory at the expense of slower backward pass." | ||
| }, | ||
| ) | ||
|
|
There was a problem hiding this comment.
So technically you wanna overwrite by setting the default to be True always?
always doing the grad_checkpointing
I guess in most of the trainer scripts most of us set this to True, so I believe its a good time to change the default!
There was a problem hiding this comment.
Yes! that's the underlying motivation. We should do that with all the arguments that match:
most of the trainer scripts most of us set this to [VALUE]
| learning_rate (`float`, *optional*, defaults to `1e-6`): | ||
| Initial learning rate for [`AdamW`] optimizer. The default value replaces that of | ||
| [`~transformers.TrainingArguments`]. |
There was a problem hiding this comment.
Giving the added
Note that default values in this class may differ from those in [
~transformers.TrainingArguments]."
there is no need to double-document
qgallouedec
changed the title
gradient_checkpointing=Truegradient_checkpointing=True
|
No, not yet, I have to fix the tests before merging. |
What does this PR do?
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.