wip: comment out prepend full_text #3079
Conversation
| if let Some(token) = authorization_token { | ||
| builder = builder.with_token(Some(token)); | ||
| } | ||
| let mut builder = ApiBuilder::new() |
There was a problem hiding this comment.
this chunk will be reversed, sorry it got in here
|
What you did here is effectively deactivate its job. Which is really simply to decode all the decoder tokens, instead of only the new ones. I was under the impression that T5 was a encoder/decoder model so I'm surprised to see a decoder-only model here. I'm pretty sure the "bug" will simply end up being a tokenizer issue, where possibly you're hitting a very old T5 non flash version that doesn't support some flags used in those tokenizers. Thanks for the fix for your use case, I'll leave it up for others to see, but if we're going to fix it, we need to find the root cause and fix that instead. |
|
@Narsil thanks for looking into this and explaining it to me to give me a sense of what the root of the problem could be. it was nice to see madlad being served with good results and may just use this for the time being. looking forward to tracking this solution and learning more about this great project💪 |
What does this PR do?
This is an investigation as to why using
"return_full_text": trueas a parameter in hitting the/generateendpoint produces a valid translation while running madlad400Background: #1416 (comment)
Fixes #1416
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.