Mismatch between valid scoring during training and real evaluation #2309
Description
We have a mismatch because:
The reference is made of the detokenized of tokenized target.... I know it should not be an issue at first glance.
The thing is that when you tokenize the target, you can get "<unk>" tokens and when you detokenize you lose the information.
As a result the scoring metric is in most cases better than it should be.
@l-k-11235 @anderleich
If one of you wants to fix this, welcome.