Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Normalize German umlaut #125

Open
michelole opened this issue Dec 3, 2019 · 2 comments
Open

Normalize German umlaut #125

michelole opened this issue Dec 3, 2019 · 2 comments
Labels
P2 High priority issues, a COULD

Comments

@michelole
Copy link
Member

michelole commented Dec 3, 2019 •
edited
Loading

Normalize German umlaut ("ä" -> "ae", etc.) before applying filtering rules, preferably at training time to make models (a bit) denser.

Branched from #87.

A German stemmer (#123) would fix it.
@michelole michelole added the P1 Higher priority issues, a SHOULD label Dec 3, 2019
@michelole
Copy link
Member Author

This does not seem a real problem in our training set (e.g. "ueber*", which would expand both ÜZ and ÜLT, occurs only 2 times), downgrading.

@michelole michelole added P2 High priority issues, a COULD and removed P1 Higher priority issues, a SHOULD labels Dec 3, 2019
@michelole
Copy link
Member Author

Consider using unidecode instead of the old transliterate_to_seven_bit function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P2 High priority issues, a COULD
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@michelole