Full word matches don't work properly with Chinese characters

[lonvia]

Searching for 小学校 should yield a couple of full word matches, e.g. this. Instead it seems to prefer random partial matches.

Oddly enough, there is a word entry for "小学校" which transliterates to "xiao". This needs investigation as well.

[lonvia]

This is going to need some more fundamental changes to solve. Han script doesn't work well with the current search based on transliterated partials because too many han characters match the same transliteration.

[lonvia]

I think the best way forward here is to exclude Chinese characters from transliteration for the internal index.

To make sure that Chinese names still can be found using Latin script, we'd need to add a sanitizer that adds a transliterated name variant for places that only have a name in Chinese characters.

Additional bonus of doing a manual transliteration beforehand is that we can actually take into account different transliterations for Japanese and Chinese usage. (Cantonese is a bit more tricky because the sanitizer may not have the necessary location information.)

[wangshamian]

Yeah, exclude Chinese characters from transliteration and add special analyzer in token-analysis, my idea is that Chinese query characters be analyzed to Chinese phrases (using word segmentation algorithms e.g. jieba), then the Chinese phrases saved in Word Table.