Tokenizer is only recognized for German models

[MihailMiller]

BPE tokenizer not found in any of: ['/root/.cache/espnet/models--speechcatcher--wordcab_speechcatcher_english_espnet_streaming_transformer_35k_train_size_m_raw_en_bpe1024/snapshots/c77f02d1bc3dd924912373336ab3434fd6431060/exp/asr_train_asr_streaming_transformer_size_m_raw_en_bpe1024/bpe.model', '/root/.cache/espnet/models--speechcatcher--wordcab_speechcatcher_english_espnet_streaming_transformer_35k_train_size_m_raw_en_bpe1024/snapshots/c77f02d1bc3dd924912373336ab3434fd6431060/data/de_token_list/bpe_unigram1024/bpe.model', '/root/.cache/espnet/models--speechcatcher--wordcab_speechcatcher_english_espnet_streaming_transformer_35k_train_size_m_raw_en_bpe1024/snapshots/c77f02d1bc3dd924912373336ab3434fd6431060/exp/asr_train_asr_streaming_transformer_size_m_raw_en_bpe1024/../data/de_token_list/bpe_unigram1024/bpe.model']

In speech2text_streaming.py there is this block:

# Load BPE tokenizer if available
self.tokenizer = None
self.token_list = None
if HAS_SENTENCEPIECE:
    bpe_paths = [
        self.model_dir / "bpe.model",
        self.model_dir.parent.parent / "data/de_token_list/bpe_unigram1024/bpe.model",
        self.model_dir / "../data/de_token_list/bpe_unigram1024/bpe.model",
    ]
    ...

For English models that’s wrong – the actual bpe.model is under some .../en_token_list/... or similar. The robust fix is to make this search generic:

if HAS_SENTENCEPIECE:
    # Search generically for any bpe.model under the snapshot root
    snapshot_root = self.model_dir.parent.parent
    bpe_paths = [self.model_dir / "bpe.model"]
    bpe_paths.extend(snapshot_root.rglob("bpe.model"))
    ...

Then it will pick up whatever bpe.model is present (English, German, Spanish) instead of hard-coding de_token_list.