关于LiSenNet的MACs和RTF

[xiaocui-iii]

您好，我对你们团队的工作很感兴趣，我运行了你们的代码，其中只显示了LiSenNet网络的Params，并没有显示它的MACs和RTF值，于是我自行写了代码计算它们的值，与论文中的值有出入，请问你们能分享计算这两个值的代码吗

[hyyan2k]

你好，MACs的计算可以参照如下代码：

from model.generator import LiSenNet
from ptflops import get_model_complexity_info


model = LiSenNet()
macs, params = get_model_complexity_info(model, (16000,), as_strings=True, backend='pytorch', print_per_layer_stat=True, verbose=True)
print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
print('{:<30}  {:<8}'.format('Number of parameters: ', params))

RTF值可通过累计推理多个音频的处理时间（仅forward）后除以音频总时长获得，RTF值与设备关系较大。

[xiaocui-iii]

好的，感谢

[xiaocui-iii]

您好，我在阅读LiSenNet论文时，看到里面用了WSJ0+ESC50这个数据集，这个数据集是你们团队将WSJ0和ESC50两个数据集自己混合的吗，我在google上没有找到已混合好的，如果我想复现文中的数据可以自己写代码混合这两个数据集吗，或者如果方便的话，是否可以提供下载数据集的链接或者混合数据集的代码？

[hyyan2k]

WSJ0-ESC50数据集是我们自己混的，这里可以提供混合代码供参考。

首先需要对ESC50数据集的样本做基于能量阈值的静音段去除，可以参考espnet中的detect_non_silence函数：
from espnet2.train.preprocessor import detect_non_silence

然后进行混合：

from pathlib import Path
import soundfile as sf
import numpy as np
import random
from copy import deepcopy

speech_dir = Path('<path-to-speech-dir>')
noise_dir = Path('<path-to-esc50-dir>')
set_name = 'train'
speech_dur = 4.0  # 4s
noise_durs = [0.5, 2.0]  # 0.5s ~ 2.0s
snr_dbs = [-10, 5]

clean_dir = Path(f'./{set_name}_clean')
noisy_dir = Path(f'./{set_name}_noisy')
start_end = Path(f'./{set_name}_start_end.txt')
clean_dir.mkdir(parents=True, exist_ok=True)
noisy_dir.mkdir(parents=True, exist_ok=True)
start_end = open(start_end, 'w')

speech_list = list(speech_dir.rglob('*.wav'))
noise_list = list(noise_dir.rglob('*.wav'))

######
if set_name =='train':
  noise_list = noise_list[:int(len(noise_list)*0.8)]   # for train
else:
  noise_list = noise_list[int(len(noise_list)*0.8):]  # for test
######

def add_noise_to_signal(signal, noise, snr_db):
    signal_power = (signal ** 2).mean()
    
    snr_linear = 10 ** (snr_db / 10)
    noise_power = signal_power / snr_linear
    
    noise = noise / noise.std() * np.sqrt(noise_power)
    
    return signal + noise


for ii, speech_path in enumerate(speech_list):
    print(f'\r{ii/len(speech_list):.2%}', end='')

    speech, sr = sf.read(str(speech_path))
    assert speech.ndim == 1 and sr == 16000
    speech_length = int(speech_dur*sr)

    if len(speech) < speech_length:
        speech = np.pad(speech, (0, speech_length-len(speech)), 'wrap')
    
    start_points = list(range(0, len(speech), speech_length))
    start_points[-1] = len(speech) - speech_length

    #######
    start_points = [int(np.random.randint(0, len(speech)-speech_length+1))]
    #######

    for jj, start in enumerate(start_points):
        speech_part = speech[start: start + speech_length]

        noise_path = random.sample(noise_list, k=1)[0]
        noise, sr = sf.read(str(noise_path))
        assert noise.ndim == 1 and sr == 16000
        noise_dur = random.uniform(*noise_durs)
        noise_length = int(noise_dur * sr)

        if len(noise) <= noise_length:
            noise = np.pad(noise, (0, noise_length-len(noise)), 'wrap')
        else:
            s = random.randint(0, len(noise)-noise_length)
            noise = noise[s:s+noise_length]
        
        part_start = random.randint(0, len(speech_part)-len(noise))
        part_end = part_start + len(noise)
        speech_part_copy = deepcopy(speech_part)
        speech_part_copy[part_start:part_end] = add_noise_to_signal(speech_part[part_start:part_end], noise, random.uniform(*snr_dbs))

        sf.write(clean_dir/f'audio-{ii}-{jj}.wav', speech_part, samplerate=sr)
        sf.write(noisy_dir/f'audio-{ii}-{jj}.wav', speech_part_copy, samplerate=sr)
        start_end.write(f'audio-{ii}-{jj} {part_start} {part_end}\n')

start_end.close()