语音分离：Wave_U_Net用于音频源分离的实验

作者: 峭风梳骨寒 | 来源:发表于2019-10-04 10:25 被阅读0次

2019年9月30号，我老师打电话给我，给我安排任务，要我10月1号到7号期间做啥做啥，哈哈，9月30号给我布置任务。要我复现一篇文章的代码，我上谷歌学术查阅这篇文章，先发这篇文章的网站链接：https://arxiv.org/abs/1806.03185 我去github上查阅这篇文章的代码，有幸找到了，先说说我是如何复现代码的，我先放github网址：https://github.com/f90/Wave-U-Net 我从github上下来代码之后，这是给的两个demo结果，参加SISEC语音分离竞赛的算法。一个是M5模型把人声和乐器声分离的demo，我放上网址：https://sisec18.unmix.app/#/unmix/Side+Effects+Project+-+Sing+With+Me/STL1一个M6模型，适合多仪器分离的demo，我同样也放上网址：https://sisec18.unmix.app/#/unmix/Side+Effects+Project+-+Sing+With+Me/STL2 <img class="rich_pages" data-backh="68" data-backw="574" data-before-oversubscription-url="https://mmbiz.qpic.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGul1orxn7vJaOgyIdFhaLzyCUrlmK4PzmZSX69VRicb4g3AxbAWd3asOA/?wx_fmt=png" data-ratio="0.11871657754010695" data-s="300,640" src="https://img.haomeiwen.com/i3104601/f88df5cff65039aa" data-type="png" data-w="935" style="border-radius: 28px;width: 100%;height: auto;"> <img class="rich_pages" data-backh="65" data-backw="574" data-before-oversubscription-url="https://mmbiz.qpic.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGu0dLziaY1v8xgVH2rboccAibsR1Ao8wr8ZkdeYmb3KqXs3XDG9zV6y0rg/?wx_fmt=png" data-ratio="0.1125" data-s="300,640" src="https://img.haomeiwen.com/i3104601/16813381a96a99c5" data-type="png" data-w="960" style="border-radius: 32px;width: 100%;height: auto;"> 这个代码执行环境有要求，需要cuda9.0的环境，我开始在cuda8上面执行代码，直接报错了，因此想要复现这篇文章的代码，就需要cuda9的环境，如果要进行训练的话，需要gpu进行，tensorflow-gpu框架下面。要求安装的包在文档里面有一个requirements.txt。 <img class="rich_pages" data-backh="101" data-backw="574" data-before-oversubscription-url="https://mmbiz.qpic.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGutrDK13htx907hyM83icCpqPmcqSNNpibYEon0CNrrzRuRVviauE5gNoNQ/?wx_fmt=png" data-ratio="0.17609391675560299" data-s="300,640" src="https://img.haomeiwen.com/i3104601/4b49f40fd8aaadb8" data-type="png" data-w="937" style="border-radius: 24px;width: 100%;height: auto;"> 在anaconda prompt 执行代码，我在anaconda里面新建了一个tf-GPU的环境，在这个环境下面，我先激活这个环境activate tf-GPU，到这个环境中来，在按要求执行pip install -r requirements.txt，他会帮我把所需要的包都安装好，非常的方便。一条指令就可以实现。 第三步是训练代码， 训练代码的话，我们需要数据集，这篇文章的代码给了两个数据集，<img class="rich_pages" data-backh="246" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGuOEfN2WH1WzgRB46M2WBhpoFLjQZUbQLz3eA7BYQOUyveRTiblmdoUkw/0?wx_fmt=png" data-ratio="0.42887249736564803" data-s="300,640" src="https://img.haomeiwen.com/i3104601/153149b2f0f34052" data-type="png" data-w="949" style="border-radius: 60px;width: 100%;height: auto;"> 一个是MUSDB18数据集，放上代码下载链接：https://sigsep.github.io/datasets/musdb.html一个是CCMixter数据集，同样放上代码链接：https://members.loria.fr/ALiutkus/kam/ 这些数据集都是用来训练模型用的，我后面也自己训练模型了，本文提供了三个模型的下载，一分别是M4,M5,M6。三个模型，放上模型下载链接：https://www.dropbox.com/s/oq0woy3cmf5s8y7/models.zip?dl=1<img class="rich_pages" data-ratio="0.733739837398374" data-s="300,640" src="https://img.haomeiwen.com/i3104601/efa05a2e6683a4cf" data-type="png" data-w="984" style=""> 下载好模型之后，我们就可以直接进行测试了。 <img class="rich_pages" data-backh="336" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGu5N4FKAaWVZ9uXltGBRaTeuHS1fhm6rjibBODiatdh9uI9712vJ1vgbMA/0?wx_fmt=png" data-ratio="0.5845360824742268" data-s="300,640" src="https://img.haomeiwen.com/i3104601/00572737a759678c" data-type="png" data-w="970" style="width: 100%;height: auto;border-radius: 20px;"><section class="xmt-style-block" data-style-type="7" data-tools="新媒体排版" data-id="9169"><section class="Powered-by-XIUMI V5" style="box-sizing: border-box;" powered-by="xiumi.us"><section class="" style="margin: 10px 0%;box-sizing: border-box;"><section class="" style="display: inline-block;width: 100%;vertical-align: top;border-style: solid;border-width: 1px;border-radius: 0px;border-color: rgb(241, 241, 241);box-sizing: border-box;"><section class="Powered-by-XIUMI V5" style="box-sizing: border-box;" powered-by="xiumi.us"><section class="" style="text-align: center;margin-top: 10px;margin-bottom: 10px;box-sizing: border-box;"><section class="" style="max-width: 100%;vertical-align: middle;display: inline-block;overflow: hidden !important;box-sizing: border-box;"><svg xmlns="http://www.w3.org/2000/svg" x="0px" y="0px" viewbox="0 0 113.8 21.3" style="vertical-align: middle;max-width: 100%;box-sizing: border-box;" width="113.8"><g style="box-sizing: border-box;"><path style="fill-rule: evenodd;clip-rule: evenodd;box-sizing: border-box;" d="M97,9.2c5.5-3.4,12.3-4.2,14.5-1.4c1.2,1.5,1.6,3.5-0.4,4.5 c-2,1-3.1-0.6-2.9-1.6c0.2-1,1.4-1.6,2.5-1.2c-0.7-1.2-3.1-1.1-3.9,0.9c-0.8,1.9,1.5,3.9,4.3,2.8c2.8-1.1,3.5-4.4,1.6-6.2 C109,3.4,102,5.2,97.8,8.1c2.1-3.8-1.2-8.3-5.4-8.1c-3.9,0.2-6.4,5.4-4.1,7.4c2.2,2,4.2-0.6,2.8-1.6c-1.2-0.9-1.7,0.4-2.2,0 c-0.5-0.4-0.3-3.9,2.9-4.4c4.2-0.7,6.8,4.1,4,7.3c-3,3.5-9.7,1.1-17.1-4.3c-6.8-5-17.2-6.4-21.8,2.3c5.8-6.9,15.8-4.9,20.8-1.2 c8.7,6.3,14.6,7.2,17.9,4.9C96.1,10,96.6,9.6,97,9.2z" fill="rgb(127, 61, 28)"></path><path style="fill-rule: evenodd;clip-rule: evenodd;box-sizing: border-box;" d="M16.8,9.2C11.3,5.9,4.5,5,2.3,7.8c-1.2,1.5-1.6,3.5,0.4,4.5 c2,1,3.1-0.6,2.9-1.6C5.3,9.6,4.1,9,3.1,9.5C3.8,8.3,6.2,8.4,7,10.3c0.8,1.9-1.5,3.9-4.3,2.8C-0.2,12.1-0.8,8.8,1.1,7 C4.8,3.4,11.8,5.2,16,8.1c-2.1-3.8,1.2-8.3,5.4-8.1c3.9,0.2,6.4,5.4,4.1,7.4c-2.2,2-4.2-0.6-2.8-1.6c1.2-0.9,1.7,0.4,2.2,0 C25.5,5.5,25.3,2,22,1.5c-4.2-0.7-6.8,4.1-4,7.3c3,3.5,9.7,1.1,17.1-4.3c6.8-5,17.2-6.4,21.8,2.3c-5.8-6.9-15.8-4.9-20.8-1.2 c-8.7,6.3-14.6,7.2-17.9,4.9C17.6,10,17.2,9.6,16.8,9.2z" fill="rgb(127, 61, 28)"></path><path style="fill-rule: evenodd;clip-rule: evenodd;box-sizing: border-box;" d="M56.9,8.5c-1.1,0.1-1.2-3.7-5.5-2.3c0,0.2,0,0.5,0.2,0.7 c-0.3,0.9-2.5,1-3,2.4c0.6,0.2,0.7,0.3,0.9,0.6c-0.9,2.1-0.3,3.6-1.1,5.1c1.1-0.2,3.1,0.7,4.2-0.2c0.2,0.8-0.3,0.9-0.2,1.7 c0.6,0.2,1.5,0.4,1.9,1.6c0.4,1.6,2,2.7,2.4,3.1c0.4-0.7,1.8-1,2.3-2.3c0.6-1.6,0.6-1.9,2-2.3c0.1-0.9-0.4-0.9-0.2-1.7 c1.2,0.9,3.1-0.2,4.2,0c-0.8-1.6-0.1-2.9-1-5c0.2-0.3,0.3-0.5,0.9-0.6c-0.5-1.4-2.6-1.5-2.9-2.4c0.2-0.2,0.2-0.5,0.2-0.7 C58.2,4.9,58,8.4,56.9,8.5L56.9,8.5z M56.7,9.4c-0.7,0.7-1.5,1.4-2.3,2.1C54,12.3,53,13.5,52,14.2c0.9-0.9,1.8-1.9,2.1-2.6 c-1.3,1-2.7,1.9-4.4,2.7c1.4-0.8,2.8-1.7,4-2.7c-1.2,0.4-2.8,0.6-4.1,0.5c1.9-0.1,3.9-0.5,4.7-1c0.6-0.5,1.1-1,1.6-1.4 c-1.5,0-3.6-0.7-6.4-0.5c2.2-0.5,4.4,0.2,6.2,0.2c-1.1-1-1.9-2.3-3.4-2.7c1.7,0.4,2.6,1.8,3.9,2.7c0.2-0.1,0.3-0.3,0.5-0.4l0.1-0.4 L57,8.9c0.1,0.1,0.3,0.3,0.4,0.4c1.3-0.9,2.2-2.3,3.9-2.7C59.8,7.1,59,8.4,58,9.4c1.8,0,4-0.8,6.2-0.3c-2.8-0.2-4.9,0.5-6.4,0.5 c0.5,0.5,1.1,1,1.7,1.4c0.8,0.5,2.8,0.8,4.7,0.8c-1.3,0.1-2.9,0-4.1-0.4c1.2,1,2.6,1.9,4.1,2.7c-1.7-0.8-3.2-1.7-4.5-2.7 c0.4,0.7,1.3,1.7,2.2,2.6c-1-0.7-2.1-1.9-2.5-2.8c-0.8-0.6-1.6-1.3-2.3-2l0,1.2c0.2,0.7,1.3,1.9,2.4,2.9c-0.9-0.6-1.9-1.7-2.4-2.5 c0,0.2-0.1,1.2,0,1.3c0.2,0.8,1.7,2.4,3.3,3.6c-1.2-0.7-2.5-2-3.2-3.1c-0.1,1,0.6,3.8,1.5,5.5c-0.8-1.3-1.4-3.3-1.5-4.7l-0.1,6.1 l-0.1-5.5c-0.2,1.3-0.7,3-1.3,4.2c0.7-1.5,1.2-3.7,1.3-4.9l0-0.5c0,0,0-0.1,0-0.1c-0.7,1.1-2,2.4-3.2,3.1c0.4-0.3,2.6-2,3.2-3.5 l0-1.4c-0.5,0.9-1.4,1.9-2.2,2.5c1-1,2-2.2,2.2-2.9L56.7,9.4z" fill="rgb(201, 109, 56)"></path></g></svg></section></section></section></section></section></section></section> 进行测试，要使用我们的预训练最佳人声分离模型（M5-HighSR）快速演示一首示例歌曲，只需执行一下即可<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">python Predict.py with cfg.full_44KHz</code>将包含在此存储库的<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">audio_examples</code>子文件夹中的歌曲“ Mallory”分离为人声和伴奏。输出将保存在输入文件旁边。要将我们的预训练模型应用于您自己的任何歌曲，只需使用<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">input_path</code>参数指向其音频文件路径：<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">python Predict.py with cfg.full_44KHz input_path="/mnt/medien/Daniel/Music/Dark Passion Play/Nightwish - Bye Bye Beautiful.mp3"</code>如果要将预测保存到自定义文件夹中而不是输入歌曲所在的位置，只需添加<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">output_path</code>参数：<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">python Predict.py with cfg.full_44KHz input_path="/mnt/medien/Daniel/Music/Dark Passion Play/Nightwish - Bye Bye Beautiful.mp3" output_path="/home/daniel"</code>如果要使用我们提供的其他预训练模型（例如我们的多仪器分隔符）或您自己的模型，请使用<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">model_path</code>参数指向Tensorflow检查点文件的位置，并确保模型配置（此处：）<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">full_multi_instrument</code>匹配模型保存在检查点中。以我们的预包装多仪器模型为例：<code style="box-sizing: border-box;font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size: 13.6px;padding: 0.2em 0.4em;background-color: rgba(27, 31, 35, 0.05);border-radius: 3px;">python Predict.py with cfg.full_multi_instrument model_path="checkpoints/full_multi_instrument/full_multi_instrument-134067" input_path="/mnt/medien/Daniel/Music/Dark Passion Play/Nightwish - Bye Bye Beautiful.mp3" output_path="/home/daniel"</code>我们需要注意的几点就是，我们下载下来的模型model，解压里面有三个文件夹，复制到checkpoints中，要与代码里面的模型位置对应， <img class="rich_pages" data-backh="105" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGuugtcxfGgdu9ECgFGmia76T61GMCeZBiaKyyVOvM0oxrPeOURrRzglIfQ/0?wx_fmt=png" data-ratio="0.18262586377097728" data-s="300,640" src="https://img.haomeiwen.com/i3104601/567b229d3bcbbb73" data-type="png" data-w="1013" style="width: 100%;height: auto;border-radius: 28px;"> 代码里面的这里，在Predict.py里面，要对应好。我测试了他给的三个模型，现在给测试结果，第一个测试模型M5，人声分离最好的结果，第一个mix是混合语音，第二个vocals是人声的，第三个acc是乐器的声音，可以听到分离效果，给出的评估是这样的。SDR是源失真率，越高代表越好。 <img class="rich_pages" data-backh="34" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGuICVWN9aGnPESZhhbHzCOFBhA2Zaqf9DVkH0DBJ1PBwcjlQPALVrqRw/0?wx_fmt=png" data-ratio="0.05968331303288672" data-s="300,640" src="https://img.haomeiwen.com/i3104601/9cc5552292188014" data-type="png" data-w="821" style="border-radius: 8px;width: 574px;height: 34px;"> 第二个模型是M6模型，对多仪器声音进行分离，我现在给出分离的效果demo链接。因为一篇图文最多只能放三个语音，我把其他模型我做的实验结果放在百度网盘里面，放上链接，感兴趣的可以听听分离效果。链接：https://pan.baidu.com/s/1P9BZqY2EGDVr9XZuWWtRxw 提取码：mto1<section class="xmt-style-block" data-style-type="7" data-tools="新媒体排版" data-id="8485"><section class="xmt-style-block" data-style-type="7" data-tools="新媒体排版" data-id="8658"><p style="font-size: 16px;line-height: 25.6px;white-space: normal;max-width: 100%;min-height: 1em;color: rgb(62, 62, 62);box-sizing: border-box !important;overflow-wrap: break-word !important;background-color: rgb(255, 255, 255);" class=""><img class="" data-backh="30" data-backw="464" data-ratio="0.065625" src="https://img.haomeiwen.com/i3104601/93ec18611b099b2f" data-type="png" data-w="640" style="color: rgb(62, 62, 62);font-size: 16px;text-align: justify;line-height: 25.6px;box-sizing: border-box !important;overflow-wrap: break-word !important;visibility: visible !important;width: 670px !important;height: auto !important;" width="670px"></section></section> 接下来我想讲讲我训练模型的经过，训练模型的话，首先我们要准备好数据集，数据集的下载我在前面已经放上网址了，下载好放在对应的文件夹，这个就要看代码了。在Config.py对应代码里面。<img class="rich_pages" data-backh="104" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGuT4kriaz98gI1tVSd5c80ia3zibCrR6bhBYbMqTn7ItOCakXAHboMAkcUA/0?wx_fmt=png" data-ratio="0.18148487626031165" data-s="300,640" src="https://img.haomeiwen.com/i3104601/35ae72f0640c15c1" data-type="png" data-w="1091" style="width: 100%;height: auto;border-radius: 16px;"> 在这个代码里面，我们需要调整的是这里的路径，对应我们数据集的位置，才可以进行训练。意思是代码要能够找到数据集所在的位置，才能正确读取数据集。<img class="rich_pages" data-backh="138" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGuJemMhBiajKGXKFLkOxiaEU9uAEmdt7dJSarvG4eOtMkI3UxErLpQVnNg/0?wx_fmt=png" data-ratio="0.24109014675052412" data-s="300,640" src="https://img.haomeiwen.com/i3104601/dc8925f7331a15e0" data-type="png" data-w="954" style="width: 100%;height: auto;border-radius: 20px;">CCMixer数据集也需要看数据集的位置，databaseFlidpath这个地方的数据集位置，才可以进行训练，训练每个模型的命令，前面的图片里面有。可以直接进行训练模型，非常方便。在CCMixer.xm里面。<img class="rich_pages" data-backh="91" data-backw="574" data-before-oversubscription-url="https://mmbiz.qlogo.cn/mmbiz_png/2wBbSkC1aMKZ2K1icYQwWU0ociaib6IzibGu53C694UHgvr3pF2GtgExMibEEs5KR8eRiaS2WEVKbMibOBGT0aBJZW5Ow/0?wx_fmt=png" data-ratio="0.15934959349593497" data-s="300,640" src="https://img.haomeiwen.com/i3104601/6aa0ee5957cf65e5" data-type="png" data-w="615" style="width: 100%;height: auto;border-radius: 12px;"> 我放上我自己训练代码以及测试的视屏，对你整个操作流程应该有帮助，<span style="color: rgb(0, 0, 0);font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;text-align: left;text-indent: 34px;background-color: rgb(212, 250, 0);"> 要想成功，少不了一个耐抗耐打的好身体。 <span style="background-color: rgb(212, 250, 0);max-width: 100%;font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;text-align: left;text-indent: 34px;color: rgb(0, 0, 0);box-sizing: border-box !important;overflow-wrap: break-word !important;">山不向我走来，我便向山走去，<span style="background-color: rgb(255, 0, 0);max-width: 100%;font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;text-align: left;text-indent: 34px;color: rgb(0, 0, 0);box-sizing: border-box !important;overflow-wrap: break-word !important;"><img class="rich_pages" data-ratio="1" data-s="300,640" src="https://img.haomeiwen.com/i3104601/1669996a56339638" data-type="jpeg" data-w="258" style=""><span style="font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;text-align: center;background-color: rgb(255, 0, 0);"> 人间值得你来

语音分离：Wave_U_Net用于音频源分离的实验

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读