sheldontsui/minus-plus-network

this repo is the codebase for ICCV19 paper "Recursive Visual Sound Separation Using Minus-Plus Net"

Recursive Visual Sound Separation Using Minus-Plus Net

Xudong Xu,Bo Dai, Dahua Lin

We propose a recursive Minus-Plus Network(MP-Net) for visual sound separation task. MP-Net separates sounds recursively in the order of average energy, removing the separated sound from the mixture at the end of each prediction, until the mixture becomes empty or contains only noise.

[Project] [Paper] [Demo]

Requirements

• Python 3.6 is used. Basic requirements are listed in the 'requirements.txt'

pip install -r requirements.txt 

Dataset

MUSIC can be accessed here. Note that, Zhao. et al further proposed a new and larger dataset MUSIC21 in ICCV 2019.

VEGAS is a 10-category subset of the large-scale dataset AudioSet. VEGAS was cleaned by Zhou et al in CVPR 2018 paper Visual2Sound and can be fetched here.

Training and Testing

All the training and testing bash scripts can be found in './scripts'. For the data preparing, we follow the same procedure as PixelPlayer.

License and Citation

The usage of this software is under CC-BY-4.0.

@inproceedings{xu2019recursive,
  title={Recursive visual sound separation using minus-plus net},
  author={Xu, Xudong and Dai, Bo and Lin, Dahua},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={882--891},
  year={2019}
}

Acknowledgement

The structure of this codebase is largely borrowed from PixelPlayer.