sevagh/free-music-demixer

free website for client-side music demixing with Demucs + WebAssembly

free-music-demixer

A free static website for client-side music demixing (aka music source separation) with the Demucs v4 hybrid transformer AI model. It runs on CloudFlare Pages.

The paid PRO site contains higher-quality custom ensemble models.

demucs.cpp: transliterated the original PyTorch model Python code to C++ with Eigen3, compiled to WebAssembly with Emscripten. No quantization: the weights of Demucs v4 htdemucs and htdemucs_6s are 81 MB and 53 MB respectively, stored as float16. Anything smaller affects the quality of the network, and compression only gets down to ~70 MB: not worth the extra loading time.

Dev instructions

The wav-file-encoder project has been vendored in; I manually compiled the Typescript file to Javascript with these commands:

npm install typescript
npx tsc --module es6 ../vendor/wav-file-encoder/src/WavFileEncoder.ts

Demucs v4

Fewer memory issues from segmented design (largest track tested is ~7 minutes, 'Georgia Wonder - Siren').

'Georgia Wonder - Siren' (takes ~41 minutes):

vocals          ==> SDR:   7.261  SIR:  13.550  ISR:  13.158  SAR:   6.763
drums           ==> SDR:  10.629  SIR:  17.819  ISR:  17.373  SAR:  10.829
bass            ==> SDR:  10.593  SIR:  19.696  ISR:  12.244  SAR:  10.007
other           ==> SDR:   6.324  SIR:   9.005  ISR:  13.223  SAR:   6.067

'Georgia Wonder - Siren', ~9 minutes with 8 workers:

vocals          ==> SDR:   7.181  SIR:  14.328  ISR:  12.764  SAR:   6.563
drums           ==> SDR:  10.695  SIR:  17.893  ISR:  17.335  SAR:  10.816
bass            ==> SDR:  10.588  SIR:  19.700  ISR:  12.098  SAR:   9.927
other           ==> SDR:   6.238  SIR:   8.757  ISR:  13.886  SAR:   6.144

'Zeno - Signs' (takes ~20 minutes):

vocals          ==> SDR:   8.326  SIR:  18.257  ISR:  15.927  SAR:   8.311
drums           ==> SDR:  10.041  SIR:  18.413  ISR:  17.054  SAR:  10.692
bass            ==> SDR:   3.893  SIR:  12.221  ISR:   7.076  SAR:   3.237
other           ==> SDR:   7.432  SIR:  11.422  ISR:  14.161  SAR:   8.201

'Zeno - Signs', ~5 minutes with 8 workers, 0.75s overlap + weighted sum (same as demucs segmenting):

vocals          ==> SDR:   8.297  SIR:  18.114  ISR:  15.731  SAR:   8.335
drums           ==> SDR:  10.007  SIR:  18.465  ISR:  16.983  SAR:  10.665
bass            ==> SDR:   4.054  SIR:  12.487  ISR:   6.728  SAR:   3.035
other           ==> SDR:   7.349  SIR:  11.118  ISR:  14.236  SAR:   8.159

(best sound to my ears/least segment boundaries, anecdotally)

free-music-demixer's Issues

Model producing garbage outputs?

For some reason, the only result I'm getting (at least for the file I've tried) is just garbage noise. I'm not sure if there's anything I'm doing wrong in terms of the format I'm uploading (I've tried 24-bit FLAC and 16-bit WAV, both to no avail) or whether it's just a song that doesn't work well, but something clearly isn't working right for this song.

If there are any other details I can provide that might help with debugging this, please let me know.

Out of memory on website issue

Hello,

I've been attempting to try this online tool in order to see how capable it is to demix some of my favourite tracks.
So, I've used an 8.8 MB .mp3 file as input, and the website seems not able to process the result.

I've noticed the dev log keeps stuck on this line:

[WASM/C++ 17:28:32] Getting waveforms from istft
[WASM/C++ 17:28:34] Copying waveforms

And if I take a look at JS console, this message is printed:

Uncaught (in promise) DOMException: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': Data cannot be cloned, out of memory.
    at https://freemusicdemixer.com/worker.js:25:13

I think I'll have to try running this locally without a web interface so that it can get enough memory to work properly.
Just wanted to let you know, I think UI should handle this error to avoid keep user waiting in vain.

CompileError: WebAssembly.instantiateStreaming(): Compiling function #21 failed: Wasm SIMD unsupported @+3930

Dear @sevagh ,
I wanted to "downlad weights" but it was aborted with the error message in javascript console:

CompileError: WebAssembly.instantiateStreaming(): Compiling function #21 failed: Wasm SIMD unsupported @+3930

I don't understand much of WASM. Could you please help?

My System: Firefox or Chromium
Ubuntu 22.04.3 LTS
AMD® Phenom(tm) ii x4 955 processor × 4

Separate guitar tracks?

First of all, thanks for this! I've been doing tabs for many years, and learned different ways to trying to separate tracks as best as possible, but this takes it to another level.

It would be nice to have the melody potentially separated by different guitar/keyboards/etc tracks. I hope this can be achieved eventually.

Separate vocals mixed

Hi,i have a old capture record from TV in the 90, was record in a one of the first dual audio vcr of the epoque, so they save in the tape the original track in v.o+dubbed track. Unfortunly these vcr uses a fine separation line in the tape to separate the audio Channels, and the tape get this line deteriorated over the years, so today the vcr can't separate the audio and read the the two audio tracks at the same time, resulting in a audio track with the v.o+dubbed track playing at the same time.

My questions is if with the help of a Ia can be possible isolate and separate these audio voice tracks to their normal state.

Out of memory

3080 GPU

  0%|                                                                                                                                                                                         | 0/1 [00:06<?, ?it/s]
Traceback (most recent call last):
  File "/home/user/.virtualenvs/demixer/bin/umx", line 8, in <module>
    sys.exit(separate())
             ^^^^^^^^^^
  File "/free-music-demixer/vendor/open-unmix-pytorch/openunmix/cli.py", line 167, in separate
    estimates = predict.separate(
                ^^^^^^^^^^^^^^^^^
  File "/free-music-demixer/vendor/open-unmix-pytorch/openunmix/predict.py", line 78, in separate
    estimates = separator(audio)
                ^^^^^^^^^^^^^^^^
  File "/home/user/.virtualenvs/demixer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/free-music-demixer/vendor/open-unmix-pytorch/openunmix/model.py", line 314, in forward
    targets_stft = targets_stft.permute(0, 5, 3, 2, 1, 4).contiguous()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.56 GiB (GPU 0; 9.77 GiB total capacity; 5.71 GiB already allocated; 1.37 GiB free; 6.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How to run it?

Is there a compiled version somewhere?

Errors in Safari 17.2.1 and Firefox 17.2.1 (developer edition)

running macOS 14.2.1 and a MacBook Air M1

Unhandled Promise Rejection: RuntimeError: Out of bounds memory access (evaluating '(_free=Module["_free"]=Module["asm"]["x"]).apply(null,arguments)')

This is from Safari, FF shows similar errors. File is not processed and no progress bar shows up. No errors in dev logs:

[Javascript 21:09:14] Fetched and cached model files
[Javascript 21:09:14] Selected model: demucs-6s
[Javascript 21:09:40] Initializing 4 workers!
[Javascript 21:09:41] Beginning demix job
[Javascript 21:10:35] Worker 2 is ready!
[Javascript 21:10:35] Worker 3 is ready!
[Javascript 21:10:35] Worker 0 is ready!
[Javascript 21:10:35] Worker 1 is ready!