Unable to reproduce same error metrics as claimed in the paper

[bhavyaghai]

Unable to reproduce same error metrics as claimed in the paper

[dr-costas]

Hi,

Regarding your first message (i.e. before editing), if you want to re-train the model and follow the presented process at the paper that you mention, you should also use the same dataset mentioned in the same paper (i.e. DSD plus MedleyDB stems).

I have some questions though about your presented results. In the original mail that I got for the notification (also can be seen by your message edits) you mention that you retrained the model and you got

Median SDR: 4.04 dB | Median SIR: 7.14 dB

now you are mentioning that you used our saved weights and you got the exact (i.e. as your original message, re-training with only DSD100 dataset) numbers. I mean, you first mention that you re-trained the model with a different dataset and got X numbers and then you mention that you use our pre-trained weights and you get the same X numbers. Can you please clarify what is happening here?

As far as I remember, we tested the saved weights in two different systems, in different countries, with different O.S.'s. There might be the case that there is something happening at your setup (or the process that you follow) that you got it wrong from the paper. If you clarify, then it would be a help for me, to help you.

For your second question, if you get the weights from the 100th epoch and then you train more, you do not re-training the model but you training it more. Also, as I already mentioned above, there is the case that you used different dataset, i.e. only the DSD100 without the MedleyDB stems.

Finally, are you using the exact code from this repo? I mean, you did not used any other code, e.g. for the evaluation part, right?

[dr-costas]

Also, I can confirm that we tested again our pre-trained weights and we get the exact same values as those reported at the paper.

[bhavyaghai]

Thanks for your response. Please discard my unedited comments. For clarification, I haven't modified any code in evaluation or any hyperparameters. As previously reported, I ran two scenarios. In the first case, I tested the model on pretrained weights (linked in this repo). In the second case, I trained the model from scratch and tested the model based on weights from 100th epoch. Error metrics which I have reported are specific to DSD 100 dataset. As you pointed out, I should add medleyDB together with DSD100 and then test the model again. I hope I will get the same error metrics as the paper.

[dr-costas]

Still, please note that we tested again today our preteained weights and we got the same numbers as in the paper.

If you are using the preteained weights and you are getting different scores, then either you have different test dataset than ours or different evaluation scheme/functions.

[Js-Mim]

Hi,

i was able to produce the following error scores using the exact conda/pip requirements that this repository provides:

For python 3.5 i tried both GPU and CPU modes. They provide exactly the same results as the second attached image. Please take into account, in the paper the non-bleeding/non-instrumental stems of MedleydB are used to enrich the training subset of DSD100. The testing subset is left as it is. The reported results are from the 100th epoch. Further training degrades slightly the objective performance.

We could assist a bit more if you could please provide a list of your python environment dependencies and a screenshot/log of the testing routine.

[dr-costas]

Hi,

@bhavyaghai
Is everything OK? Did you found your error?

[bhavyaghai]

Hi @dr-costas ,
Thanks for reaching out. I guess adding medleyDB to DSD100 will do the trick. I have submitted my request to download medleyDB dataset and awaiting their response. As soon as I get the access, I will test this model again and hopefully get better results.

[dr-costas]

Hi,

@bhavyaghai I fail to understand what are you saying.

If you want to test the model, then you need only the pre-trained weights and the testing dataset. The pre-trained weights are in Zenodo and the testing dataset is from the DSD100. We uploaded (@Js-Mim did) a screen shot from testing the model and the results are exactly as reported.

If you want to train/re-train the model, then you need only the dataset (DSD100 + MedleyDB).

What exactly is that you want?