Fail to reproduce result

[inhwanbae]

Fail to reproduce result

[InhwanBae]

Hi @Pranav-chib,

I'll share the results of my tests from the past week and some interesting findings with you. First, I was able to reproduce the table in the paper using the GitHub code I uploaded. What's interesting is that using the latest PyTorch release significantly reduces both the training speed and final performance. It even caused errors that interrupted the training. I recommend using version 1.6 or an older version as much as possible. (up to 1.11 seems to converge well)

Secondly, I have good news for you. I've made some updates to the source code a bit. Using the new codebase, you should be able to get better results!

[InhwanBae]

Thanks for your interest in my work! In my experience, when training on the ETH set, early stopping between 16 and 32 epochs significantly improves performance.

Even so, FDE 1.4 is a value I've never seen before. Actually, I have not run training since early 2021 and only evaluated with pre-trained weights. I will check it out with the published code and let you know.

[LOCKE98]

Hello, esteemed author. I have recently embarked upon replicating your code and encountered similar circumstances. Instead of creating a separate conda environment based on the setup you provided, I attempted to reproduce it using my existing environment. Now, I am facing a few issues. Firstly, when training on five datasets simultaneously with a single GPU, the training process is intermittently interrupted and does not reach the designated number of epochs. Additionally, there is a significant deviation in performance metrics. I am now prepared to attempt the replication using your recommended configuration.

[InhwanBae]

Hi @LOCKE98, Changing or commenting out the seed may help solve the interruptions. Since I also ran 40 model training on an 8GPU server, running multiple experiments on a single GPU might not be the cause of the interruption.