Wonder the training time

[BUAADreamer]

Wonder the training time

[Cuberick-Orion]

For the filtering stage, on CIRR with one A100 80G, the model takes 4 minutes to train an epoch (incl. per-epoch validation), in total it takes about 2-3 hours till its peak performance. Likewise for Fashion-IQ.

[Cuberick-Orion]

Hi,

To answer your question, on one A100 80G, when training with the CIRR dataset, the candidate re-ranking stage takes around 40 minutes per epoch, similar for training on Fashion-IQ.

In total we trained for 80 epochs on CIRR (50 on Fashion-IQ) until it peaks so the entire training phase took a bit over 2 days for CIRR (around 1 day for Fashion-IQ).

Of course, in development we usually don't have to train till peak to gauge its performance so if you are keen on a faster turn-around for development just examine the curve upfront and decide if it's worth your time to keep going.

Here is how it goes for CIRR (we extend the epochs to 100 just to verify its peak at ~80):

[BUAADreamer]

Oh I see. But this is the reranker training time?
I want to know the time of dual encoder training time(first stage), thanks!!

[BUAADreamer]

Thanks!!