Any data on expected training time? Also, Are there plans for realtime visualized output?

[aliaksandrsiarohin]

Any data on expected training time? Also, Are there plans for realtime visualized output?

[AliaksandrSiarohin]

Hi, thank you for your interest,
I usually train on 2 Titan X with 12GB of memory. It take 2 days for dataset like TaiChi and VoxCeleb. For simple dataset like Nemo, training can be done in 24 hours. With 1 gpu number should be multiplied by 2.

The problem with visualization is that it take a lot of time to compute. If you need more frequent visualizations you can change the parameter num_repeats

first-order-model/config/taichi-256.yaml

Line 93 in 720bdd8

num_repeats: 150

. Basically the number of passes over all the videos is num_epochs * num_repeats. You can decrease number of repeats and increase number of epochs. Because visualization is drawn after each epoch, you will get more frequent visualizations at a cost of some efficiency.

Regards, AS

[ExponentialML]

Hi, thank you for your interest,
I usually train on 2 Titan X with 12GB of memory. It take 2 days for dataset like TaiChi and VoxCeleb. For simple dataset like Nemo, training can be done in 24 hours. With 1 gpu number should be multiplied by 2.

The problem with visualization is that it take a lot of time to compute. If you need more frequent visualizations you can change the parameter num_repeats

first-order-model/config/taichi-256.yaml

Line 93 in 720bdd8

num_repeats: 150

. Basically the number of passes over all the videos is num_epochs * num_repeats. You can decrease number of repeats and increase number of epochs. Because visualization is drawn after each epoch, you will get more frequent visualizations at a cost of some efficiency.

Regards, AS

Excellent. You rock!

[kartikJ-9]

Can i train this on colab. How many videos I should keep for train and test? I want the output quality to be 256/512 resolution. I can see that the fashion dataset had 500 videos for train and 100 for test. My dataset is also similar to fashion with a person in the frame. So will I be able to train the model in colab if I use 500,100 for train,test? Please help me with this. Thanks

[AliaksandrSiarohin]

Most likely not possible, colab gpus is rather slow. And it will shut down and kick you out after 24 hours.

[MentalGear]

@kartikJ-9 Have you made progress with training a higher res model on colab or elsewhere? Would appreciate if you could share what you tried.