Can I train a model with SBR to detect any dataset?

[d-x-y]

Can I train a model with SBR to detect any dataset?

[D-X-Y]

1, We failed to train one model to work fine for many other datasets.

2, SBR could help in this case, but we found that SBR is sensitive to many aspects, such as hyper-parameters and resolution. It would cost much time to tune it in another dataset/model.

3, Since different videos may have different LK iterations to converge, it is not that straightforward to implement it by batch.

[Jar7]

Thank you for your reply!

1. Could you please confirm that you failed to train the model by the way you mentioned in the following issue?

You need to retrain the model on another video. If you want to use the trained model on any video, it requires to train this model on a very large number of videos.

Originally posted by @D-X-Y in facebookresearch/supervision-by-registration#40 (comment)

2. Could you please share any experience or suggestion on the initialization of the hyper-parameters ? How to decide the value? The training process is very slow, after one epoch the lk loss is almost the same (0.4-0.7), is it normal?

[D-X-Y]

Hi @Jar7 ,
1, Yes, training the model with SBR on the target video is necessary to get a visually good effect.

2, It is hard to give a precise hyper-parameter for me right now. In my experience, I would debug to see the intermedia LK results on the training data, and see whether these hyper-parameters cause many failure cases or not.

[Jar7]

Hi @Jar7 ,
1, Yes, training the model with SBR on the target video is necessary to get a visually good effect.

2, It is hard to give a precise hyper-parameter for me right now. In my experience, I would debug to see the intermedia LK results on the training data, and see whether these hyper-parameters cause many failure cases or not.

Thank you!

[Jar7]

Hi @D-X-Y,
Excuse me, May I have further questions..

I found that the pretrained model cpm_vgg16-epoch-049-050.pth output stable landmarks for demo-sbr.mp4. Has the demo-sbr,mp4 been trained? How many epoch should be trained for each video?

Have you found why a model for any video cannot be trained? Is it because the lk operation is sensitive to the video? If the lkloss is changed to another opt-flow loss, whether another additive loss would help to stabilize the output of backbone?