Best way to fine-tune the model

[xuebinqin]

Best way to fine-tune the model

I am also curious about the layers to freeze, and I thought about some ways, like

1. freezing the blocks with larger inputs, like En1 and En2,
2. freezing all encoders
3. freezing the first few Conv+BN+RELU layers for each block
4. 2 plus 3
5. 1 plus 3

Any idea about which is better for U-2-Net?
@Nathanua @PytaichukBohdan

[PytaichukBohdan]

I am also curious about the layers to freeze, and I thought about some ways, like

1. freezing the blocks with larger inputs, like En1 and En2,
2. freezing all encoders
3. freezing the first few Conv+BN+RELU layers for each block
4. 2 plus 3
5. 1 plus 3

Any idea about which is better for U-2-Net?
@Nathanua @PytaichukBohdan

I've tried pretty much the same :)
But the results became blurry and hasn't come back to the original state.

Have you managed to get any progress?

[pietz]

I wasn't successful finetuning the smaller model on 200 images. It overfitted before delivering meaningful results.

Something that worked for me was to add my dataset to the DUTS TR dataset and train on that. My approach was as follows:

1. Validate that my pipeline and the pretrained model lead to similar results on DUTS TE as reported in the paper.
2. Validate that training for a few epochs using the DUTS TR data still leads to similar performance on DUTS TE.
3. Train on DUTS TR + my dataset while watching the the performance on both DUTS TE and my own validation set.

That way you can be sure that the model improves on your task while still generalising to the DUTS dataset. All of this is of course only recommended if your task can be seen as some form of salient segmentation. Otherwise this will not work properly.

[PytaichukBohdan]

@pietz for how long did you train the model to produce results?

I tried combining my own dataset (200 or 3000 images of people) with DUTS TR. But after first epochs the result became worse and worse

[pietz]

@pietz for how long did you train the model to produce results?

I tried combining my own dataset (200 or 3000 images of people) with DUTS TR. But after first epochs the result became worse and worse

I used the smaller pretrained model, set the learning rate to 0.0001, increased batch size to 32 and trained for anywhere between 1 and 50 epochs. The results say roughly the same. Make sure your transformations match the original code. I used PyTorch transformations only instead of the custom functions in this repo.

[SomeoneDaniel]

I wasn't successful finetuning the smaller model on 200 images. It overfitted before delivering meaningful results.

Something that worked for me was to add my dataset to the DUTS TR dataset and train on that. My approach was as follows:

1. Validate that my pipeline and the pretrained model lead to similar results on DUTS TE as reported in the paper.
2. Validate that training for a few epochs using the DUTS TR data still leads to similar performance on DUTS TE.
3. Train on DUTS TR + my dataset while watching the the performance on both DUTS TE and my own validation set.

That way you can be sure that the model improves on your task while still generalising to the DUTS dataset. All of this is of course only recommended if your task can be seen as some form of salient segmentation. Otherwise this will not work properly.

Great mind think alike. I am doing the same things as yours. However, when I validating the training(your second step) I've got MAE: 0.067 on DUTS TE for the model u2net instead of 0.044 from paper. I am not sure how the gap occurs. Any ideas?

[pietz]

I wasn't successful finetuning the smaller model on 200 images. It overfitted before delivering meaningful results.
Something that worked for me was to add my dataset to the DUTS TR dataset and train on that. My approach was as follows:

1. Validate that my pipeline and the pretrained model lead to similar results on DUTS TE as reported in the paper.
2. Validate that training for a few epochs using the DUTS TR data still leads to similar performance on DUTS TE.
3. Train on DUTS TR + my dataset while watching the the performance on both DUTS TE and my own validation set.

That way you can be sure that the model improves on your task while still generalising to the DUTS dataset. All of this is of course only recommended if your task can be seen as some form of salient segmentation. Otherwise this will not work properly.

Great mind think alike. I am doing the same things as yours. However, when I validating the training(your second step) I've got MAE: 0.067 on DUTS TE for the model u2net instead of 0.044 from paper. I am not sure how the gap occurs. Any ideas?

Does the gap appear after training or aren't you able reproduce the results using the pretrained model?

If it's the latter there is something wrong with your code. I carefully reengineered the official code until I arrived at 0.054. I'm only using the smaller model.

If you see a gap after training, I'd try to start with an extremely small learning rate. Maybe training the larger model is less stable. SGD might be a better option than Adam for fine-tuning.