Problem with channel_pruning and other models

[progamergov]

Problem with channel_pruning and other models

[JaledMC]

I'm suscribed to your repo, and it always amazes me your dedication. Keep this fantastic work!

[ProGamerGov]

@JaledMC I finally figured out how to implement -normalize_gradients and I've added it to the master branch!

cbcd023

[ProGamerGov]

@JaledMC You can get somewhat similar results to neural-style's -normalize_gradients parameter, if you set the content weight value to 0 with: -content_weight 0.

[ProGamerGov]

@JaledMC For some reason my code is not detecting the correct model definition. I have these lines in CaffeLoader.py: https://github.com/ProGamerGov/neural-style-pt/blob/master/CaffeLoader.py#L172-L173, where I try to detect the model based on the name it has. I had to remake my own version of Loadcaffe from Torch7, and because PyTorch models lack a prototxt file, my code determines what model it is by the model's name.

When I run:

python neural_style.py -backend cudnn -model_file models/channel_pruning.pth

It seems to work properly when I run it on Ubuntu or Windows, so could there be an issue with bash? But I would have thought argparse would convert the parameter input to a string. What Python version were you using? And did you make sure that you had the most up to date version of neural-style-pt?

[JaledMC]

Thanks! For some reason, it was searching NIN keys, no matter the input_model. Anyway, I updated your repo as you said, and it works, both python 3.5 and 3.6.9.

PD: I have tried to add -normalize_gradient, but at this moment, the output images don't change much with the style_weight.

100 iterations with style_weight 3000

200 iterations with style_weight 3000

If I find any solution, will make a pull request.

[ProGamerGov]

@JaledMC The original neural-style has gradient normalization in the backward pass, while your code had gradient normalization in the forward pass.

Justin Johnson's code also does the total variance denoising in the backward pass as well. That's why neural-style-pt's total variance denoising is different than neural-style's total variance denoising, because I do it in the forward pass.

[JaledMC]

You are totally right, and the code differences are clear. I only gave attention to the Gram implementation and didn't notice it. My fault.

Thank you so much. Because there is no problem, I'm going to close this issue.

[ProGamerGov]

@JaledMC It may be possible to use backward hooks to make gradient normalization possible, but I was never able to get it working. Though I was also a lot worse at programming at the time I tried to get it working.

[ProGamerGov]

So, this is what one would think you would use to replicate neural-style's -gradient_normalization feature in PyTorch, as it's literally the exact same functions:

tensor.div(torch.norm(tensor, 1) + 1e-8)

Torch7's torch.norm(), looks similar to PyTorch's Torch.norm, but I think there may be differences: https://pytorch.org/docs/stable/_modules/torch/functional.html#norm

It also looks like -normalize_gradients from neural-style, may not actually be gradient normalization, and instead a form of gradient scaling: jcjohnson/neural-style#374. Though @jcjohnson seems to have thought it was gradient normalization when he implemented it: jcjohnson/neural-style@0c5d5d5

[JaledMC]

@ProGamerGov, excuse my delay. I didn't see your replies. My fault. Pytorch in my conda enviroment broke down, and resist to load caffe models. I fixed it today, and new runs can be done.

As you said, I used tensor.div(torch.norm(tensor, 1) + 1e-8), and didn't work. Some changes to mimic official repo only gave me errors :/ But I am going to try -content_weight 0.

As always, thanks for your amazing work