richzhang/colorization-pytorch

PyTorch reimplementation of Interactive Deep Colorization

colorization-pytorch's Issues

Hi,In your paper the generator loss function is huber loss but in your pytorch version, the loss function is L1 loss

So the L1 loss function more suit for Colorization task?

Errors when running CPU only

Im trying to run using CPU only in a VM. I keep getting the following error during the first part of the batch script:

`[Network G] Total number of parameters : 34.187 M

No handlers could be found for logger "visdom"
create web directory ./checkpoints/siggraph_class_small/web...
Traceback (most recent call last):
File "train.py", line 61, in
model.optimize_parameters()
File "/home/testing/Desktop/colorization-pytorch-master/models/pix2pix_model.py", line 193, in optimize_parameters
self.forward()
File "/home/testing/Desktop/colorization-pytorch-master/models/pix2pix_model.py", line 123, in forward
self.fake_B_dec_max = self.netG.module.upsample4(util.decode_max_ab(self.fake_B_class, self.opt))
File "/home/testing/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 576, in getattr
type(self).name, name))
AttributeError: 'SIGGRAPHGenerator' object has no attribute 'module'
mkdir: cannot create directory ‘./checkpoints/siggraph_class’: File exists
cp: cannot stat './checkpoints/siggraph_class_small/latest_net_G.pth': No such file or directory
`

test.py is not compatible with python 3

Hi, I am trying to test your model in google colab. I put a dummy image a dummy folder:
'/content/colorization-pytorch/dataset/ilsvrc2012/val/MyImages'

I tested these two following commands:
#!python test.py --name siggraph_caffemodel --mask_cent 0
!python test.py --name siggraph_retrained

Both led to the same error:

Traceback (most recent call last):
File "test.py", line 53, in
data_raw[0] = util.crop_mult(data_raw[0], mult=8)
File "/content/colorization-pytorch/util/util.py", line 277, in crop_mult
return data[:,:,h:h+Hnew,w:w+Wnew]
TypeError: slice indices must be integers or None or have an index method

I fixed it by casting indices to integers. Then I got another error in test.py line 57:
img_path = [string.replace('%08d_%.3f' % (i, sample_p), '.', 'p')]

string.replace is deprecated in python 2.7, and does not exists in python 3, see here: https://docs.python.org/2.7/library/string.html?highlight=string%20replace#string.replace

However, when I change my google colab execution to python 2, test.py works like a charm.

num_threads or nThreads?

I've noticed that the base options for this repository uses opt.nThreads, but I can't actually see anywhere this is used.

I've also done some digging and in the data folder init.py the CustomDatasetLoader calls opt.num_threads, which I believe is what the pix2pix repo uses in their base options, although I don't think this is used either in this project?

Is the data loading for this project actually multi-threaded?
In train.py torch.utils.data.DataLoader(dataset, batch_size=opt.batch_size, shuffle=True) is called, should I change this to include num_workers=int(opt.nThreads) if I want to speed up the data loading?

Thanks for the amazing code/project! :)

How to understand the code in utils about encode_ab_ind ...function?

This is little comment about some functions in utils. Can you explain the ab transformation in detail? Thank you very much.

Convert into ONNX

How can I export ONNX model from latest_net_G.pth?

decode_ind_ab

in utils decode_ind_ab(), the calculations is

    data_a = data_q/opt.A
    data_b = data_q - data_a*opt.A
    data_ab = torch.cat((data_a, data_b), dim=1)

however I believe according to how the encoding was done we should have instead something like
data_a = (data_q - data_b)/opt.A

I'm imagining this would have to be solved using linear programming or something.
I was just wondering if this is something you're aware of and whether I am missing something?

My issue is that when I use decode_ind_ab currently all my b values come through as -1 as
with

    data_a = data_q/opt.A (eq1)
    data_b = data_q - data_a*opt.A (eq2)

we can sub eq1 into eq2 to show that

data_b = data_q - data_q = 0

which then gets scaled and shifted to -1 before being returned.

Subsampling Operation

In subsampling why don't use convolution to do it?

Undefined name 'visuals' in ./util/util.py

flake8 testing of https://github.com/richzhang/colorization-pytorch on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./util/util.py:70:18: F821 undefined name 'visuals'
        subset = visuals
                 ^
1     F821 undefined name 'visuals'
1

Classification output in SIGGRAPHGenerator, #channel=529 ?

Hi,

I'm reading the implementation detail of the SIGGRAPHGenerator, but had some trouble understanding the classification output of this network.
Classification output in SIGGRAPHGenerator:

colorization-pytorch/models/networks.py

Line 315 in 9fd9bd8

model_class=[nn.Conv2d(256, 529, kernel_size=1, padding=0, dilation=1, stride=1, bias=use_bias),]

Is 529 the number of quantized color Q? While I suppose this number to be 313 instead...
Could anyone please explain this number?

Thanks,

Why in your code has GANLoss,? Your Paper has nor GAN nerwork

In you paper , no information for gan ,why in your code have gan? what is the use in your code

I do the four steps following the train_siggraph.sh, but didn't get the same prefect result as caffemodel.pth.

I do not modify any parameters, is there any problem? or i should change some train parameters? Hope you can help me ,thanks!

Custom Dataset

The model is a train on ILSVRC2012. Could someone please help me out in clarifying how to train it on my custom dataset?

Interactive colorization does not work for retrained model.

Hi, thanks for your great work. I tried the interactive colorization GUI from https://github.com/junyanz/interactive-deep-colorization with your provided retrained model, but it doesn't work. The screen shot is as following:

I also tried the caffe model, it works well.

Global hints network

Hello, Mr Zhang. I've read this source code, but I can't find the global hint module. In the pix2pix_model.py file, the forward module is as following:

def forward(self):
        (self.fake_B_class, self.fake_B_reg) = self.netG(self.real_A, self.hint_B, self.mask_B)
        self.fake_B_dec_max = self.netG.module.upsample4(util.decode_max_ab(self.fake_B_class, self.opt))
        self.fake_B_distr = self.netG.module.softmax(self.fake_B_class)

        self.fake_B_dec_mean = self.netG.module.upsample4(util.decode_mean(self.fake_B_distr, self.opt))

        self.fake_B_entr = self.netG.module.upsample4(-torch.sum(self.fake_B_distr * torch.log(self.fake_B_distr + 1.e-10), dim=1, keepdim=True))
        # embed()

The local hint module is applied, but I can't find the global hints module, and the ground truth of color distribution is also not calculated. Maybe I miss some important things, can you provide me some information? Thank you!

Training tends to yellowish colors

Hi,
Thanks for this Pytorch implementation. I'm following your exact tutorial to train the model on my Mac machine CPU, the problem is that as I monitor the visdom console, I see the output fake_reg image goes more yellowish as the losses curve goes down. I've had this same exact issue while training the original colorization model (i.e. the one without hints) introduced by the paper authors. I don't know why this actually happens, any ideas?

Thank you!

I found the error in util.py

In the line number 248, average patch values are not appropriately computed.
in utils add_color_patches_rand_gt() (line number 248), the calculations is
torch.mean(torch.mean(data['B'][nn,:,h:h+P,w:w+P],dim=2,keepdim=True),dim=1,keepdim=True).view(1,C,1,1)
however, I believe that this code should be changed to
torch.mean(torch.mean(data['B'][nn,:,h:h+P,w:w+P],dim=2,keepdim=True),dim=-1,keepdim=True).view(1,C,1,1)

The previous code calculates the mean value within the ab channel, while the second calculates within the patch values.

Discriminator in the pretrained model

So I can see that there is a discriminator defined in the graph, if lambda_ GAN > 0. Do you guys have pretrained models that use the discriminator? I am curious about the difference in performance with and without discriminator?
Or what are hyperparameters if providing a pretrained model is not possible?

Where are the local and global hints networks defined?

I cannot find a reference to either the local or global hints networks anywhere in the source code. I've had a look at the definition of the SIGGRAPHGenerator class, but that seems to be just the main colorization network. I would be very grateful for pointing me to the relevant lines. Thank you.

How can I use the pretrained code for example-based colorization

In your paper you say using the Global Hints Network, you can colorize a grayscale image using global histograms from an input color image. I want to know how to implement this using the pretrained model. Thank you.

Mac running test.py error

I tried to running the test.py but got this error. Any thoughts?

Traceback (most recent call last):
  File "/Users/spikeyuan/PycharmProjects/pythonProject6/colorization-pytorch/test.py", line 61, in <module>
    model.test(True)  # True means that losses will be computed
  File "/Users/spikeyuan/PycharmProjects/pythonProject6/colorization-pytorch/models/base_model.py", line 56, in test
    self.forward()
  File "/Users/spikeyuan/PycharmProjects/pythonProject6/colorization-pytorch/models/pix2pix_model.py", line 123, in forward
    self.fake_B_dec_max = self.netG.module.upsample4(util.decode_max_ab(self.fake_B_class, self.opt))
  File "/opt/anaconda3/envs/pythonProject6/lib/python2.7/site-packages/torch/nn/modules/module.py", line 576, in __getattr__
    type(self).__name__, name))
AttributeError: 'SIGGRAPHGenerator' object has no attribute 'module'

I run this project on Colab,Then I encounter a problem

The error information is No such file or directory: './dataset/ilsvrc2012/val/1/ILSVRC2012_val_00000059.JPEG',But I can find this image in the directory,Your prompt attention to my question is appreciated,Thanks for your help!!

'SIGGRAPHGenerator' object has no attribute 'model'

when running on MAC（python3.5.4）,I encountered the following problem:'SIGGRAPHGenerator' object has no attribute 'model'
I wonder how to fix it?
Traceback (most recent call last):
File "test.py", line 40, in
model.setup(opt)
File "/Users/joanna/Desktop/colorization/colorization-pytorch-master/models/base_model.py", line 42, in setup
self.load_networks(opt.which_epoch)
File "/Users/joanna/Desktop/colorization/colorization-pytorch-master/models/base_model.py", line 136, in load_networks
self.__patch_instance_norm_state_dict(state_dict, net, key.split('.'))
File "/Users/joanna/Desktop/colorization/colorization-pytorch-master/models/base_model.py", line 116, in __patch_instance_norm_state_dict
self.__patch_instance_norm_state_dict(state_dict, getattr(module, key), keys, i + 1)
File "/Users/joanna/opt/anaconda3/envs/color/lib/python3.5/site-packages/torch/nn/modules/module.py", line 576, in getattr
type(self).name, name))
AttributeError: 'SIGGRAPHGenerator' object has no attribute 'model'

Model training Loss graph

Plot.ly Link: https://plot.ly/~Ugness/1/

I am training the model with ILSVRC2012 training set and same options as your implementation and my loss graph looks like above.
I am afraid that my model's loss reduces correctly. Can you check this loss graph or share your loss graph?
Thanks.