xiefan-guo/ctsdg

[ICCV 2021] Image Inpainting via Conditional Texture and Structure Dual Generation

ctsdg's Issues

How was Corrupted Edge obtained?

I'm a beginner student and very interested in your paper. I don't understand one thing. How did the input of Structure Decoder in the generator, Corrupted Edge come from? There does not seem to be such a image in the training data set.

During training, an error occurred

Hello, author, I think your work is amazing, but when I started training, he made a mistake, I can't solve it temporarily, can you help me？ thank you!
Traceback (most recent call last):
File "E:/code/CTSDG-main/train.py", line 102, in
train(opts, image_data_loader, generator, discriminator, extractor, generator_optim, discriminator_optim, is_cuda)
File "E:\code\CTSDG-main\trainer.py", line 45, in train
output, projected_image, projected_edge = generator(input_image, torch.cat((input_edge, input_gray_image), dim=1), mask)
File "D:\Anaconda3\envs\torch18\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "E:\code\CTSDG-main\models\generator\generator.py", line 248, in forward
dc_texture = torch.cat((dc_texture, ec_textures[ec_texture_skip]), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 3 and 2 (The offending index is 0)

quantitative results on celeba

Dear authors, thank you for making this great work public.

I did not find quantitative results for celeba in your paper, such as ssim and psnr.And I want to know the results on celeba in your experiment.
I saw you post a pre-trained model about celeba. Can I calculate the indicator based on this model?

Hope to hear from you, thank you very much!

The question of discriminator adversarial loss

criteria/loss.py
loss_adversarial = criterion(real_pred, real_target) + criterion(fake_pred, fake_target) +
criterion(real_pred_edge, edge) + criterion(fake_pred_edge, edge)
I think the former three "criterion(real_pred, real_target), criterion(fake_pred, fake_target), criterion(real_pred_edge, edge)" are right, but the last one "criterion(fake_pred_edge, edge)", I think the second parameter is the "non-truth edge" instead of "the ground_truth edge".
I need your help to help me to understand the "criterion(fake_pred_edge, edge)".

图片的大小不匹配

作者你好，就是我在用这个代码的时候，我用我自己的数据集，为什么一直会报通道数不匹配的问题，这个问题该怎么解决呢？

Question about the way to obtain damage edge information during testing

Hi, I found that in your paper, the corrupted edge is also included as part of the model input during both testing and training process. And in the training and testing code you released, the corrupted edge is obtained by dot-multiplying the complete edge by the mask, where the complete edge is obtained from the complete image.

However, in practice or testing, we don't have complete images, so we are not able to get complete edges. And you are not pre-training a model for recontructing corrupted edges like Edge Connect does, which is the contribution you claimed in your paper.

As we know, the edge extracted from a corrupted image is not equal to the complete edge multiplied by the mask, the former will have less information in the vicinity of the damaged area.

I think there is something wrong with the way you are testing: replace extracting edges from corrupted images with multiplying the full edges by the mask, there is a leakage of information at the boundaries of corrupted regions. And I believe this leakage will help the improvement of inpainting performance. Can you explain it?

Solid work.

Thanks for your research.

Quantitative evaluation

Hi,
Thank you for your great works. I couldn't find a section for quantitative results. Would you provide me with the pipeline with the quantitative results? And are you using irregular masks for different masks or do you use your own masks?
Are they isolated for testing or they are included in the training time too?

关于批量测试修复实验的问题

作者您好，我在使用您的程序进行测试时发现，当我对多张图像进行修复时，result文件夹中修复后图像的顺序与我输入的待修复图像和掩码图像不能一一对应，请问应该如何调整才能使批量修复时获得有序的结果？期待您的回复！

I

受损边缘图和灰度图

您好，我想问一下这个由原始完整图像处理成为Corrupted Edge和Gray-scale Image这部分的内容具体体现在程序里边的哪一部分呢？

关于注意力cfa模块的问题

您好，我想请问一下关于注意力模块CFA的问题。在计算受损区域与已知区域的余弦相似性时，是如何区分受损区域与已知区域的？期待您的回答，万分感谢！

Error while testing

python test.py --pre_trained ./checkpoints/places2.pt --image_root ./images/ --mask_root ./masks/ --result_root ./result/ --number_eval 1

Sorry to bother you in this way.
When I tried to test this project following the instruction in markdown, this error shows up.

 Cuda is available
OrderedDict([('pre_trained', './checkpoints/places2.pt'),
             ('image_root', './images/'),
             ('mask_root', './masks/'),
             ('num_workers', 4),
             ('batch_size', 1),
             ('load_size', (256, 256)),
             ('sigma', 2.0),
             ('mode', 'test'),
             ('result_root', './result/'),
             ('number_eval', 1)])
start test...
  0%|                                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 64, in <module>
    output, __, __ = generator(input_image, torch.cat((input_edge, input_gray_image), dim=1), mask)
  File "/root/anaconda3/envs/CTSDG/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/CTSDG/models/generator/generator.py", line 97, in forward
    ec_textures['ec_t_1'], ec_textures['ec_t_masks_1'] = self.ec_texture_1(ec_textures['ec_t_0'], ec_textures['ec_t_masks_0'])
  File "/root/anaconda3/envs/CTSDG/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/CTSDG/models/generator/pconv.py", line 103, in forward
    images, masks = self.conv(images, masks)
  File "/root/anaconda3/envs/CTSDG/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/CTSDG/models/generator/pconv.py", line 50, in forward
    padding=self.padding, dilation=self.dilation, groups=1)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 9, 256, 256] to have 3 channels, but got 9 channels instead

And I have no clue how to fix it.

About the mask dataset size for training

What is the size range of this training dataset? 30%? 40%? still have

Code for dividing mask pictures.

Hello, is there a code to divide mask pictures according to mask ratio? Thank you.

关于代码问题

    projected_image = self.texture_feature_projection(dc_texture)
    projected_edge = self.structure_feature_projection(dc_structure)

我想请问一下，这部分代码是运行哪里的？

mask数据集通道数不匹配

作者你好，我跑你代码的时候，发现不管是训练还是测试模型的时候，mask_root都只能用你的那个irregular_mask数据集才不会报错，而当我用我自己的mask数据集时，就会出现

而当我把训练代码

image_in_channels=3改为9时，又会报出这个错误

@xiefan-guo 请问这个问题该怎么解决。

Code？

Code？

测试问题

你好，最近我用你所提供的预训练模型和我自己训练的模型进行评估，但是他们的结果都比论文中给出的指标好太多了，我能直接引用论文中的指标吗？

.

about experiment detail？

Thank you for your excellent work, I would really like to reproduce your model as a baseline for comparison, at your convenience can you tell us the number of training and fine-tuning iterations for each dataset?Thanks ！

关于在测试时获取损坏边缘信息的方式疑问

你好，我发现在你的论文中，无论是测试还是训练过程损坏的edge都是作为模型的输入的一部分。并且在你发布的训练和测试代码中，损坏的edge是通过将完整的edge点乘masks获得的，其中完整的edge是从完整的图像中获得的。

但是，在实际或测试中，我们并没有完整的图像，因此我们无法获得完整的edge。而且您并没有像 Edge Connect 那样预训练一个用于修复损坏的edge的模型，这也是您在论文中声称的贡献。

然而，从损坏图像中提取的edge并不等于从完整图像中提取出来的完整edge点乘mask，前者在损坏区域的附近会具有更少的信息。

我认为你对你的模型的测试方式有问题：通过将完整边缘乘以掩码替换从损坏的图像中提取边缘，在损坏区域的边界处存在信息泄漏。我相信这种泄漏将有利于修复性能。请问能解释一下吗？

Pre-trained models availability

Hello, when will the pre-trained models be available?

Who has run the finished model, based on the dataset Celeba

Dear author, please delete this question for me, thank you~

关于代码 CFA模块中第一部分RAL部分的疑问

作者你好，我在阅读您的代码的CFA模块RAL部分中有些地方没有看懂 比如代码中‘rate’的设置，以及‘background_kernel_size’设置为2*rate，而且这部分在论文中也没有提到 想问下这部分在哪有详细的叙述或者是参考了哪篇论文的实现

为什么在palace2训练很差

为什么在palace2数据集上训练的效果很差呀 psnr在15w次迭代的时候就到28点多，之后再怎么训练哪怕到50w次也还是这么多，不变了

测试指标

你好，最近我用您提供的预训练模型，还有我自己训练的模型在巴黎街景上进行了测试，我不知道是不是评价指标的版本问题，测得的结果要比论文中给出来的好太多，所以，您能提供一下您的测试指标代码嘛，以便更好的进行比较！！！急求！！！谢谢！

代码

运行中出现这个，请指点

Could you give me some little experience that why you choice **178** for the celebA dataset drop size?

Here I can see you add a comment.

https://github.com/Xiefan-Guo/CTSDG/blob/def02741ba1a110e70ee1a6365868e5d43731132/datasets/transform.py#L9

Maybe you have some special reason for that, cause other size will not generate nice result.

And that will be helpful for me.

Testing results are not as expected

@xiefan-guo Hey. Thanks for releasing the codebase.

I tested your pretrained model on celeba, however, the results are not as expected. Even the identity of the person belongs to the training set, still it does not generalize well. What could be the reason?