fnzhan/rabit

Bi-level feature alignment for versatile image translation and manipulation [ECCV 2022]

Bi-level feature alignment for versatile image translation and manipulation

Preparation

Clone the Synchronized-BatchNorm-PyTorch repository.

cd models/networks/
git clone https://github.com/vacancy/Synchronized-BatchNorm-PyTorch
cp -rf Synchronized-BatchNorm-PyTorch/sync_batchnorm .
cd ../../

VGG model for computing loss. Download from here, move it to models/.

Datasets

For the datasets for translation, please refer to CoCosNet.

For the datasets for image editing, you can download it from Google Drive.

Translation Results

Some prediction results of our model are provided in Google Drive.

Training

Then run the command

bash train_ade.sh

Citation

If you use this code for your research, please cite our papers.

@article{zhan2021rabit,
  title={Bi-level feature alignment for versatile image translation and manipulation},
  author={Zhan, Fangneng and Yu, Yingchen and Wu, Rongliang and Cui, Kaiwen and Xiao, Aoran and Lu, Shijian and Shao, Ling},
  journal={arXiv preprint arXiv:2107.03021},
  year={2021}
}

Acknowledgments

This code borrows heavily from CoCosNet. We also thank SPADE, Synchronized Normalization.

rabit's Issues

about the validation data split

Dear author,
Sorry, I'm a beginner in this field. To my knowledge, data will be splited into train, validation and test set, and we can calculate the FID score on validation data for model selection during training. However, I didn't find the validation set split and related code in the RABIT, as well as CoCosNet, which makes me confused. Thanks in advance.

about some bugs in you code

Hi zhan, I appreciate your work so much! However , I met some bugs when I tried to use your code.

1、models/networks/correspondence.py (line 385)

       elif self.opt.warp_mask_losstype == 'cycle':
            # f_div_C_v = F.softmax(f_WTA.transpose(1, 2), dim=-1)
            f_WTA_v = f.transpose(1, 2)
            f_div_C_v = f_WTA_v / f_WTA_v.sum(-1).view(-1, N, 1)

            seg = F.interpolate(seg_map, scale_factor=1 / self.down, mode='nearest')
            channel = seg.shape[1]
            seg = seg.view(batch_size, channel, -1)
            seg = seg.permute(0, 2, 1)
            warp_mask_to_ref = torch.matmul(f_div_C_v, seg)  # 2*1936*channel
            warp_mask = torch.matmul(f_div_C, warp_mask_to_ref)  # 2*1936*channel
            warp_mask = warp_mask.permute(0, 2, 1).contiguous()
            coor_out['warp_mask'] = warp_mask.view(batch_size, channel, feat_height, feat_width)  # 2*3*44*44

'f' and 'f_div_C' is not defined

2、models/networks/generator.py (line 47)

     self.conv_img1 = nn.Conv2d(1, seg, 3, padding=1)
     self.up = nn.Upsample(scale_factor=2)

'seg' is not defined

would you please fix these bugs?

about non-differentiable top-k

Nice Works!

Being very interested in your work，But I found that your top-k ranking removes the differentiable top-k method and uses softmax instead, what is the reason for this ?

关于semantic position encoding(SPE)的问题

您好！
对您的工作非常感兴趣，仔细阅读您的工作后发现您代码中使用的是coordconv全局坐标，而非semantic position encoding(SPE)，暂时没看到SPE相关代码，有代码更新更好，万分感谢！

about bi-level warp function "align_feature" and "align_feature_v"

dear author,
I cannot understand the bi-level warp function "align_feature" and "align_feature_v". The align_feature_v seems like the inverse function of the align_feature. Can you explain more about them, especially about details of bi-level warp and its inverse function. Thank you very much!