zdaxie/pixpro

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

pixpro's Issues

Question about lars implementation

Hi, zdaxie.

thank you for sharing code.

I have a question about Lars optimizer.

PixPro/contrast/lars.py

Line 133 in e390d6b

adaptive_lr = self.trust_coef * param_norm / (grad_norm + self.eps)

In above code line, I think the correct code as follows:
adaptive_lr = self.trust_coef * param_norm / (grad_norm + weight_decay * param_norm)

What do you think about it ?

Ablation study with larger image size

Hi, thank you for the great work.
I would like to know if anyone has done experiments pre-training with larger Image Size, especially when pre-training for object detection/segmentation tasks ?
Thank you

What is max_bin_diag?

Thanks for open source. What is the function of max_bin_diag?

Training log could be released?

Loss Curve

Thanks for your great work.
Could you please show your loss curve in traing? Or could you let me know the loss value of PixPro at the end of training?

About the "pixpro_instance_loss" question ???

Thank you very much for your work and code sharing. In the Pixpro.py, I found that in "pixpro_instance_loss", 'pred_instance_1' has one more predictor layer than 'proj_instance_1_ng'. I don’t quite understand why this is necessary. I am waiting for your reply. Thank you very much.

   if self.pixpro_ins_loss_weight > 0.:
        proj_instance_1 = self.projector_instance(feat_1)
        pred_instacne_1 = self.predictor(proj_instance_1)
        pred_instance_1 = F.normalize(self.avgpool(pred_instacne_1).view(pred_instacne_1.size(0), -1), dim=1)

        proj_instance_2 = self.projector_instance(feat_2)
        pred_instance_2 = self.predictor(proj_instance_2)
        pred_instance_2 = F.normalize(self.avgpool(pred_instance_2).view(pred_instance_2.size(0), -1), dim=1)

    # compute key features
    with torch.no_grad():  # no gradient to keys
        self._momentum_update_key_encoder()  # update the key encoder

        feat_1_ng = self.encoder_k(im_1)  # keys: NxC
        proj_1_ng = self.projector_k(feat_1_ng)
        proj_1_ng = F.normalize(proj_1_ng, dim=1)

        feat_2_ng = self.encoder_k(im_2)
        proj_2_ng = self.projector_k(feat_2_ng)
        proj_2_ng = F.normalize(proj_2_ng, dim=1)

        if self.pixpro_ins_loss_weight > 0.:
            proj_instance_1_ng = self.projector_instance_k(feat_1_ng)
            proj_instance_1_ng = F.normalize(self.avgpool(proj_instance_1_ng).view(proj_instance_1_ng.size(0), -1),
                                             dim=1)

            proj_instance_2_ng = self.projector_instance_k(feat_2_ng)
            proj_instance_2_ng = F.normalize(self.avgpool(proj_instance_2_ng).view(proj_instance_2_ng.size(0), -1),
                                             dim=1)

Checkpoint with Instance Branch

Hello, thanks for your great work. Could you please release the checkpoint with instance branch of BYOL(the implementation in code)?

Does feature maps from different levels share the same projection MLP?

As for the implementation of pre-training backbone and FPNs, I wonder whether feature maps from different levels share the same projection MLP?

How to reproduce semantic segmentation?

Can you release the downstream task of semantic segmentation code?

FileNotFoundError: [Errno 2] No such file or directory: './data/imagenet/train_map.txt'

Hi @ancientmooner @zdaxie ,

Thanks for your contribution, but why there need a txt file to load ImageNet? How can I get it?

the pre-training loss

The pre-training losses are always negative (like -199.03), is that normal?

How to use instance-level self-supervised learning on FPN?

when using instance-level self-supervised learning on FPN， do we need to add loss on each level of FPN or still in C5 ?

How to reproduce semantic segmentation?

Can you release the downstream task of semantic segmentation code?

Few implementations that do not match the original paper.

The similarity computation is cosine-related function in the paper, but here is torch.bmm, may be such function is not important.

About Pre-Training documents

Could you share the documents obtained from the Pre-Training? Thank you

pretrain：Gradient overflow

I tried to pretrain the model, it shows "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 131072.0", is that right result? Could you show your loss curve?

About pixcontrast loss

Thank you very much for your work. After reading the code, I have a question.
In the paper, the final loss consists of $l_{pixcontrast}, l_{pixpro} and l_{instance}$.
But In the Pixpro.py, I just find the $l_{pixpro}$:

        # compute loss
        loss = regression_loss(pred_1, proj_2_ng, coord1, coord2, self.pixpro_pos_ratio) \
            + regression_loss(pred_2, proj_1_ng, coord2, coord1, self.pixpro_pos_ratio)

and $l_{instance}$:

      loss_instance = self.regression_loss(pred_instance_1, proj_instance_2_ng) + \
                   self.regression_loss(pred_instance_2, proj_instance_1_ng)

So I wonder where $l_{pixcontrast}$ is, do I understand wrong or where to see the negligence?

Any information about pretraining time

How long would it take to pretrain the pixpro?

Batch size ablation results

Hello, thanks for your great work.
Can you provide additional ablations obtained using different batch size ? (e.g. smaller batch size 512/256, instead of the 1024 reported in paper)
I vary the training batch size but I find that the final result vary a lot.

PixContrast loss

Hi, thank you for sharing your code!
Do you have any plan to release th code of pixcontrast part?

About your pretrained weight

About your beginning training weight.

Did you use random weight or other's pre-training weight to train your network?

PIL file open error

Hi,

Thank you for this amazing repo. I am trying to train a model. However, I am getting below error. It seems like, issue with PIL reader.

   data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/work/PixPro/contrast/data/dataset.py", line 261, in __getitem__
    image = self.loader(path)
  File "/work/PixPro/contrast/data/dataset.py", line 219, in default_img_loader
    return pil_loader(path)
  File "/work/PixPro/contrast/data/dataset.py", line 202, in pil_loader
    return img.convert('RGB')
  File "/home/users/conda/envs/PixPro/lib/python3.7/site-packages/PIL/Image.py", line 904, in convert
    self.load()
  File "/home/users/conda/envs/PixPro/lib/python3.7/site-packages/PIL/ImageFile.py", line 228, in load
    seek(offset)
ValueError: seek of closed file

I am using dual gpu setup and I have set the parameters accordingly.

Could you please help me?
Thanks

Pre-trained models with instance branch

Hello, thanks for this interesting work.
Here is one question, do you have a plan to release pre-trained models with instance branch?
Thanks.

Relative coordinates

I've noticed that you normalized the coordinates to the range from 0 to 1. Does this mean the distance threshold should stay the same for absolute coordinates?

Runtime error while training a pretrained model

Hello,

When I try to run ./tools/pixpro_base_r50_100ep.sh, it crashes with the following error messages. I was wondering if anyone knows how to fix it. Thanks.

File "main_pretrain.py", line 241, in
return forward_call(*input, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
main(opt)
File "main_pretrain.py", line 141, in main
train(epoch, train_loader, model, optimizer, scheduler, args, summary_writer)
File "main_pretrain.py", line 165, in train
loss = model(data[0], data[1], data[2], data[3])
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
return forward_call(*input, **kwargs)
File "/home/test/Desktop/PixPro/contrast/models/PixPro.py", line 226, in forward
feat_2 = self.encoder(im_2)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
return forward_call(*input, **kwargs)
File "/home/test/Desktop/PixPro/contrast/resnet.py", line 200, in forward
x = self.conv1(x)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return module_to_run(*inputs[0], **kwargs[0])
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 463, in forward
return forward_call(*input, **kwargs)
File "/home/test/Desktop/PixPro/contrast/models/PixPro.py", line 226, in forward
feat_2 = self.encoder(im_2)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return self._conv_forward(input, self.weight, self.bias)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 460, in _conv_forward
return forward_call(*input, **kwargs)
File "/home/test/Desktop/PixPro/contrast/resnet.py", line 200, in forward
self.padding, self.dilation, self.groups)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/apex/amp/wrap.py", line 21, in wrapper
args[i] = utils.cached_cast(cast_fn, args[i], handle.cache)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/apex/amp/utils.py", line 97, in cached_cast
x = self.conv1(x)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
if cached_x.grad_fn.next_functions[1][0].variable is not x:
IndexError: tuple index out of range
return forward_call(*input, **kwargs)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 460, in _conv_forward
self.padding, self.dilation, self.groups)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/apex/amp/wrap.py", line 21, in wrapper
args[i] = utils.cached_cast(cast_fn, args[i], handle.cache)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/apex/amp/utils.py", line 97, in cached_cast
if cached_x.grad_fn.next_functions[1][0].variable is not x:
IndexError: tuple index out of range
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2554351) of binary: /home/test/.conda/envs/test/bin/python
Traceback (most recent call last):
File "/home/test/.conda/envs/test/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/test/.conda/envs/test/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/launch.py", line 195, in
main()
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/launch.py", line 191, in main
launch(args)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/launch.py", line 176, in launch
run(args)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/run.py", line 756, in run
)(*cmd_args)
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/test/.conda/envs/test/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 248, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

main_pretrain.py FAILED

Reproduction

Any one successfully reproduce the results?

Miss SyncBatchNorm in Pixel Propagation Module

PixPro/contrast/models/PixPro.py

Line 141 in e390d6b

self.value_transform = MLP2d(in_dim=256, inner_dim=256, out_dim=256)

If I set pixpro_transform_layer=2, it will use Batchnorm between 2 linear layers, but it seems that you didn't convert this batchnorm to SyncBatchNorm.

distance threshold setting when using P3-P6 to do self-supervised learning?

when using P3-P6 of FPN to do self-supervised learning, how should we set the distance threshold? Setting it to be 0.7 each level or setting to different value on different level?