megvii-model/weightnet

WeightNet

This repository provides MegEngine implementation for "WeightNet: Revisiting the Design Space of Weight Network".

Requirement

• MegEngine 0.5.1 (https://github.com/MegEngine/MegEngine)

Citation

If you use these models in your research, please cite:

@inproceedings{ma2020weightnet, 
            title={WeightNet: Revisiting the Design Space of Weight Networks},  
            author={Ma, Ningning and Zhang, Xiangyu and Huang, Jiawei and Sun, Jian},  
            booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},  
            year={2020} 
}

Usage

Train:

    python3 train.py --dataset-dir=/path/to/imagenet

Eval:

    python3 test.py --data=/path/to/imagenet --model /path/to/model --ngpus 1

Inference:

    python3 inference.py --model /path/to/model --image /path/to/image.jpg

Trained Models

• OneDrive download: Link

Results

• Comparison under the same #Params and the same FLOPs.

Model	#Params.	FLOPs	Top-1 err.
ShuffleNetV2 (0.5×)	1.4M	41M	39.7
+ WeightNet (1×)	1.5M	41M	36.7
ShuffleNetV2 (1.0×)	2.2M	138M	30.9
+ WeightNet (1×)	2.4M	139M	28.8
ShuffleNetV2 (1.5×)	3.5M	299M	27.4
+ WeightNet (1×)	3.9M	301M	25.6
ShuffleNetV2 (2.0×)	5.5M	557M	25.5
+ WeightNet (1×)	6.1M	562M	24.1

• Comparison under the same FLOPs.

Model	#Params.	FLOPs	Top-1 err.
ShuffleNetV2 (0.5×)	1.4M	41M	39.7
+ WeightNet (8×)	2.7M	42M	34.0
ShuffleNetV2 (1.0×)	2.2M	138M	30.9
+ WeightNet (4×)	5.1M	141M	27.6
ShuffleNetV2 (1.5×)	3.5M	299M	27.4
+ WeightNet (4×)	9.6M	307M	25.0
ShuffleNetV2 (2.0×)	5.5M	557M	25.5
+ WeightNet (4×)	18.1M	573M	23.5

weightnet's Issues

in the WeightNet_DW，is that "x_w = x_w.reshape(-1, 1, 1, self.ksize, self.ksize)"wrong？why the input and output channel is 1?

class WeightNet_DW(M.Module):
r""" Here we show a grouping manner when we apply WeightNet to a depthwise convolution.

The grouped fc layer directly generates the convolutional kernel, has fewer parameters while achieving comparable results.
This layer has M/G*inp inputs, inp groups and inp*ksize*ksize outputs.

"""
def __init__(self, inp, ksize, stride):
    super().__init__()

    self.M = 2
    self.G = 2

    self.pad = ksize // 2
    inp_gap = max(16, inp//16)
    self.inp = inp
    self.ksize = ksize
    self.stride = stride

    self.wn_fc1 = M.Conv2d(inp_gap, self.M//self.G*inp, 1, 1, 0, groups=1, bias=True)
    self.sigmoid = M.Sigmoid()
    self.wn_fc2 = M.Conv2d(self.M//self.G*inp, inp*ksize*ksize, 1, 1, 0, groups=inp, bias=False)

# x_gap是经过AGP的值
def forward(self, x, x_gap):
    x_w = self.wn_fc1(x_gap)
    x_w = self.sigmoid(x_w)
    x_w = self.wn_fc2(x_w)

    x = x.reshape(1, -1, x.shape[2], x.shape[3])
    x_w = x_w.reshape(-1, 1, 1, self.ksize, self.ksize)
    x = F.conv2d(x, weight=x_w, stride=self.stride, padding=self.pad, groups=x_w.shape[0])
    x = x.reshape(-1, self.inp, x.shape[2], x.shape[3])
    return x

5-D tensors as weight for Conv2d?

Thanks for this outstanding work, I'm wondering why the weightnet.py use 5D tensors as weights for F.conv2d?
Does this really work for F.conv2d?

Actually this will raise errors for PyTorch, so is there a different behavior using MegEngine?
e.g. the code here.

Looking forward to your reply.

RetinaNet-WeightNet can not convergence ，with loss = 1.72

Though warming up is adopted, it also can't convergence.

Why is the bias of wn_fc2 of class WeightNet in weightnet.py set to be false?

Hi,

I am trying to apply WeightNet to my own project and have a question about the second grouped fully connected layer of WeightNet: why is the bias set to false? Is there any reason behind this?

pytorch version

When I convert it to the Pytorch version, I have the following problems. Can you help me solve them？thanks
error:
x_gap = x.mean(axis=2,keepdims=True).mean(axis=3,keepdims=True)
TypeError: mean() received an invalid combination of arguments - got (axis=int, keepdims=bool, ), but expected one of:

• ()
• (torch.dtype dtype)
• (tuple of ints dim, torch.dtype dtype)
  didn't match because some of the keywords were incorrect: axis, keepdims
• (tuple of ints dim, bool keepdim, torch.dtype dtype)
• (tuple of ints dim, bool keepdim)
  didn't match because some of the keywords were incorrect: axis, keepdims

Should the standard pointwise convolution in the skip path of Resblock (upsample/downsample=True) be kept or also replaced by WeightNet?

Hi,

I have a question about applying WeightNet to residual blocks with upsampling/downsampling. In this case, the skip path also has a 1x1 convolution. Should this conv layer also be replaced by WeightNet? Or it is better to keep it as standard convolution?

如何迁移学习自己的数据集

error with distributed training

Hi,
Thank you for this great work and code!
When I tried to run the code with multi-gpus, I got an error in the following line.

WeightNet/train.py

Line 156 in ea318d8

loss = dist.all_reduce_sum(loss, "train_loss") / dist.get_world_size()

And the error log is as follows:

loss = dist.all_reduce_sum(loss, "train_loss") / dist.get_world_size()
File "/opt/conda/lib/python3.7/site-packages/megengine/distributed/functional.py", line 238, in all_reduce_sum
assert _group_check(key, nr_ranks), "key, nr_ranks should be set at the same time"
AssertionError: key, nr_ranks should be set at the same time

I'm new to using MegEngine and could not find any related issues by googling.
Please let me know how to resolve this error.
Thank you!