nvlabs/pacnet

Pixel-Adaptive Convolutional Neural Networks (CVPR '19)

Pixel-Adaptive Convolutional Neural Networks

Project page | Paper | Video

Pixel-Adaptive Convolutional Neural Networks
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz.
CVPR 2019.

License

Copyright (C) 2019 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).

Installation

• Make sure you have Python>=3.5 (we recommend using a Conda environment).
• Add the project directory to your Python paths.
• Install dependencies:
  • PyTorch v0.4-1.1 (incl. torchvision) with CUDA: see PyTorch instructions.
  • Additional libraries:

    pip install -r requirements.txt

• (Optional) Verify installation:

  python -m unittest 

Layer Catalog

We implemented 5 types of PAC layers (as PyTorch Module):

• PacConv2d: the standard variant
• PacConvTranspose2d: the transposed (fractionally-strided) variant for upsampling
• PacPool2d: the pooling variant
• PacCRF: Mean-Field (MF) inference of a CRF
• PacCRFLoose: MF inference of a CRF where the MF steps do not share weights

More details regarding each layer is provided below.

PacConv2d

PacConv2d is the PAC counterpart of nn.Conv2d. It accepts most standard nn.Conv2d arguments (including in_channels, out_channels, kernel_size, bias, stride, padding, dilation, but not groups and padding_mode), and we make sure that when the same arguments are used, PacConv2d and nn.Conv2d have the exact same output sizes. A few additional optional arguments are available:

    Args (in addition to those of Conv2d):
        kernel_type (str): 'gaussian' | 'inv_{alpha}_{lambda}[_asym][_fixed]'. Default: 'gaussian'
        smooth_kernel_type (str): 'none' | 'gaussian' | 'average_{sz}' | 'full_{sz}'. Default: 'none'
        normalize_kernel (bool): Default: False
        shared_filters (bool): Default: False
        filler (str): 'uniform'. Default: 'uniform'

    Note:
        - kernel_size only accepts odd numbers
        - padding should not be larger than :math:`dilation * (kernel_size - 1) / 2`

When used to build computation graphs, this layer takes two input tensors and generates one output tensor:

in_ch, out_ch, g_ch = 16, 32, 8         # channel sizes of input, output and guidance
f, b, h, w = 5, 2, 64, 64               # filter size, batch size, input height and width
input = torch.rand(b, in_ch, h, w)
guide = torch.rand(b, g_ch, h, w)       # guidance feature ('f' in Eq.3 of paper)

conv = nn.Conv2d(in_ch, out_ch, f)
out_conv = conv(input)                  # standard spatial convolution

pacconv = PacConv2d(in_ch, out_ch, f)   
out_pac = pacconv(input, guide)         # PAC 
out_pac = pacconv(input, None, guide_k) # alternative interface
                                        # guide_k is pre-computed 'K' (see Eq.3 of paper) 
                                        # of shape [b, g_ch, f, f, h, w]. packernel2d can be 
                                        # used for its creation.  

Use pacconv2d (in conjunction with packernel2d) for its functional interface.

PacConvTranspose2d

PacConvTranspose2d is the PAC counterpart of nn.ConvTranspose2d. It accepts most standard nn.ConvTranspose2d arguments (including in_channels, out_channels, kernel_size, bias, stride, padding, output_padding, dilation, but not groups and padding_mode), and we make sure that when the same arguments are used, PacConvTranspose2d and nn.ConvTranspose2d have the exact same output sizes. A few additional optional arguments are available: , and also a few additional ones:

    Args (in addition to those of ConvTranspose2d):
        kernel_type (str): 'gaussian' | 'inv_{alpha}_{lambda}[_asym][_fixed]'. Default: 'gaussian'
        smooth_kernel_type (str): 'none' | 'gaussian' | 'average_{sz}' | 'full_{sz}'. Default: 'none'
        normalize_kernel (bool): Default: False
        shared_filters (bool): Default: False
        filler (str): 'uniform' | 'linear'. Default: 'uniform'

    Note:
        - kernel_size only accepts odd numbers
        - padding should not be larger than :math:`dilation * (kernel_size - 1) / 2`

Similar to PacConv2d, PacConvTranspose2d also offers two ways of usage:

in_ch, out_ch, g_ch = 16, 32, 8             # channel sizes of input, output and guidance
f, b, h, w, oh, ow = 5, 2, 8, 8, 16, 16     # filter size, batch size, input height and width
input = torch.rand(b, in_ch, h, w)
guide = torch.rand(b, g_ch, oh, ow)         # guidance feature, note that it needs to match 
                                            # the spatial sizes of the output

convt = nn.ConvTranspose2d(in_ch, out_ch, f, stride=2, padding=2, output_padding=1)
out_convt = convt(input)                    # standard transposed convolution

pacconvt = PacConvTranspose2d(in_ch, out_ch, f, stride=2, padding=2, output_padding=1)   
out_pact = pacconvt(input, guide)           # PAC 
out_pact = pacconvt(input, None, guide_k)   # alternative interface
                                            # guide_k is pre-computed 'K' 
                                            # of shape [b, g_ch, f, f, oh, ow].
                                            # packernel2d can be used for its creation.  

Use pacconv_transpose2d (in conjunction with packernel2d) for its functional interface.

PacPool2d

PacPool2d is the PAC counterpart of nn.AvgPool2d. It accepts most standard nn.AvgPool2d arguments (including kernel_size, stride, padding, dilation, but not ceil_mode and count_include_pad), and we make sure that when the same arguments are used, PacPool2d and nn.AvgPool2d have the exact same output sizes. A few additional optional arguments are available: , and also a few additional ones:

    Args:
        kernel_size, stride, padding, dilation
        kernel_type (str): 'gaussian' | 'inv_{alpha}_{lambda}[_asym][_fixed]'. Default: 'gaussian'
        smooth_kernel_type (str): 'none' | 'gaussian' | 'average_{sz}' | 'full_{sz}'. Default: 'none'
        channel_wise (bool): Default: False
        normalize_kernel (bool): Default: False
        out_channels (int): needs to be specified for channel_wise 'inv_*' (non-fixed) kernels. Default: -1

    Note:
        - kernel_size only accepts odd numbers
        - padding should not be larger than :math:`dilation * (kernel_size - 1) / 2`

Similar to PacConv2d, PacPool2d also offers two ways of usage:

in_ch, g_ch = 16, 8                     # channel sizes of input and guidance
stride, f, b, h, w = 5, 2, 64, 64       # stride, filter size, batch size, input height and width
input = torch.rand(b, in_ch, h, w)
guide = torch.rand(b, g_ch, h, w)       # guidance feature 

pool = nn.AvgPool2d(f, stride)
out_pool = pool(input)                  # standard spatial convolution

pacpool = PacPool2d(f, stride)   
out_pac = pacpool(input, guide)         # PAC 
out_pac = pacpool(input, None, guide_k) # alternative interface
                                        # guide_k is pre-computed 'K'
                                        # of shape [b, g_ch, f, f, h, w]. packernel2d can be 
                                        # used for its creation.  

Use pacpool2d (in conjunction with packernel2d) for its functional interface.

PacCRF and PacCRFLoose

These layers offer a convenient way to add a CRF component at the end of a dense prediction network. They performs approximate mean-field inference under the hood. Available arguments include:

    Args:
        channels (int): number of categories.
        num_steps (int): number of mean-field update steps.
        final_output (str): 'log_softmax' | 'softmax' | 'log_Q'. Default: 'log_Q'
        perturbed_init (bool): whether to perturb initialization. Default: True
        native_impl (bool): Default: False
        fixed_weighting (bool): whether to use fixed weighting for unary/pairwise terms. Default: False
        unary_weight (float): Default: 1.0
        pairwise_kernels (dict or list): pairwise kernels, see add_pairwise_kernel() for details. Default: None

Usage example:

# create a CRF layer for 21 classes using 5 mean-field steps
crf = PacCRF(21, num_steps=5, unary_weight=1.0)

# add a pariwise term with equal weight with the unary term
crf.add_pairwise_kernel(kernel_size=5, dilation=1, blur=1, compat_type='4d', pairwise_weight=1.0)

# a convenient function is provided for creating pairwise features based on pixel color and positions
edge_features = [paccrf.create_YXRGB(im, yx_scale=100.0, rgb_scale=30.0)] 
output = crf(unary, edge_features)

# Note that we use constant values for unary_weight, pairwise_weight, yx_scale, rgb_scale, but they can 
# also take tensors and be learned through backprop.

Experiments

Joint upsampling

Joint depth upsampling on NYU Depth V2

• Train/test split is provided by Li et al.

• Test with one of our pre-trained models:

  python -m task_jointUpsampling.main --load-weights weights_depth/x8_pac_weights_epoch_5000.pth \
                                      --download \
                                      --factor 8 \
                                      --model PacJointUpsample \
                                      --dataset NYUDepthV2 \
                                      --data-root data/nyu

  	4x	8x	16x
  Bilinear	RMSE: 5.43	RMSE: 8.36	RMSE: 12.90
  PacJointUpsample	RMSE: 2.39 | download	RMSE: 4.59 | download	RMSE: 8.09 | download
  PacJointUpsampleLite	RMSE: 2.55 | download	RMSE: 4.82 | download	RMSE: 8.52 | download
  DJIF	RMSE: 2.64 | download	RMSE: 5.15 | download	RMSE: 9.39 | download

• Train from scratch:

  python -m task_jointUpsampling.main --factor 8 \
                                      --data-root data/nyu \
                                      --exp-root exp/nyu \
                                      --download \
                                      --dataset NYUDepthV2 \
                                      --epochs 5000 \
                                      --lr-steps 3500 4500

  See python -m task_jointUpsampling.main -h for the complete list of command line options.

Joint optical flow upsampling on Sintel

• Train/val split (1 - train, 2 - val) is provided in meta/Sintel_train_val.txt (original source):

  • Validation: 133 pairs
    • ambush_6 (all 19)
    • bamboo_2 (last 25)
    • cave_4 (last 25)
    • market_6 (all 39)
    • temple_2 (last 25)
  • Training: remaining 908 pairs

• Test with one of our pre-trained models:

  python -m task_jointUpsampling.main --load-weights weights_flow/x8_pac_weights_epoch_5000.pth \
                                      --download \
                                      --factor 8 \
                                      --model PacJointUpsample \
                                      --dataset Sintel \
                                      --data-root data/sintel

  	4x	8x	16x
  Bilinear	EPE: 0.4650	EPE: 0.9011	EPE: 1.6281
  PacJointUpsample	EPE: 0.1042 | download	EPE: 0.2558 | download	EPE: 0.5921 | download
  DJIF	EPE: 0.1760 | download	EPE: 0.4382 | download	EPE: 1.0422 | download

• Train from scratch:

  python -m task_jointUpsampling.main --factor 8 \
                                      --data-root data/sintel \
                                      --exp-root exp/sintel \
                                      --download \
                                      --dataset Sintel \
                                      --epochs 5000 \
                                      --lr-steps 3500 4500

  See python -m task_jointUpsampling.main -h for the complete list of command line options.

Semantic segmentation

• Test with one of the pre-trained models:

  python -m task_semanticSegmentation.main --data-root data/voc \ 
                                           --exp-root exp/voc \
                                           --download \
                                           --load-weights fcn8s_from_caffe.pth \
                                           --model fcn8s \
                                           --test-split val11_sbd \
                                           --test-crop -1

  	miou (val / test)	model name	weights
  Backbone (FCN8s)	65.51% / 67.20%	fcn8s	download
  PacCRF	68.90% / 69.82%	fcn8s_crfi5p4d5641p4d5161	download
  PacCRF-32	68.52% / 69.41%	fcn8s_crfi5p4d5321	download
  PacFCN (hot-swapping)	67.44% / 69.18%	fcn8spac	download
  PacFCN+PacCRF	69.87% / 71.34%	fcn8spac_crfi5p4d5641p4d5161	download

  Note that the last two models requires argument --test-crop 512.

• Generate predictions

  Use the --eval pred mode to save predictions instead of reporting scores. Predictions will be saved under exp-root/outputs_*_pred, and can be used for VOC evaluation server:

  python -m task_semanticSegmentation.main \
  --data-root data/voc \
  --exp-root exp/voc \
  --load-weights fcn8s_paccrf_epoch_30.pth \
  --test-crop -1 \
  --test-split test \
  --eval pred \
  --model fcn8s_crfi5p4d5641p4d5161 
  
  cd exp/voc
  mkdir -p results/VOC2012/Segmentation
  mv outputs_test_pred results/VOC2012/Segmentation/comp6_test_cls
  tar zcf results_fcn8s_crf.tgz results

  Note that since there is no publicly available URL for the test split of VOC, when using the test split, the data files need to be downloaded from the official website manually. Simply place the downloaded VOC2012test.tar under the data root and untar.

• Train models

  As an example, here shows the commands for the two-stage training of PacCRF:

  # stage 1: train CRF only with frozen backbone
  python -m task_semanticSegmentation.main \
  --data-root data/voc \
  --exp-root exp/voc/crf_only \
  --load-weights-backbone fcn8s_from_caffe.pth \
  --train-split train11 \
  --test-split val11_sbd \
  --train-crop 449 \
  --test-crop -1 \
  --model fcn8sfrozen_crfi5p4d5641p4d5161 \
  --epochs 40 \
  --lr 0.001 \
  --lr-steps 20
  
  # stage 2: train CRF and backbone jointly
  python -m task_semanticSegmentation.main \
  --data-root data/voc \
  --exp-root exp/voc/joint \
  --load-weights-backbone fcn8s_from_caffe.pth \
  --load-weights exp/voc/crf_only/weights_epoch_40.pth \
  --train-split train11 \
  --test-split val11_sbd \
  --train-crop 449 \
  --test-crop -1 \
  --model fcn8s_crfi5lp4d5641p4d5161 \
  --epochs 30 \
  --lr 0.0000001 \
  --lr-steps 20

See python -m task_semanticSegmentation.main -h for the complete list of command line options.

Citation

If you use this code for your research, please consider citing our paper:

@inproceedings{su2019pixel,
  author    = {Hang Su and 
	       Varun Jampani and 
	       Deqing Sun and 
	       Orazio Gallo and 
	       Erik Learned-Miller and 
	       Jan Kautz},
  title     = {Pixel-Adaptive Convolutional Neural Networks},
  booktitle = {Proceedings of the IEEE Conference on Computer 
               Vision and Pattern Recognition (CVPR)},
  year      = {2019}
}

pacnet's Issues

About how to train a fcn8spac from scratch, thanks

Hi,
I want to use the pacconv2d in my code, so I try to running your code firstly but I meet a proplem.
The PacConv2dFn in pac.py:
in_mul_k = cols.view(bs, ch, *kernel.shape[2:]) * kernel
The error is: RuntimeError: shape '[1, 256, 3, 3, 112, 112]' is invalid for input of size 29419776
I train the model with the command as: >python -m task_semanticSegmentation.main --data-root data/voc --exp-root exp/voc/fcn8spac2 --train-split train11 --test-split val11_sbd --train-crop 449 --test-crop 512 --model fcn8spac --epochs 40 --lr 0.001 --lr-steps 20
How can I solve this problem or whether do my parameter exit error?
My environment is: python 3.7, cuda 11.1 and pytorch 1.9

Got an error when using PacConv2d

I want to use PacConv2d, but I got an error when I ran the code. It seems that it was an inplace operation problem, and I don't know why.

Traceback (most recent call last):
File "main.py", line 167, in
train()
File "main.py", line 147, in train
loss.backward()
File "/usr/local/lib/python2.7/dist-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python2.7/dist-packages/torch/autograd/init.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
File "/usr/local/lib/python2.7/dist-packages/torch/autograd/function.py", line 76, in apply
return self._forward_cls.backward(self, *args)
File "/usr/local/lib/python2.7/dist-packages/torch/autograd/function.py", line 188, in wrapper
outputs = fn(ctx, *args)
File "/home/xq/contrast_convlstm/pac.py", line 182, in backward
input, kernel, weight = ctx.saved_tensors
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

The error located at class PacConv2dFn(Function) backward function.
I hope your advice.
Thanks.

How to train with cuda in pytorch?

when I use the pacconvtranspose2d, there is a bug.
expected object of backed CUDA but got backend CPU for argument #2 'mat2'
How to deal with it?
Thank you.

Questions on using the CRF layer

Hello,

I would like to ask some questions the CRF layers that you proposed.

Suppose I can get some CNN (e.g. FCN or DeepLab) using:

cnn = get_cnn()

This CNN can be trained with the usual cross-entropy loss without any issue.

Now to put a CRF on top of it, I define some model for combining a CNN and a CRF like this:

class CNNCRF(torch.nn.Module):
    def __init__(self, cnn, crf):
        super().__init__()
        self.cnn = cnn
        self.crf = crf

    def forward(self, x):
        unaries = self.cnn(x)
        edge_feat = [paccrf.create_YXRGB(x, 80.0, 13.0),
                 paccrf.create_position_feats(x.shape[2:], 3.0, bs=x.shape[0], device=x.device)]             
    return self.crf(unaries, edge_feat)

Then I create a CRF layer and add it to the model:

compat = '2d'
kernel_size = 11
blur=4
dilation = 1
crf_params = dict(num_steps=crf_iterations, perturbed_init=True, fixed_weighting=False, unary_weight=0.8,
pairwise_kernels=[
    dict(kernel_size=kernel_size, dilation=dilation, blur=blur, compat_type=compat, spatial_filter=False,
        pairwise_weight=2.0),
    dict(kernel_size=kernel_size, dilation=dilation, blur=blur, compat_type=compat, spatial_filter=False,
        pairwise_weight=0.6)])
crf = paccrf.PacCRF(num_classes, **crf_params)
model = CNNCRF(cnn, crf)  

I tried training this model by freezing the CNN part (the total number of trainable parameters of the model is then 885) using Adam optimizer with a learning rate of 1e-4, but it was very difficult to converge. Could you please tell me if what I did above is correct?

Thank you in advance for your response.

some basic questions

Hi ,I’m really interested in this work.But as a newcomer in this field ,I got some basic questions after I read the code.

1:Why the channels of the guidance should be half of the input in the examples of the PacConv2d?
2:Are there any rules or the relations of the shape of feature(guide),K(Gussain kernel in this papre), weight(convolution kernel)?

I’m sorry to bother you,I would appreciate it if you can answer me these questions or just help me get a better understanding of this great work. Thank you so much.

memory is sufficient ,but cuda out of memory

Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/arc-wzl2611/mEMC-Net-master/main1.py", line 373, in
main()
File "/home/arc-wzl2611/mEMC-Net-master/main1.py", line 341, in main
log_test = test(model, test_loader, device, last_epoch, init_lr, args.loss, perf_measures, args)
File "/home/arc-wzl2611/mEMC-Net-master/main1.py", line 102, in test
output = apply_model(model, lres, guide, args.factor)
File "/home/arc-wzl2611/mEMC-Net-master/main1.py", line 26, in apply_model
out = net(lres, guide)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/arc-wzl2611/mEMC-Net-master/models.py", line 250, in forward
x = self.up_convts[i](x, guide_cur)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/arc-wzl2611/mEMC-Net-master/pac.py", line 786, in forward
self.output_padding, self.dilation, self.shared_filters, self.native_impl)
File "/home/arc-wzl2611/mEMC-Net-master/pac.py", line 498, in pacconv_transpose2d
shared_filters)
File "/home/arc-wzl2611/mEMC-Net-master/pac.py", line 252, in forward
output = torch.einsum('ijklmn,jokl->iomn', (in_mul_k, weight))
File "/usr/local/lib/python3.5/dist-packages/torch/functional.py", line 243, in einsum
return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: CUDA error: out of memory

autocast support

Hi,

I am trying to use pacnet in my model like in this example: https://github.com/NVlabs/pacnet/blob/12d52b6ebdd8e8afa0d2e54486ba77fbb3697a53/task_semanticSegmentation/fcn8s.py

However, when I try to implement Mixed-Precision training with torch.cuda.amp (with torch 1.11), I have an error in the backward method.

  File "<path>/model/pac.py", line 179, in backward
    grad_in_mul_k = torch.einsum('iomn,ojkl->ijklmn', (grad_output, weight))
  File "/opt/conda/lib/python3.8/site-packages/torch/functional.py", line 325, in einsum
    return einsum(equation, *_operands)
  File "/opt/conda/lib/python3.8/site-packages/torch/functional.py", line 327, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: expected scalar type Half but found Float 

I am using the th14 branch. Is there a fix or a work-around for this?

You may please delete this issue as this is a duplicate. Unfortunately, I cannot delete once posted.

Thanks!

Support 1D

Please support 1d equivalents

Reason for scaling mask input

Hello,

I observed that you've scaled the input to the mask over here.

Is there any significance to this scaling operation?

CRF usage

HI, first of all thank you for this amazing work,

I am working on a semantic segmentation problem, and using CRF as a post processing step, currently I am using pydensecrf, which is a CPU implementation, it would really speed things up if I can replace it with your version of CRF, the only problem is that I can't find all the parameters corresponding to this kernel defined in the fully connected CRF paper:

The kernel above is the one used in dense CRF, and for seeing the provided example, when calling create_YXRGB I've seen that we only provide theta_alpha and theta_beta for the appearance kernel, but I can seem to find the parameters for the smoothness kernel and the weighting factors between the two kernels, I think i am missing something, I wonder how can I specify them.

Thanks.

PacConv3d and AMP

Great work, Han.
Does it support PacConv3d and PacTransposeConv3D? Does it support PyTorch native automatic mixed
precision (AMP)? What will the computational cost if I implement it?

Is pac allow groups argument?

"It accepts all standard nn.Conv2d arguments (e.g. kernel_size, stride, padding, dilation), and we make sure that when the same arguments are used", is groups is allow to use?

Thank you!

PacConv3d

Is there any chance for a quick implementation of 3D convolutions with PAC? Thanks!

Debugging Error

Hi, I'm a bit confused how to deal with this error. Can you help?

/home/joseph/pacnet-master/task_jointUpsampling/main.py:122: UserWarning: genfromtxt: Empty input file: "exp/sintel/train.log"
log = np.genfromtxt(log_path, delimiter=',', skip_header=1, usecols=(0,))
/home/joseph/pacnet-master/task_jointUpsampling/main.py:122: UserWarning: genfromtxt: Empty input file: "exp/sintel/test.log"
log = np.genfromtxt(log_path, delimiter=',', skip_header=1, usecols=(0,))
Traceback (most recent call last):
File "/home/joseph/anaconda3/envs/pac/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/joseph/anaconda3/envs/pac/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/joseph/pacnet-master/task_jointUpsampling/main.py", line 348, in
main()
File "/home/joseph/pacnet-master/task_jointUpsampling/main.py", line 322, in main
log_test = test(model, test_loader, device, last_epoch, init_lr, args.loss, perf_measures, args)
File "/home/joseph/pacnet-master/task_jointUpsampling/main.py", line 86, in test
output = apply_model(model, lres, guide, args.factor)
File "/home/joseph/pacnet-master/task_jointUpsampling/main.py", line 22, in apply_model
out = net(lres, guide)
File "/home/joseph/anaconda3/envs/pac/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/joseph/pacnet-master/task_jointUpsampling/models.py", line 245, in forward
x = self.up_convts[i](x, guide_cur)
File "/home/joseph/anaconda3/envs/pac/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/joseph/pacnet-master/pac.py", line 795, in forward
self.output_padding, self.dilation, self.shared_filters, self.native_impl)
File "/home/joseph/pacnet-master/pac.py", line 507, in pacconv_transpose2d
shared_filters)
File "/home/joseph/pacnet-master/pac.py", line 261, in forward
output = torch.einsum('ijklmn,jokl->iomn', (in_mul_k, weight))
File "/home/joseph/anaconda3/envs/pac/lib/python3.6/site-packages/torch/functional.py", line 211, in einsum
return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: CUDA out of memory. Tried to allocate 2.69 GiB (GPU 0; 10.92 GiB total capacity; 5.74 GiB already allocated; 1.86 GiB free; 2.78 GiB cached)

A question about updating the weight of the kernel

Thanks for sharing the code! But I still have a question.

In a pac layer, a kernel will be calculated according to the guide input. And the kernel will be calculated with the target input to get the output tensor. So when the network does the backpropagation, will the relationship between the weight of the kernel and the guide input change? Or the weight of the kernel is updated only because of the gradient?

I would be very grateful if you can give me some help. Thanks!

Pytorch 1.6 autocast

Are there any plans on adding support of autocasting of Pytorch 1.6?
Currently the weights and kernels are sticking to float and the gradients are half which give a type missmatch error in backpropagation.

For running my example i'm using the code of the 1.4 branch.

PAC_CRF step setting question

Hi

Should we set CRF step to 1 when training the whole model? And step to whatever we want when in the inference processing?

Because when I set CRF step to num>1, it will cause the in-place operation in the training process.

Any TensorFlow implementation?

As above.

Pytorch 1.2.0: Not Implemented Error

Hello, I am currently running Pytorch 1.2.0 with Cuda 9.2 and Python 3.7, but it seems I ran into "NotImplementedError" for all unittest results. Can anyone provide any solutions?

I ran python test_pac.py and here is the results.

E.E/home/pc3387/Desktop/Reference/pacnet/pac.py:447: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  output = output / (norm + torch.tensor(empty_mask, dtype=input.dtype, device=input.device))
.EE.EEEEEEEEE.EEEE
======================================================================
ERROR: test_conv_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "test_pac.py", line 145, in test_conv_all_grad
    (im, im_k, conv_w, conv_b)))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_input_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 114, in test_conv_input_grad
    self.assertTrue(_gradcheck(conv, (im, im_k)))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_sum_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "test_pac.py", line 334, in test_conv_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "test_pac.py", line 179, in test_conv_transpose_all_grad
    (im, im_k, conv_w, conv_b)))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_input_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 157, in test_conv_transpose_input_grad
    self.assertTrue(_gradcheck(conv, (im, im_k)))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_sum_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "test_pac.py", line 356, in test_conv_transpose_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_two_impl_match (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 271, in test_conv_transpose_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_with_kernel_input_sum_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 406, in test_conv_transpose_with_kernel_input_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_two_impl_match (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 211, in test_conv_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_with_kernel_input_sum_all_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 390, in test_conv_with_kernel_input_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_with_kernel_input_two_impl_match (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 241, in test_conv_with_kernel_input_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_kernel_sum_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 375, in test_kernel_sum_grad
    (im,), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_kernel_two_impl_match (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 309, in test_kernel_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 189, in test_pool_grad
    self.assertTrue(_gradcheck(pool, (im, im_k)))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_sum_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 366, in test_pool_sum_grad
    self.assertTrue(_gradcheck(lambda x, y: pool(x, y).sum(), (im, im_k), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_two_impl_match (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 293, in test_pool_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_with_kernel_input_sum_grad (__main__.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "test_pac.py", line 419, in test_pool_with_kernel_input_sum_grad
    (im, im_k), rtol=0.01))
  File "test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

----------------------------------------------------------------------
Ran 21 tests in 10.272s

FAILED (errors=17)
(Pytorch) pc3387@pc3387:~/Desktop/Reference/pacnet$ python -m unittest
E.E/home/pc3387/Desktop/Reference/pacnet/pac.py:447: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  output = output / (norm + torch.tensor(empty_mask, dtype=input.dtype, device=input.device))
.EE.EEEEEEEEE.EEEE
======================================================================
ERROR: test_conv_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 145, in test_conv_all_grad
    (im, im_k, conv_w, conv_b)))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_input_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 114, in test_conv_input_grad
    self.assertTrue(_gradcheck(conv, (im, im_k)))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_sum_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 334, in test_conv_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 179, in test_conv_transpose_all_grad
    (im, im_k, conv_w, conv_b)))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_input_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 157, in test_conv_transpose_input_grad
    self.assertTrue(_gradcheck(conv, (im, im_k)))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_sum_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 28, in call_wrapped
    f(self, *args, native_impl=True)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 356, in test_conv_transpose_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 271, in test_conv_transpose_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_transpose_with_kernel_input_sum_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 406, in test_conv_transpose_with_kernel_input_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 286, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 211, in test_conv_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_with_kernel_input_sum_all_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 390, in test_conv_with_kernel_input_sum_all_grad
    (im, im_k, conv_w, conv_b), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_conv_with_kernel_input_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 241, in test_conv_with_kernel_input_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 196, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_kernel_sum_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 375, in test_kernel_sum_grad
    (im,), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_kernel_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 309, in test_kernel_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 130, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 189, in test_pool_grad
    self.assertTrue(_gradcheck(pool, (im, im_k)))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_sum_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 366, in test_pool_sum_grad
    self.assertTrue(_gradcheck(lambda x, y: pool(x, y).sum(), (im, im_k), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 293, in test_pool_two_impl_match
    out.sum().backward()
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

======================================================================
ERROR: test_pool_with_kernel_input_sum_grad (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 29, in call_wrapped
    f(self, *args, native_impl=False)
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 419, in test_pool_with_kernel_input_sum_grad
    (im, im_k), rtol=0.01))
  File "/home/pc3387/Desktop/Reference/pacnet/test_pac.py", line 21, in _gradcheck
    return gradcheck(f, x0, rtol=rtol, atol=atol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 279, in gradcheck
    analytical, reentrant, correct_grad_sizes = get_analytical_jacobian(tupled_inputs, o, nondet_tol=nondet_tol)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 155, in get_analytical_jacobian
    retain_graph=True, allow_unused=True)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in grad
    inputs, allow_unused)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/pc3387/Desktop/Reference/pacnet/pac.py", line 343, in backward
    ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
  File "/home/pc3387/anaconda3/envs/Pytorch/lib/python3.7/site-packages/torch/_thnn/utils.py", line 27, in __getattr__
    raise NotImplementedError
NotImplementedError

----------------------------------------------------------------------
Ran 21 tests in 9.632s

FAILED (errors=17)

Error in unittest when testing PacConv

Setup:

• Pytorch Build: Pytorch 1.1 (Torchvision 0.3.0)
• OS: Pop_OS (Ubuntu Disco Dingo Derivative)
• Package: pip (instead of recommended conda)
• Python: python 3.7
• CUDA: CUDA 10 (with nvidia-driver-418)

.../home/danish/Documents/Repositories/pacnet/pac.py:447: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  output = output / (norm + torch.tensor(empty_mask, dtype=input.dtype, device=input.device))
............E.....
======================================================================
ERROR: test_kernel_two_impl_match (test_pac.PacConvTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/danish/Documents/Repositories/pacnet/test_pac.py", line 309, in test_kernel_two_impl_match
    out.sum().backward()
  File "/home/danish/Documents/Repositories/pacnet/env/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/danish/Documents/Repositories/pacnet/env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/danish/Documents/Repositories/pacnet/env/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/danish/Documents/Repositories/pacnet/env/lib/python3.7/site-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/home/danish/Documents/Repositories/pacnet/pac.py", line 126, in backward
    grad_diff = grad.expand_as(cols) * (2 * diff)
RuntimeError: CUDA out of memory. Tried to allocate 576.00 MiB (GPU 0; 3.95 GiB total capacity; 2.02 GiB already allocated; 453.06 MiB free; 586.00 MiB cached)

----------------------------------------------------------------------
Ran 21 tests in 32.108s

FAILED (errors=1)

Reason for No ReLU

Is there any intuition behind having no ReLU layers after the last conv layers and all of the pac_t layers?

torch 1.4.0 cannot find type2backend

I think something as been shuffled around in torch. I cannot find type2backend which was previously in torch._thnn in 0.4.
Can we port this to torch 1.4.0?

JBU training error

Hello, i want to apply your PacConvTransposed2d to my model. But when do training,
Traceback (most recent call last):
File "C:\Users\Qyun\anaconda3\envs\pix_net\lib\site-packages\torch\autograd\function.py", line 189, in wrapper
outputs = fn(ctx, *args)
File "C:\projects\pacnet-master\pac.py", line 286, in backward
ctx._backend.Im2Col_updateGradInput(ctx._backend.library_state,
File "C:\Users\Qyun\anaconda3\envs\pix_net\lib\site-packages\torch_thnn\utils.py", line 27, in getattr
raise NotImplementedError
NotImplementedError
this error occurred.
and the same error is occurred when i do training task_jointUpsampling. How can I solve this problem?

Error with pacconv: trying to differentiate twice a function that was marked with @once_differentiable

Hi,

I tried to use pacnet in one my code but when using the function :
pacconv = PacConv2d(in_ch, out_ch, f)
out_pac = pacconv(input, None, guide_k)
In my case input has shape(2,1,200,200) and guide_k (2,1,3,3,200,200).

The forward pass of my code is ok but when computing the backward pass I got this error message:
trying to differentiate twice a function that was marked with @once_differentiable

Do you have any idea of this issue ? Replacing the pacconv by a simple convolution layer removes the error.

Thanks in advance

Implementing Pacnet in keras

Can we expect any Keras implementation for the same functions?

Batch size

Can i set a larger batch size for example 16? I learn that the default batch size is 1.
Actually, i have some questiones about the kernel, if i input an batch(B) of different images and correponding guided features into the pacconv. Different guided features will produce different kernels. There are multiple different kernels in a batch that will increase the complexity of the forward calculation. How do you deal with this kind of problem?

Question about Transpose

I would like to ask whether the transpose operator PacConvTranspose2d is a indeed the transpose operator of PacConv2d or a generic spatially-varying upsampling operator. A transpose/adjoint operator should pass the adjoint test https://en.wikipedia.org/wiki/Hermitian_adjoint, however it is not possible to reproduce the test with success. Based on your test file, I created the following tests:

    def _allclose(x1, x2, rtol=1e-5, atol=1e-10):
        assert np.allclose(x1.cpu(), x2.cpu(), rtol=rtol, atol=atol)

    @repeat_impl_types
    def test_adjoint_const_kernel_th(self, native_impl):
        bs, sz, k_ch = 1, 128, 5
        args = dict(in_channels=3, out_channels=3, kernel_size=5, stride=1, padding=2, dilation=1)
        k_with_d = (args['kernel_size'] - 1) * args['dilation'] + 1
        im_1 = th.rand(bs, args['in_channels'], sz, sz).to(self.device)
        im_2 = th.rand(bs, args['in_channels'], sz, sz).to(self.device)


        conv_w = th.rand(args['in_channels'], args['out_channels'],
                         args['kernel_size'], args['kernel_size']).to(self.device)
        conv_b = th.zeros(args['out_channels']).to(self.device)
        conv_th = nn.Conv2d(**args).to(self.device)
        conv_t_th = nn.ConvTranspose2d(**args).to(self.device)
        conv_th.weight.data[:] = conv_t_th.weight.data[:] = conv_w
        conv_th.bias.data[:] = conv_t_th.bias.data[:] = conv_b
        res1 = conv_th(im_1).detach().reshape(-1)
        res2 = conv_t_th(im_2).detach().reshape(-1)
        _allclose(res1.dot(im_2.reshape(-1)).detach(), res2.dot(im_1.reshape(-1)).detach())


    @repeat_impl_types
    def test_adjoint_const_kernel_pac(self, native_impl):
        bs, sz, k_ch = 1, 6, 5
        args = dict(in_channels=3, out_channels=3, kernel_size=5, stride=1, padding=2, dilation=1)
        k_with_d = (args['kernel_size'] - 1) * args['dilation'] + 1
        sz_out = (sz - 1) * args['stride'] - 2 * args['padding'] + k_with_d + 0 #args['output_padding']
        im_1 = th.rand(bs, args['in_channels'], sz, sz).to(self.device)
        im_2 = th.rand(bs, args['in_channels'], sz, sz).to(self.device)
        im_k = th.rand(bs, k_ch, sz_out, sz_out).to(self.device)
        conv_w = th.rand(args['in_channels'], args['out_channels'],
                         args['kernel_size'], args['kernel_size']).to(self.device)
        conv_b = th.zeros(args['out_channels']).to(self.device)
        conv = pac.PacConv2d(native_impl=native_impl, **args).to(self.device)
        conv_t = pac.PacConvTranspose2d(native_impl=native_impl, **args).to(self.device)
        conv.weight.data[:] = conv_t.weight.data[:] = conv_w
        conv.bias.data[:] = conv_t.bias.data[:] = conv_b
        res1 = conv(im_1, im_k).detach().reshape(-1)
        res2 = conv_t(im_2, im_k).detach().reshape(-1)

        _allclose(res1.dot(im_2.reshape(-1)).detach(), res2.dot(im_1.reshape(-1)).detach())

The Pytorch implementations of Conv2D and ConvTranspose2d pass the test with success, however PacConv2d and PacConvTranspose2d fail to pass the test.

Best regards,
Filippos

how to generate predictions using fcn8spac?

Hello,

I am impressed by your great work. And I have some questions about your code.
I used model fcn8spac to train the VOC2012. And after training, I got many pth files. But when I want to generate predictions for VOC test dataset, I can only generate some black images. And I got the normal acc on VOC training data.
And my script is 'CUDA_VISIBLE_DEVICES=3 python -m task_semanticSegmentation.main --data-root data/voc --exp-root exp/voc/fcn8s_pac_but_change_to_pad --load-weights pacnet/exp/voc/fcn8s_pac_but_change_to_pad/weights_epoch_40.pth --test-crop 512 --test-split test --eval pred --model fcn8spac'.
Could you please give me some advices on how to finish that?
Best,
Amose.

About "Native Implemention"

Hi Suhang! Thanks for your sharing the code! I just wonder why there is a native implemention to control the computational flow. Is there any theoretical or performance differences between the native implemantion and non-native implemention?

about the crf model