Giter Site home page Giter Site logo

e2cnn's Introduction

General E(2)-Equivariant Steerable CNNs

Documentation | Experiments | Paper | Thesis Gabriele | Thesis Maurice | new escnn library


Check out our new escnn library which extends e2cnn to a wider class of equivariance groups.

While we will still provide some support for this older version, this library is deprecated and we plan to slowly abandon it in favour of the newer version escnn. Note that escnn already includes all features of e2cnn and many more. You can find a short summary of the main differences in the new version here.



e2cnn is a PyTorch extension for equivariant deep learning.

Equivariant neural networks guarantee a specified transformation behavior of their feature spaces under transformations of their input. For instance, classical convolutional neural networks (CNNs) are by design equivariant to translations of their input. This means that a translation of an image leads to a corresponding translation of the network's feature maps. This package provides implementations of neural network modules which are equivariant under all isometries E(2) of the image plane $\mathbb{R}^2$, that is, under translations, rotations and reflections. In contrast to conventional CNNs, E(2)-equivariant models are guaranteed to generalize over such transformations, and are therefore more data efficient.

The feature spaces of E(2)-Equivariant Steerable CNNs are defined as spaces of feature fields, being characterized by their transformation law under rotations and reflections. Typical examples are scalar fields (e.g. gray-scale images or temperature fields) or vector fields (e.g. optical flow or electromagnetic fields).

feature field examples

Instead of a number of channels, the user has to specify the field types and their multiplicities in order to define a feature space. Given a specified input- and output feature space, our R2conv module instantiates the most general convolutional mapping between them. Our library provides many other equivariant operations to process feature fields, including nonlinearities, mappings to produce invariant features, batch normalization and dropout. Feature fields are represented by GeometricTensor objects, which wrap a torch.Tensor with the corresponding transformation law. All equivariant operations perform a dynamic type-checking in order to guarantee a geometrically sound processing of the feature fields.

E(2)-Equivariant Steerable CNNs unify and generalize a wide range of isometry equivariant CNNs in one single framework. Examples include:

For more details we refer to our NeurIPS 2019 paper General E(2)-Equivariant Steerable CNNs.

The library also supports equivariant Steerable partial differential operators as described in Steerable Partial Differential Operators for Equivariant Neural Networks.


The library is structured into five subpackages with different high-level features:

Component Description
e2cnn.group implements basic concepts of group and representation theory
e2cnn.kernels solves for spaces of equivariant convolution kernels
e2cnn.diffops solves for spaces of equivariant differential operators
e2cnn.gspaces defines the image plane and its symmetries
e2cnn.nn contains equivariant modules to build deep neural networks

Demo

Since E(2)-steerable CNNs are equivariant under rotations and reflections, their inference is independent from the choice of image orientation. The visualization below demonstrates this claim by feeding rotated images into a randomly initialized E(2)-steerable CNN (left). The middle plot shows the equivariant transformation of a feature space, consisting of one scalar field (color-coded) and one vector field (arrows), after a few layers. In the right plot we transform the feature space into a comoving reference frame by rotating the response fields back (stabilized view).

Equivariant CNN output

The invariance of the features in the comoving frame validates the rotational equivariance of E(2)-steerable CNNs empirically. Note that the fluctuations of responses are discretization artifacts due to the sampling of the image on a pixel grid, which does not allow for exact continuous rotations.

For comparison, we show a feature map response of a conventional CNN for different image orientations below.

Conventional CNN output

Since conventional CNNs are not equivariant under rotations, the response varies randomly with the image orientation. This prevents CNNs from automatically generalizing learned patterns between different reference frames.

Experimental results

E(2)-steerable convolutions can be used as a drop in replacement for the conventional convolutions used in CNNs. Keeping the same training setup and without performing hyperparameter tuning, this leads to significant performance boosts compared to CNN baselines (values are test errors in percent):

model CIFAR-10 CIFAR-100 STL-10
CNN baseline 2.6   ± 0.1   17.1   ± 0.3   12.74 ± 0.23
E(2)-CNN * 2.39 ± 0.11 15.55 ± 0.13 10.57 ± 0.70
E(2)-CNN 2.05 ± 0.03 14.30 ± 0.09   9.80 ± 0.40

The models without * are for a fair comparison designed such that the number of parameters of the baseline is approximately preserved while models with * preserve the number of channels, and hence compute. For more details we refer to our paper.

Getting Started

e2cnn is easy to use since it provides a high level user interface which abstracts most intricacies of group and representation theory away. The following code snippet shows how to perform an equivariant convolution from an RGB-image to 10 regular feature fields (corresponding to a group convolution).

from e2cnn import gspaces                                          #  1
from e2cnn import nn                                               #  2
import torch                                                       #  3
                                                                   #  4
r2_act = gspaces.Rot2dOnR2(N=8)                                    #  5
feat_type_in  = nn.FieldType(r2_act,  3*[r2_act.trivial_repr])     #  6
feat_type_out = nn.FieldType(r2_act, 10*[r2_act.regular_repr])     #  7
                                                                   #  8
conv = nn.R2Conv(feat_type_in, feat_type_out, kernel_size=5)       #  9
relu = nn.ReLU(feat_type_out)                                      # 10
                                                                   # 11
x = torch.randn(16, 3, 32, 32)                                     # 12
x = nn.GeometricTensor(x, feat_type_in)                            # 13
                                                                   # 14
y = relu(conv(x))                                                  # 15

Line 5 specifies the symmetry group action on the image plane $\mathbb{R}^2$ under which the network should be equivariant. We choose the cyclic group C8, which describes discrete rotations by multiples of 2π/8. Line 6 specifies the input feature field types. The three color channels of an RGB image are thereby to be identified as three independent scalar fields, which transform under the trivial representation of C8. Similarly, the output feature space is in line 7 specified to consist of 10 feature fields which transform under the regular representation of C8. The C8-equivariant convolution is then instantiated by passing the input and output type as well as the kernel size to the constructor (line 9). Line 10 instantiates an equivariant ReLU nonlinearity which will operate on the output field and is therefore passed the output field type.

Lines 12 and 13 generate a random minibatch of RGB images and wrap them into a nn.GeometricTensor to associate them with their correct field type. The equivariant modules process the geometric tensor in line 15. Each module is thereby checking whether the geometric tensor passed to them satisfies the expected transformation law.

Because the parameters do not need to be updated anymore at test time, after training, any equivariant network can be converted into a pure PyTorch model with no additional computational overhead in comparison to conventional CNNs. The code currently supports the automatic conversion of a few commonly used modules through the .export() method; check the documentation for more details.

A hands-on tutorial, introducing the basic functionality of e2cnn, is provided in introduction.ipynb. Code for training and evaluating a simple model on the rotated MNIST dataset is given in model.ipynb.

More complex equivariant Wide Resnet models are implemented in e2wrn.py. To try a model which is equivariant under reflections call:

cd examples
python e2wrn.py

A version of the same model which is simultaneously equivariant under reflections and rotations of angles multiple of 90 degrees can be run via:

python e2wrn.py --rot90

Dependencies

The library is based on Python3.7

torch>=1.1
numpy
scipy

Optional:

pymanopt
autograd

The following packages

sympy
rbf

are required to use the steerable differential operators.

Check the branch legacy_py3.6 for a Python 3.6 compatible version of the library.

Installation

You can install the latest release as

pip install e2cnn

or you can clone this repository and manually install it with

pip install git+https://github.com/QUVA-Lab/e2cnn

NOTE: the rbf dependency (necessary to use differential operators) can not be installed automatically since PyPI does not support direct dependencies (see here). You can install rbf manually by following these instructions.

Cite

The development of this library was part of the work done for our paper General E(2)-Equivariant Steerable CNNs. Please cite this work if you use our code:

@inproceedings{e2cnn,
    title={{General E(2)-Equivariant Steerable CNNs}},
    author={Weiler, Maurice and Cesa, Gabriele},
    booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
    year={2019},
}

The implementation of steerable PDOs are part of the work done in the paper Steerable Partial Differential Operators for Equivariant Neural Networks. Please, cite it if you use this code in your own work:

@inproceedings{jenner2022steerable,
    title={Steerable Partial Differential Operators for Equivariant Neural Networks},
    author={Erik Jenner and Maurice Weiler},
    booktitle={International Conference on Learning Representations},
    year={2022},
    url={https://openreview.net/forum?id=N9W24a4zU}
}

Feel free to contact us.

License

e2cnn is distributed under BSD Clear license. See LICENSE file.

e2cnn's People

Contributors

ahyunseo avatar braun-steven avatar ejnnr avatar fsherry avatar gabri95 avatar marcelroed avatar mauriceweiler avatar nijkah avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

e2cnn's Issues

Conversion to torchscript or ONNX

I'm working on optimizing my model inference, trying conversion to torchscript as a first step. When I call torch.jit.script() on my model, I hit:

name = '_weights_ranges', item = {('irrep_0,0', 'regular'): (0, 288)}

    def infer_type(name, item):
        # The forward function from Module is special; never use this annotations; we
        # need to infer type directly using JIT.  I originally wanted to write
        # this test as isinstance(class_annotations[name], Callable) but
        # isinstance on typing things doesn't seem to work: isinstance(list, Callable)
        # is also true!
        if name in class_annotations and class_annotations[name] != torch.nn.Module.__annotations__["forward"]:
            attr_type = torch.jit.annotations.ann_to_type(class_annotations[name], _jit_internal.fake_range())
        elif isinstance(item, torch.jit.Attribute):
            attr_type = torch.jit.annotations.ann_to_type(item.type, _jit_internal.fake_range())
        else:
>           attr_type = torch._C._jit_try_infer_type(item)
E           RuntimeError: Cannot create dict for key type '(str, str)', only int, float, Tensor and string keys are supported

This pytorch code resides here:
https://github.com/pytorch/pytorch/blob/22902b9242853a4ce319e7c5c4a1c94bc00ccb7a/torch/jit/_recursive.py#L126

Torch can't trace through a Dict[Tuple[str,str],_], which is used here:

coefficients = weights[self._weights_ranges[io_pair][0]:self._weights_ranges[io_pair][1]]

My goal is to get the model to run as fast as possible on NVIDIA hardware, probably using tensorrt. Is there another known-good conversion path?

The above error was with torch 1.6.0, e2cnn v0.1.

ZeroPad2D on GeometricTensor

Is it possible to do ZeroPad2D on the geometric tensor? Specifically, I want my input image and output image to have the same dimension and I am using kernel size = 4. With the normal CNN, I can achieve this with pad = torch.nn.ZeroPad2d((2, 1, 2, 1)). How can I do this type of padding with equivariant CNNs?

State dict different in train and eval

Just a small problem I just encountered, currently a convolutional models state_dict in train mode is different to its state_dict in eval mode. Which results in a "Missing Keys" error when loading a state_dict saved during training into a model in eval mode. It is not critical though as loading with "strict=False" and then running model.eval() restores the filter and the expanded_bias.

If ``mode=True``, sets the module in training mode and discards the :attr:`~e2cnn.nn.R2Conv.filter` and

Init of Field Dropout

When creating a FieldDropout Layer, I get the following error: TypeError: cannot assign 'torch.LongTensor' object to parameter 'indices_8' (torch.nn.Parameter or None required). Initializing PointwiseDropout with the same parameters works.

Is this a known issue?

R2Conv acts on 5-D tensor

There is a situation when images are stacked twice. The shape of the structured image tensor is (B, S, C, H, W) where B represents batch size and S the number of stacked images.
Could R2conv directly act or broadcast on such a 5-D tensor?

equivariance in C8 space

Hi,

When I'm testing the equivariance for a sample example in C8 space following the "introduction.ipynb", I've observed it to fail. So, I wanted to check if there is a correct way of evaluating the c8 space equivariance? Further, It would help me if you could point out how could one go about defining an architecture such that equivariance could be observed across all angles of C8 space?

My code:

import torch

from e2cnn import gspaces
from e2cnn import nn


def conv3x3(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=1,
            dilation=1, bias=False):
    """3x3 convolution with padding"""
    return nn.R2Conv(in_type, out_type, 3,
                      stride=stride,
                      padding=padding,
                      dilation=dilation,
                      bias=bias,
                      sigma=None,
                      frequencies_cutoff=lambda r: 3*r,
                      )

def conv1x1(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=0,
            dilation=1, bias=False):
    """1x1 convolution with padding"""
    return nn.R2Conv(in_type, out_type, 1,
                      stride=stride,
                      padding=padding,
                      dilation=dilation,
                      bias=bias,
                      sigma=None,
                      frequencies_cutoff=lambda r: 3*r,
                      )


r2_act = gspaces.Rot2dOnR2(N=8)
feat_type_in = nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = nn.FieldType(r2_act, 3*[r2_act.regular_repr])

conv = conv3x3(feat_type_in, feat_type_out)

x = torch.randn(4, 1, 32, 32)
x = nn.GeometricTensor(x, feat_type_in)

assert isinstance(x.tensor, torch.Tensor)

y = conv(x)

assert y.type == feat_type_out

print(y.tensor.shape)


for g in r2_act.testing_elements:
    # transform the input with the current group element according to the input type
    x_transformed = x.transform(g)
    
    # feed the transformed input in the convolutional layer
    y_from_x_transformed = conv(x_transformed)
    
    # the result should be equivalent to rotating the output produced in the 
    # previous block according to the output type
    y_transformed_from_x = y.transform(g)
    
    assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g

print("success")

result:

Traceback (most recent call last):
  File "e2_test.py", line 62, in <module>
    assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g
AssertionError: 1

Rotation equivariance on a specific angle

Hi, @Gabri95 , thank you for your nice work! And I have got inspiration from your work!
But a question (may be very simple) confused me:
How to generate rotation equivariant features on a specific angle (e.g., 45, 33.3), but not all the 360 degree features.
In other words, I just want the features on a particular orientation.
I guess the FieldType defined in SO(2) can resolve this problem, but I dont know how to use related functions.

Layout of weight tensors sampled from SteerableBasis

Hi admins, Thanks for the nice paper and handy code.

out = np.zeros((self.shape[0], self.shape[1], self.dim, angles.shape[1]))

The weight tensor return by SteerableBasis.sample has the shape of out_channels x in_channels x dim x len(angles). dim is the summation of basis dimensions of all input/output representation pairs. I feel that this shape is redundant since each basis is only used by its corresponding input/output pair.

for ii, in_size in enumerate(self.in_sizes):
out_position = 0
for oo, out_size in enumerate(self.out_sizes):
if self.bases[ii][oo] is not None:
dim = self.bases[ii][oo].dim
block = out[
out_position:out_position+out_size,
in_position:in_position+in_size,
basis_count:basis_count+dim,
...
]
self.bases[ii][oo].sample(angles, out=block)

The sampling code also implies that the current weight tensor is very sparse by filling a 3D tensor with 2 for loops. I feel that theoretically out_channels x in_channels x 4 x len(angles) should be enough to save all sampled basis as a non-trivial irrep pair usually corresponds to 4 bases.

Could you please let me know if this interpretation is correct? Thanks!

Counting FLOPs for e2cnn

Hello,

Thank you for sharing this amazing work!

While playing it a little bit, I wonder how many flops the network consumes.

I found that the existing library (e.g., https://github.com/1adrianb/pytorch-estimate-flops) does not work for this as they don't support equivariant modules.

Is there any simple method for this one? If the counting for kernel construction parts would be hard, I just want to know how to translate the model into torch model as in test phase to count using the existing library.

Should non-trivial irreps support bias or not?

Hi @Gabri95 ,

I noticed the current implementation only supports bias for trivial representation as commented:

if bias:
# bias can be applied only to trivial irreps inside the representation
# to apply bias to a field we learn a bias for each trivial irreps it contains
# and, then, we transform it with the change of basis matrix to be able to apply it to the whole field
# this is equivalent to transform the field to its irreps through the inverse change of basis,
# sum the bias only to the trivial irrep and then map it back with the change of basis

From a naive intuition, I feel that bias acted on the norm of an irrep seems also equivariant. I didn't find relevant description in the paper. So just a short question for any relevant reference to deny this intuition. Thanks!

Feature Request: Code for generating equivariance visualizations

I really like your visualizations of rotation equivariance (i.e. vectorfield.gif and conventional_cnn.gif
), and would like to generate similar ones for my network. Is the code for generating those something that could be easily added, or should I go ahead write my own code for that?

Need equivariant ResNet & VGG like architectural networks implemented in E2CNN library

Thank you for this great library for equivariant steerable CNNs. However, I am not able to build an equivariant model for my project through the examples given in this repository. Can you include examples for ResNet & VGG like architectures or can someone tell me how to induce equivariance in these steerable networks. For example, I tried building equivariant model based on WideResNet example given in the repository but I am struggling to induce equivariance and getting only invariance through that.

R2Conv breaks with kernel_size=2 (list index out of range)

This looks like a really nice library and I want to test it out, so I am converting an existing neural network to a rotational invariant one.

In one of my layers, I have a convolution with kernel_size=2. This breaks in the R2Conv while kernel_size=3 or more works fine.

Is there a reason why a kernel_size=2 convolution cannot work in the rotational setting or is this a bug?

Minimal example:

    r2_act = gspaces.Rot2dOnR2(N=8)
    in_type = nn.FieldType(r2_act, 3*[r2_act.trivial_repr])
    out_type = nn.FieldType(r2_act, 6*[r2_act.regular_repr])

    conv = nn.R2Conv(in_type, out_type, kernel_size=2, initialize=False, padding=0).cuda()
    x = torch.ones(size=[1,3,190,190]).cuda()
    x = nn.GeometricTensor(x, in_type)
    conv(x)

Error:

list index out of range in basisexpansion_blocks.py in line 327

Auswahl_451

Error when running model.py, something to do with RandomRotation?

When running model.ipynb, I get the error shown below when training, in box [9]:

When commenting out the RandomRotation in the train_transform function, there are no errors and I able to run the model without errors.

Any idea what could be wrong?

And thanks a lot for this really nice implementation and tutorial!


TypeError Traceback (most recent call last)
in
2 for epoch in range(51):
3 model.train()
----> 4 for i, (x, t) in enumerate(train_loader):
5 if i > 20:
6 break

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in next(self)
343
344 def next(self):
--> 345 data = self._next_data()
346 self._num_yielded += 1
347 if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
383 def _next_data(self):
384 index = self._next_index() # may raise StopIteration
--> 385 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
386 if self._pin_memory:
387 data = _utils.pin_memory.pin_memory(data)

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in (.0)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

in getitem(self, index)
21 image = Image.fromarray(image)
22 if self.transform is not None:
---> 23 image = self.transform(image)
24 return image, label
25

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
68 def call(self, img):
69 for t in self.transforms:
---> 70 img = t(img)
71 return img
72

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
1001 angle = self.get_params(self.degrees)
1002
-> 1003 return F.rotate(img, angle, self.resample, self.expand, self.center, self.fill)
1004
1005 def repr(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in rotate(img, angle, resample, expand, center, fill)
727 fill = tuple([fill] * 3)
728
--> 729 return img.rotate(angle, resample, expand, center, fillcolor=fill)
730
731

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in rotate(self, angle, resample, expand, center, translate, fillcolor)
1913
1914 return self.transform((w, h), AFFINE, matrix, resample,
-> 1915 fillcolor=fillcolor)
1916
1917 def save(self, fp, format=None, **params):

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in transform(self, size, method, data, resample, fill, fillcolor)
2203 raise ValueError("missing method data")
2204
-> 2205 im = new(self.mode, size, fillcolor)
2206 if method == MESH:
2207 # list of quads

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in new(mode, size, color)
2373 color = ImageColor.getcolor(color, mode)
2374
-> 2375 return Image()._new(core.fill(mode, size, color))
2376
2377

TypeError: must be real number, not tuple

wrapping pytorch operations - grid_sample

Hello,

Thank you for your nice work!

I'm about to use the e2cnn based network for my project.
I need to use the method grid_sample in PyTorch and its underlying implementation is just some interpolation.
https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample
I noticed the upsample in this repo simply computes the value and reassigns the output type.
https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/nn/modules/r2upsampling.py#L86
I think I can do exactly like this with grid_sample.
Please let me know if there will be any issues with this.

Ahyun

concatenation operation

Hello, I am looking for something equivalent to torch.cat, but takes GeometricTensors as inputs and outputs another GeometricTensor. I am developing a rotation-equivariant version of U-net and I need something equivalent to torch.cat for the skip connections. I haven't been able to find something inside the E2CNN library yet that seems to do this.

Thanks again for a really nice framework, I credited it in a recent publication.

Slow model instantiation

Thank you for a very well-crafted and important package. I have been using this package to test a rotation-equivariant UNet type network. I already see VERY good results; the standard UNet has 92 % accuracy on our dataset, whereas the most naive rotation-equivariant implementation achieves 99 %(!)

I would deploy this model right now if it was faster to instantiate: It currently takes 16 seconds, not including loading parameters and moving the model to GPU (which takes less than a second).

I can make some changes to our existing software to make this work and the accuracy improvements would certainly be worth it; however I am hoping for a way to make instantiation a bit faster.

Thanks,
Jacob

Learning of kernels

Thank you for this wonderful implementation of equivariant models. I'm currently using them to investigate segmentation problems. I have a question, which I'm really cannot find a clear answer to. What is the difference between the parameters weights and filter in R2Conv?
I'm confused as to how to connect the terms: if a regular square kernel is sampled on a PolarBasis, we obtain rotated versions of the filter. These are in turn used to in R2Conv. Does that mean, they are learnt directly in the back propagation phase?

About the equivalence of wide_resnet

Dear Author:
I change the wide_resnet by deleting the pooling and linear operation after out=out.tensor:

   def forward(self, x):

        # wrap the input tensor in a GeometricTensor
        x = enn.GeometricTensor(x, self.in_type)
        
        out = self.conv1(x)
        out = self.layer1(out)
        
        out = self.layer2(self.restrict1(out))
        
        out = self.layer3(self.restrict2(out))
        
        out = self.bn(out)
        out = self.relu(out)
        
        # extract the tensor from the GeometricTensor to use the common Pytorch operations
        out = out.tensor
        
        # b, c, w, h = out.shape
        # out = F.avg_pool2d(out, (w, h))
        
        # out = out.view(out.size(0), -1)
        # out = self.linear(out)
        return out

But find that the equivalence does not exist anymore:

    print()
    print('TESTING INVARIANCE:                    ')
    print('REFLECTIONS along the VERTICAL axis:   ' + (str(torch.norm(y-y_fv))))
    print('REFLECTIONS along the HORIZONTAL axis: ' + (str(torch.norm(y-y_fh))))
    print('90 degrees ROTATIONS:                  ' + (str(torch.norm(y-y90))))
    print('REFLECTIONS along the 45 degrees axis: ' + (str(torch.norm(y-y90_fh))))

The output is:

REFLECTIONS along the VERTICAL axis:   tensor(1.9236, grad_fn=<CopyBackwards>)
REFLECTIONS along the HORIZONTAL axis: tensor(2.0343, grad_fn=<CopyBackwards>)
90 degrees ROTATIONS:                  tensor(2.0621, grad_fn=<CopyBackwards>)
REFLECTIONS along the 45 degrees axis: tensor(2.0622, grad_fn=<CopyBackwards>)

May I have a ask about the reason?

MaskModule

Hi all,
Thank you very much for your amazing framework!
I would like to ask a few things. Since I am trying to build an auto-encoder for image segmentation based on your framework, I was wondering about how and if I should use your mask module.
I have seen your example and read the documentation for the mask module. My question is, should I use it often? I have many blocks of sequential layers with different operations like convolutions, pooling, upsampling etc.
Do you think it's necessary to add a lot of them to maintain rotation equivariance?

Thank you.

Legacy 3.6 branch UnicodeDecodeError during installation

When running python setup.py install on the legacy_3.6 branch, I'm getting:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-pv1d_p_r/e2cnn/setup.py", line 19, in <module>
    long_description = f.read()
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 6561: ordinal not in range(128)

If I hotfix this by setting long_description = "", the installation works.

(Python 3.6.9)

Feature request: e2cnn equivalent to torch.nn.ModuleList

As a minor convenience, it would be nice to have an equivariant equivalent to torch.nn.ModuleList.

The main use cases:

  • Same as ModuleList, contain modules that are (elsewhere) hooked up in some arbitrary DAG, rather than Sequential, and still have them get registered properly.
  • Support .export() to a torch.nn.ModuleList containing .exported() children.
  • Aid mechanical 1:1 porting from pytorch to e2cnn equivalents.

It would not subclass EquivariantModule, since it can't meaningfully conform to most of that API, but it would probably get used mostly in EquivariantModule subclass implementations.

I'm currently trying out the following pattern in my code:

# Inside a bigger class that is itself implementing .export()... with self.up_path also a ModuleList...
def export(self):
        self.eval()
        up_path_exported = torch.nn.ModuleList()
        for module in self.up_path:
            up_path_exported.append(module.export())

If the EquivariantModules were already living in a e2cnn.nn.ModuleList, it would just be:

def export(self):
    self.eval()
    up_path_exported = self.up_path.export()

So a couple lines of boilerplate would be saved, but the discovery aspect is probably worth more... "just build everything using e2cnn 1:1 equivalents, train, then .export()"

Feature Request: Slicing of GeometricTensors

I'm porting over this function to e2cnn:

    def center_crop(self, layer:torch.Tensor, target_size:List[int]):
        _, _, layer_height, layer_width = layer.size()
        diff_y = (layer_height - target_size[0]) // 2
        diff_x = (layer_width - target_size[1]) // 2
        return layer[
            :, :, diff_y : (diff_y + target_size[0]), diff_x : (diff_x + target_size[1])
        ]

When I pass a GeometricTensor into it, I hit "TypeError: 'GeometricTensor' object is not subscriptable" where the tensor is sliced.

That's probably too much info. I only need to slice along the spatial dimensions... I will experiment with casting back to torch.Tensor, but please let me know if you have a better workaround or update up your sleave. Thanks!

Module export

I'm working on creating an equivariant network for classification task. Based on the example available here, it seems that only Equivariant module can be exported to pytorch. As shown below, an assertion error occurs as torch.nn.Linear is not an Equivariant module.

spc = e2cnn.gspaces.Rot2dOnR2(8)
in_ = e2cnn.nn.FieldType(s, [s.trivial_repr])
out_ = e2cnn.nn.FieldType(s, [s.regular_repr]*16)

net = SequentialModule(
    R2Conv(in_, out_, 3, bias=False),
    ReLU(out_, inplace=True),
    PointwiseMaxPool(out_, kernel_size=2, stride=2),
    GroupPooling(out_),
    torch.nn.Linear(in_channels, out_channels)
)

Is there another way in which, the entire net can be exported to pytorch after training?

Equivariance and Image Resolution

Hi Gabriele, thank you very much for the great library!

Is equivariance guaranteed for images of arbitrary sizes/resolutions? It seems like equivariance is satisfied only for certain sizes. As an example, the C4-invariant 10-4 Wide ResNet defined and tested in examples/e2wrn.py seems to only be invariant to 90-degree rotations for images with widths of the form 4k+3.

To exemplify this, the output of this snippet

m = Wide_ResNet(10, 4, 0.3, initial_stride=1, N=4, f=False, r=0, num_classes=10)
m.eval()
for img_size in range(30, 50, 1):
    x = torch.randn([1, 3, 30, img_size])
    xrot = torch.rot90(x, k=1, dims=[2, 3])
    with torch.no_grad():
        z = m(x)
        zrot = m(xrot)
        print(f'Image size is 30x{img_size}. Equivariant:' +\
              ('YES' if torch.allclose(z, zrot, atol=1e-6) else 'NO'))

is

Image size is 30x30. Equivariant: NO
Image size is 30x31. Equivariant: NO
Image size is 30x32. Equivariant: NO
Image size is 30x33. Equivariant: YES
Image size is 30x34. Equivariant: NO
Image size is 30x35. Equivariant: NO
Image size is 30x36. Equivariant: NO
Image size is 30x37. Equivariant: YES
Image size is 30x38. Equivariant: NO
Image size is 30x39. Equivariant: NO
Image size is 30x40. Equivariant: NO
Image size is 30x41. Equivariant: YES
Image size is 30x42. Equivariant: NO
Image size is 30x43. Equivariant: NO
Image size is 30x44. Equivariant: NO
Image size is 30x45. Equivariant: YES
Image size is 30x46. Equivariant: NO
Image size is 30x47. Equivariant: NO
Image size is 30x48. Equivariant: NO
Image size is 30x49. Equivariant: YES.

This seems like an artifact caused by dilation/pooling but I am not sure whether there's any other factor causing this behaviour. Is this the case?

Lastly, would we observe a similar behaviour if we are working with field types composed of irreps of SO(2)? I don't have an intuition on how dilation is going to interact with guaranteeing equivariance there.

Thanks in advance!

O(2) group, irreps, and PyTorch DDP.

Hello,

Thank you for your nice work. I'm a heavy user of this library :)
Recently I'm working on O(2) groups and I have some questions about irreps.
I'm following your approach here to use every possible irreps in the fibergroup.

irreps = []
for n, irr in gc.fibergroup.irreps.items():
    if n != gc.trivial_repr.name:
        irreps += [irr] * int(irr.size // irr.sum_of_squares_constituents)
irreps = list(irreps)

What I can see is that the number of the irreps is independent to the maximum_frequency assigned when building a gspace.
My questions are

  1. What are the physical meanings of frequency / maximum_frequency?
  2. What is the difference between the different irreps items?
  3. Is there any reason for doing the above practice, rather than using specific types of irreps?

In addition, I have a minor bug to report related to the code snippet above.
In short: PyTorch DDP does not support them.
Here are the reasons and my solution.

  • DDP requires that each distributed process has the exact same model and thus the exact same parameter registration order.
  • The layers like GNorm register the parameter by iterating the unique_representations.
  • The unique representations in FieldType are implemented using set(), thus the order is non-deterministic.
  • If I run DDP, the orders of the parameters shuffle as the unique_representation of the fieldType generated with multiple irreps are not consistent across different processes, thus the run fails.

Simply fixed by changing 'field_type.py' as following, then install from the source (pip install .)

# self._unique_representations = set(self.representations)
_unique_representations = list(dict.fromkeys(self.representations).keys())
self._unique_representations = _unique_representations

I'm not raising a pull request because there might be any unexpected side effects.

Best,

Ahyun Seo

`SO(2)`-equivariant network for ImageNet

Hi Gabriele, thanks again for the great library!

I apologize in advance if the answer to this can be found in the docs/paper/thesis, I couldn't find it there.

My question is rather soft: what is a "reasonable" SO(2)-equivariant architecture? E.g. what would be the "proper" way to generalize the Wide ResNets in examples/e2wrn.py to be (approximately) equivariant wrt SO(2) rather than CN?

My attempt so far is:

  • set N=-1 with max_frequency=10 (leaving default value)
  • change instances of e2cnn.nn.InnerBatchNorm with e2cnn.nn.NormBatchNorm
    • when making an invariant model and using gspace.trivial_repr for the output of the last layer, I have used InnerBatchNorm for the corresponding BatchNorm
  • remove non-linearities (instances of e2cnn.nn.ReLU in this case)
  • use field type of the form e2cnn.nn.FieldType(gspace, [gspace.irrep(1)] * a + [gspace.irrep(2)] * b + [gspace.irrep(3)] * c) for some positive integers a, b, and c.

This gist contains my attempt to make a SO(2)-invariant ResNet-18. An equivariance test, however, shows that I am rather off. In particular, the output of

model.cuda()
model.eval()
img_size = 221
repeats = 500
rot_diffs = []
rand_diffs = []

for _ in range(repeats):
    x = torch.randn([1, 3, img_size, img_size]).cuda()
    xrot = torch.rot90(x, k=1, dims=[2, 3])
    xrand = torch.randn([1, 3, img_size, img_size]).cuda()

    with torch.no_grad():
        z, lat = model(x, with_latent=True)
        zrot, latrot = model(xrot, with_latent=True)
        zrand, latrand = model(xrand, with_latent=True)

    rot_diffs.append(torch.norm(z - zrot))
    rand_diffs.append(torch.norm(z - zrand))

print(f'l2 diff between logits of image and its 90-degree rotation: {torch.mean(torch.stack(rot_diffs)).item():.3f}')
print(f'l2 diff between logits of two arbitrary images: {torch.mean(torch.stack(rand_diffs)).item():.3f}')

is

l2 diff between logits of image and its 90-degree rotation: 0.617
l2 diff between logits of two arbitrary images: 0.739

What is going wrong here?

In case this is helpful, this is the model "topology":

MODEL TOPOLOGY:
	0 - 
	1 - conv1
	2 - conv1._basisexpansion
	3 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_1')
	4 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_2')
	5 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_3')
	6 - relu1
	7 - bn1
	8 - maxpool
	9 - layer1
	10 - layer1.0
	11 - layer1.0.conv1
	12 - layer1.0.conv1._basisexpansion
	13 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
	14 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_2')
	15 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_3')
	16 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_1')
	17 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
	18 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_3')
	19 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_1')
	20 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_2')
	21 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
	22 - layer1.0.bn1
	23 - layer1.0.conv2
	24 - layer1.0.conv2._basisexpansion
	25 - layer1.0.bn2
	26 - layer1.1
	27 - layer1.1.conv1
	28 - layer1.1.conv1._basisexpansion
	29 - layer1.1.bn1
	30 - layer1.1.conv2
	31 - layer1.1.conv2._basisexpansion
	32 - layer1.1.bn2
	33 - layer2
	34 - layer2.0
	35 - layer2.0.conv1
	36 - layer2.0.conv1._basisexpansion
	37 - layer2.0.bn1
	38 - layer2.0.conv2
	39 - layer2.0.conv2._basisexpansion
	40 - layer2.0.bn2
	41 - layer2.0.shortcut
	42 - layer2.0.shortcut.0
	43 - layer2.0.shortcut.0._basisexpansion
	44 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
	45 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
	46 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
        ...
        (layers 2.1 to 4.0 only change multiplicities in field types, no 'block_expansion_'s here) 
        ...
	92 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_1', 'irrep_0')
	93 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_2', 'irrep_0')
	94 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_3', 'irrep_0')
	95 - layer4.1.relu2
	96 - layer4.1.bn2
	97 - layer4.1.bn2.batch_norm_[1]
	98 - linear
	99 - avgpool

Thanks in advance!

about attribute R2conv.filter

Thanks for your work. I met a problem: I use R2conv in my network, I found when training R2conv module does not have attribute filter, but when testing, R2conv module has attribute 'filter', could you help me with this problem?Why does this happen

Cannot pass weights of R2Conv as a positional argument

Hi there. Many thanks for this amazing library and the whole equivariant framework, it's a very inspiring work!

I think the title of the issue pretty much sums it all up. I am trying to pass 2 positional arguments as input to an R2Conv module, both the input data and also the weight parameters.

My issue is that I get an error saying that 3 positional arguments were given when 2 were expected.
The standard conv2D module of PyTorch allows this, so I was wondering if it something wrong in my implementation or in general this behavior cannot be used for an R2Conv module.
Many thanks in advance,

Larger memory consumption and slower training speed

Hi!

First, I really appreciate your project, and it is very helpful in my project. However, I was a bit confused when I tried to compare vanilla resnet18 with e2resnet18. I found e2resnet18 consumed 4 times more GPU memory and 10 times training time. I wonder if I had done anything wrong, or it is just designed like that. Thank you very much!

Best,
Harold

Need a size parameter for e2cnn.R2Upsampling Class

Currently the e2cnn.R2Upsampling class only has four parameters: in_type, scale_factor, mode and align_corners. However, I think it would be helpful to add another "size" parameter, to be passed to the torch.nn.functional.interpolate method used inside the R2Upsampling class. Sometimes we need to up-sample the input to a specific dimension, especially when dealing with even and odd dimension issues

Very large model files when using equivariance

I'm experiencing some surprisingly large models when using equivariance. I am taking advantage of the (very convenient!) TrivialOnR2 fallback for comparisons been my non-equivariant and my equivariant model. This means all of the code is the same aside from the layer (and tensor) types.

Sizes of saved model files:

TrivialOnR2(): 18M
FlipRot(4): 733M

That's a 40X difference in size.
I get gpu OOM killed when I try to load it onto my local machine's 2GB gpu.

I was under the impression that there was no extra cost for equivariance at inference time. Even with a naive guess that you pay linearly for the equivariance, I would expect FlipRot(4) to be about 8X bigger.

I unzipped the model file (the size was unchanged, so I guess it wasn't actually compressed).

These are the largest files in the "data" subdirectory of the archive:

144.0 MiB [##########] 93996286070688
135.2 MiB [######### ] 93996315383952
103.6 MiB [####### ] 93996297647440
36.0 MiB [## ] 93996309406176
36.0 MiB [## ] 93996301356576
31.6 MiB [## ] 93996287309120
24.8 MiB [# ] 93996405222784
23.0 MiB [# ] 93996315305440
23.0 MiB [# ] 93996283837952
15.5 MiB [# ] 93996319486144
15.5 MiB [# ] 93996405222048
15.5 MiB [# ] 93996309417568

Running my model through (a slightly hacked) pytorch-model-summary:

For TrivialOnR2:

                Layer (type)         Input Shape         Param #     Tr. Param #

=====================================================================================
UNetConvBlock-1 1,856 1,856
PointwiseMaxPoolAntialiased-2 0 0
UNetConvBlock-3 9,856 9,856
PointwiseMaxPoolAntialiased-4 0 0
UNetConvBlock-5 44,288 44,288
PointwiseMaxPoolAntialiased-6 0 0
UNetConvBlock-7 186,880 186,880
PointwiseMaxPoolAntialiased-8 0 0
UNetConvBlock-9 766,976 766,976
UNetUpBlock-10 447,488 447,488
UNetUpBlock-11 125,440 125,440
UNetUpBlock-12 29,952 29,952
UNetUpBlock-13 6,784 6,784
R2Conv-14 208 208

Total params: 1,619,728
Trainable params: 1,619,728
Non-trainable params: 0

For FlipRot(4):
In [4]: s = summary(model, dummy_input, show_input=True, print_summary=True, show_hierarchical=False)

                Layer (type)         Input Shape         Param #     Tr. Param #

=====================================================================================
UNetConvBlock-1 12,608 12,608
PointwiseMaxPoolAntialiased-2 0 0
UNetConvBlock-3 74,368 74,368
PointwiseMaxPoolAntialiased-4 0 0
UNetConvBlock-5 345,344 345,344
PointwiseMaxPoolAntialiased-6 0 0
UNetConvBlock-7 1,477,120 1,477,120
PointwiseMaxPoolAntialiased-8 0 0
UNetConvBlock-9 6,099,968 6,099,968
UNetUpBlock-10 3,558,400 3,558,400
UNetUpBlock-11 992,768 992,768
UNetUpBlock-12 234,240 234,240
UNetUpBlock-13 51,584 51,584
R2Conv-14 208 208

Total params: 12,846,608
Trainable params: 12,846,608
Non-trainable params: 0

There actually ~are 8X more trainable parameters than with TrivialOnR2, so I was probably doing an unfair comparison; My TrivialOnR2 should get 8X more channels.

So the parameters take up 51 MB, leaving 682 MB of unexplained size. Do you have a guess what mistake I might have made to be causing that bloat? Thanks!

equivariant Transformer

hi~i want to ask if i can build rotated-equivariant swin transformer/Transformer anchitectural in E2CNN library?
thank you!!!

Import Error with Torch 1.9.0+cu111

Error while import with Pytorch version 1.9.0+cu111 (maybe earlier)

----> 6 from e2cnn import nn as e2nn
      7 
      8 import utils

/venv/lib/python3.8/site-packages/e2cnn/nn/__init__.py in <module>
      3 from .geometric_tensor import GeometricTensor, tensor_directsum
      4 
----> 5 from .modules import *
      6 
      7 

/venv/lib/python3.8/site-packages/e2cnn/nn/modules/__init__.py in <module>
     48 
     49 from .sequential_module import SequentialModule
---> 50 from .module_list import ModuleList
     51 from .identity_module import IdentityModule
     52 

/venv/lib/python3.8/site-packages/e2cnn/nn/modules/module_list.py in <module>
      2 from .equivariant_module import EquivariantModule
      3 
----> 4 from torch._six import container_abcs
      5 
      6 import torch

ImportError: cannot import name 'container_abcs' from 'torch._six' (/home/s51972mb/Documents/EquivariantSelfAttention/venv/lib/python3.8/site-packages/torch/_six.py)

Potential fix:
NVIDIA/apex#1049 (comment)

GeometricTensor.split(None) raises exception

Hi,

First off, thanks a lot for this nice package!

I am trying to split a regular field into all of its field components. According to GeometricTensor.split this can be done via x.split(None), which however errors out:

import torch
import e2cnn

r2_act = e2cnn.gspaces.Rot2dOnR2(N=12)

feat_type_in = e2cnn.nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = e2cnn.nn.FieldType(r2_act, [r2_act.regular_repr])

conv = e2cnn.nn.R2Conv(feat_type_in, feat_type_out, kernel_size=3, padding=1)

x = e2cnn.nn.GeometricTensor(torch.randn((1,1,32,32)), feat_type_in)
x = conv(x)

print(x.tensor.shape)

x.split(None)


>>> AssertionError: Error! "breaks" must be an increasing list of positive indexes

Am I doing something wrong here?

RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

I believe that the package needs to be updated for new versions of PyTorch. The code works with PyTorch 1.1, but it does not work for PyTorch 1.8.

This issue can be reproduced via the use of another package:

  1. git clone https://github.com/AllanYangZhou/metalearning-symmetries.git
  2. python generate_synthetic_data.py --problem 2d_rot8

Another issue is the support of machine without CUDA. To reproduce this issue, execute the following two steps on a CPU-only machine:

  1. git clone https://github.com/AllanYangZhou/metalearning-symmetries.git
  2. python generate_synthetic_data.py --problem 2d_rot8_flip

Rotated MNIST example fails if the device is 'cuda'

Fix:
cell 7:
instead of
x_transformed.to(device)
this:
x_transformed = x_transformed.to(device)

cell 9:
instead of:
loss = loss_function(y, t)
this:
loss = loss_function(y, t.to(device=device))

instead of:
correct += (prediction == t).sum().item()
this:
correct += (prediction == t.to(device=device)).sum().item()

Number of parameter changing when changing dilation

I happened to observe that when I change the dilation value, the number of parameters change. This is not the case with the standard torch.nn.Conv2D module. Is there any specific reason it happens in e2cnn.nn.R2Conv.

If this behaviour is expected, can you please direct me to the right resource.

Environment:

Python=3.7.9
torch=1.7.1
e2cnn=0.1.5

Code to reproduce issue


import torch
import torch.nn as nn
import torch.nn.functional as F

import numpy as np
import math

import e2cnn
import e2cnn.nn as enn
from e2cnn.nn import init
from e2cnn import gspaces   


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        N = 8
        self.gspace = gspaces.Rot2dOnR2(N)
        self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
        self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
        self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
                      stride=1,
                      padding=1,
                      dilation=1,
                      bias=True,
                      )
        self.invariant = enn.GroupPooling(self.out_type)
    def forward(self, x):
        x = enn.GeometricTensor(x, self.in_type)
        out = self.layer(x)
        out = self.invariant(out)
        out = out.tensor
        return out

class ModelDilated(nn.Module):
    def __init__(self):
        super(ModelDilated, self).__init__()
        N = 8
        self.gspace = gspaces.Rot2dOnR2(N)
        self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
        self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
        self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
                      stride=1,
                      padding=2,
                      dilation=2,
                      bias=True,
                      )
        self.invariant = enn.GroupPooling(self.out_type)
    def forward(self, x):
        x = enn.GeometricTensor(x, self.in_type)
        out = self.layer(x)
        out = self.invariant(out)
        out = out.tensor
        return out


if __name__=="__main__":
    m = Model()
    md = ModelDilated()
    ip = torch.randn(1,3,100,100)
    op1 = m(ip)
    op2 = md(ip)

    totalParams = sum(p.numel() for p in m.parameters())
    totalParams2 = sum(p.numel() for p in md.parameters())
    print(totalParams, totalParams2)
    print(op1.shape, op2.shape)

Output

304 400
torch.Size([1, 16, 100, 100]) torch.Size([1, 16, 100, 100])

Result of equivariance test changes after training model

Hi, thanks again for a very nice package and your support of it.

When I am running my own architecture, there seems to be some kind of problem with its equivariance properties after training.

I am using my own dataset contaning biomedical images, and the classifier needs to classify cancer being present or not. So therefore there are two classes. When I run the test function on an image on the test set, I get these results:

#############################################################################

angle | 0 1 2 3 4 5 6 7 8 9
0 : [-0.0123 -0.2154]
90 : [-0.0118 -0.2155]
180 : [-0.0126 -0.2156]
270 : [-0.012 -0.2157]
#############################################################################

So the network has a high degree of equivariance.

Then I train my model, run the same test function again and I get these results:

#############################################################################
angle | 0 1 2 3 4 5 6 7 8 9
0 : [-1.6111 1.8046]
90 : [-1.0136 1.043 ]
180 : [-1.3092 1.5487]
270 : [-1.5571 1.8682]
#############################################################################

Meaning very different outputs even though I use the C4 group. Any insight into what could be happening to 'break' the equivariance properties of the network?

Compatibility with Scale Equivariance

Thank you for this great implementation of equivariant CNNs. I noticed that in your paper and in this implementation, while broad groups such as O(2) are mentioned, only transformations of translation/rotation/reflection are explicitly discussed.

In light of new work on scale-equivariant CNNs, such as this one, would scale-equivariance also be covered under O(2), since the origin is preserved under scaling? If so, how could scale-equivariance be incorporated in tandem with the transformations covered in this library?

I would appreciate any tips or thoughts you might have on this subject.

Feature Request: Equivariant Downsampling and Upsampling

I would like to build rotation equivariant networks in the UNet family, but it seems downsampling and upsampling are missing necessary building blocks.

For continuous signals it seems pretty obvious that downsampling and upsampling operators can be defined that are equivariant. i.e. if an image is downscaled by 2, then rotations happen by the same angle, translations happen in the same direction but half as far. Similar story for upscaling but translations are doubled. Both operators are lossless in an idealized infinite continuous 2d plane setting.

For discrete signals it should be possible too up to numerics, although you have to be careful to use downsampling and upsampling kernels that are (approximately) rotation symmetric, aka "radial".

Pytorch already has down and upsampling operators; maybe it's just a matter of wrapping them correctly in your framework.

FieldType.transform doesn't work on GPU tensors

When I transform a tensor that's on a GPU (using FieldType.transform or GeometricTensor.transform), I get TypeError: can't convert cuda:0 device type tensor to numpy.
The issue is that in this line, it would have to be input.detach().cpu().numpy() instead of input.detach().numpy().

Only changing that line is probably bad because it would return a tensor on CPU when a GPU tensor is fed in. You could store the device of the tensor, move the tensor to CPU, and at the end return a new tensor on the original device. Of course, moving the tensor implicitly like that isn't ideal as well because it might be a performance hit without the user being aware of it. Alternatively, maybe just document this and raise a custom Exception ("tensors need to be moved to cpu before transforming")? Not sure which solution is better.

pip install error with the legacy_py3.6 branch

Hi developers,

I have set up a python environment with python 3.6.9.

When I install the package,
pip install .

I encounter the following error
ERROR: Command errored out with exit status 1: command: /home/kaikai/anaconda3/envs/python3.6.9/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-iki7taiq cwd: /tmp/pip-req-build-nx2wsig7/ Complete output (2 lines): running egg_info error: Invalid distribution name or version syntax: e2cnn-py3.6--0.1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It turns out that parentheses are invalid characters in the "name" of setuptools.

I have verified if I change the line in e2cnn/__about__.py
from
__title__ = "e2cnn (py3.6)"
to
__title__ = "e2cnn-py3.6",
the installation will work.

Would you help take a look at this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.