quva-lab/e2cnn

E(2)-Equivariant CNNs Library for Pytorch

e2cnn's Issues

A question about cuda acceleration

Is this package accelerated by cuda?

Import Error with Torch 1.9.0+cu111

Error while import with Pytorch version 1.9.0+cu111 (maybe earlier)

----> 6 from e2cnn import nn as e2nn
      7 
      8 import utils

/venv/lib/python3.8/site-packages/e2cnn/nn/__init__.py in <module>
      3 from .geometric_tensor import GeometricTensor, tensor_directsum
      4 
----> 5 from .modules import *
      6 
      7 

/venv/lib/python3.8/site-packages/e2cnn/nn/modules/__init__.py in <module>
     48 
     49 from .sequential_module import SequentialModule
---> 50 from .module_list import ModuleList
     51 from .identity_module import IdentityModule
     52 

/venv/lib/python3.8/site-packages/e2cnn/nn/modules/module_list.py in <module>
      2 from .equivariant_module import EquivariantModule
      3 
----> 4 from torch._six import container_abcs
      5 
      6 import torch

ImportError: cannot import name 'container_abcs' from 'torch._six' (/home/s51972mb/Documents/EquivariantSelfAttention/venv/lib/python3.8/site-packages/torch/_six.py)

Potential fix:
NVIDIA/apex#1049 (comment)

Feature request: e2cnn equivalent to torch.nn.ModuleList

As a minor convenience, it would be nice to have an equivariant equivalent to torch.nn.ModuleList.

The main use cases:

• Same as ModuleList, contain modules that are (elsewhere) hooked up in some arbitrary DAG, rather than Sequential, and still have them get registered properly.
• Support .export() to a torch.nn.ModuleList containing .exported() children.
• Aid mechanical 1:1 porting from pytorch to e2cnn equivalents.

It would not subclass EquivariantModule, since it can't meaningfully conform to most of that API, but it would probably get used mostly in EquivariantModule subclass implementations.

I'm currently trying out the following pattern in my code:

# Inside a bigger class that is itself implementing .export()... with self.up_path also a ModuleList...
def export(self):
        self.eval()
        up_path_exported = torch.nn.ModuleList()
        for module in self.up_path:
            up_path_exported.append(module.export())

If the EquivariantModules were already living in a e2cnn.nn.ModuleList, it would just be:

def export(self):
    self.eval()
    up_path_exported = self.up_path.export()

So a couple lines of boilerplate would be saved, but the discovery aspect is probably worth more... "just build everything using e2cnn 1:1 equivalents, train, then .export()"

Larger memory consumption and slower training speed

Hi!

First, I really appreciate your project, and it is very helpful in my project. However, I was a bit confused when I tried to compare vanilla resnet18 with e2resnet18. I found e2resnet18 consumed 4 times more GPU memory and 10 times training time. I wonder if I had done anything wrong, or it is just designed like that. Thank you very much!

Best,
Harold

R2Conv acts on 5-D tensor

There is a situation when images are stacked twice. The shape of the structured image tensor is (B, S, C, H, W) where B represents batch size and S the number of stacked images.
Could R2conv directly act or broadcast on such a 5-D tensor?

AttributeError: Can't pickle local object 'DihedralGroup.irrep.<locals>.<lambda>'

Hi!
I found your package quite helpful! However, when I want to save the model or model.state_dict with torch.save, it failed with the error: AttributeError: Can't pickle local object 'DihedralGroup.irrep..'. I wonder if you have any solution for this? Thank you a lot!

Need a size parameter for e2cnn.R2Upsampling Class

Currently the e2cnn.R2Upsampling class only has four parameters: in_type, scale_factor, mode and align_corners. However, I think it would be helpful to add another "size" parameter, to be passed to the torch.nn.functional.interpolate method used inside the R2Upsampling class. Sometimes we need to up-sample the input to a specific dimension, especially when dealing with even and odd dimension issues

SequentialModule instead of Sequential

The doc states usage as "e2cnn.nn.Sequential(...)", but it should be "e2cnn.nn.SequentialModule(...)"

e2cnn/e2cnn/nn/modules/sequential_module.py

Line 30 in 30c5ff7

model = e2cnn.nn.Sequential(

Init of Field Dropout

When creating a FieldDropout Layer, I get the following error: TypeError: cannot assign 'torch.LongTensor' object to parameter 'indices_8' (torch.nn.Parameter or None required). Initializing PointwiseDropout with the same parameters works.

Is this a known issue?

about attribute R2conv.filter

Thanks for your work. I met a problem: I use R2conv in my network, I found when training R2conv module does not have attribute filter, but when testing, R2conv module has attribute 'filter', could you help me with this problem?Why does this happen

Result of equivariance test changes after training model

Hi, thanks again for a very nice package and your support of it.

When I am running my own architecture, there seems to be some kind of problem with its equivariance properties after training.

I am using my own dataset contaning biomedical images, and the classifier needs to classify cancer being present or not. So therefore there are two classes. When I run the test function on an image on the test set, I get these results:

#############################################################################

angle | 0 1 2 3 4 5 6 7 8 9
0 : [-0.0123 -0.2154]
90 : [-0.0118 -0.2155]
180 : [-0.0126 -0.2156]
270 : [-0.012 -0.2157]
#############################################################################

So the network has a high degree of equivariance.

Then I train my model, run the same test function again and I get these results:

#############################################################################
angle | 0 1 2 3 4 5 6 7 8 9
0 : [-1.6111 1.8046]
90 : [-1.0136 1.043 ]
180 : [-1.3092 1.5487]
270 : [-1.5571 1.8682]
#############################################################################

Meaning very different outputs even though I use the C4 group. Any insight into what could be happening to 'break' the equivariance properties of the network?

concatenation operation

Hello, I am looking for something equivalent to torch.cat, but takes GeometricTensors as inputs and outputs another GeometricTensor. I am developing a rotation-equivariant version of U-net and I need something equivalent to torch.cat for the skip connections. I haven't been able to find something inside the E2CNN library yet that seems to do this.

Thanks again for a really nice framework, I credited it in a recent publication.

ZeroPad2D on GeometricTensor

Is it possible to do ZeroPad2D on the geometric tensor? Specifically, I want my input image and output image to have the same dimension and I am using kernel size = 4. With the normal CNN, I can achieve this with pad = torch.nn.ZeroPad2d((2, 1, 2, 1)). How can I do this type of padding with equivariant CNNs?

wrapping pytorch operations - grid_sample

Hello,

Thank you for your nice work!

I'm about to use the e2cnn based network for my project.
I need to use the method grid_sample in PyTorch and its underlying implementation is just some interpolation.
https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample
I noticed the upsample in this repo simply computes the value and reassigns the output type.
https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/nn/modules/r2upsampling.py#L86
I think I can do exactly like this with grid_sample.
Please let me know if there will be any issues with this.

Ahyun

O(2) group, irreps, and PyTorch DDP.

Hello,

Thank you for your nice work. I'm a heavy user of this library :)
Recently I'm working on O(2) groups and I have some questions about irreps.
I'm following your approach here to use every possible irreps in the fibergroup.

irreps = []
for n, irr in gc.fibergroup.irreps.items():
    if n != gc.trivial_repr.name:
        irreps += [irr] * int(irr.size // irr.sum_of_squares_constituents)
irreps = list(irreps)

What I can see is that the number of the irreps is independent to the maximum_frequency assigned when building a gspace.
My questions are

1. What are the physical meanings of frequency / maximum_frequency?
2. What is the difference between the different irreps items?
3. Is there any reason for doing the above practice, rather than using specific types of irreps?

In addition, I have a minor bug to report related to the code snippet above.
In short: PyTorch DDP does not support them.
Here are the reasons and my solution.

• DDP requires that each distributed process has the exact same model and thus the exact same parameter registration order.
• The layers like GNorm register the parameter by iterating the unique_representations.
• The unique representations in FieldType are implemented using set(), thus the order is non-deterministic.
• If I run DDP, the orders of the parameters shuffle as the unique_representation of the fieldType generated with multiple irreps are not consistent across different processes, thus the run fails.

Simply fixed by changing 'field_type.py' as following, then install from the source (pip install .)

# self._unique_representations = set(self.representations)
_unique_representations = list(dict.fromkeys(self.representations).keys())
self._unique_representations = _unique_representations

I'm not raising a pull request because there might be any unexpected side effects.

Best,

Ahyun Seo

Very large model files when using equivariance

I'm experiencing some surprisingly large models when using equivariance. I am taking advantage of the (very convenient!) TrivialOnR2 fallback for comparisons been my non-equivariant and my equivariant model. This means all of the code is the same aside from the layer (and tensor) types.

Sizes of saved model files:

TrivialOnR2(): 18M
FlipRot(4): 733M

That's a 40X difference in size.
I get gpu OOM killed when I try to load it onto my local machine's 2GB gpu.

I was under the impression that there was no extra cost for equivariance at inference time. Even with a naive guess that you pay linearly for the equivariance, I would expect FlipRot(4) to be about 8X bigger.

I unzipped the model file (the size was unchanged, so I guess it wasn't actually compressed).

These are the largest files in the "data" subdirectory of the archive:

144.0 MiB [##########] 93996286070688
135.2 MiB [######### ] 93996315383952
103.6 MiB [####### ] 93996297647440
36.0 MiB [## ] 93996309406176
36.0 MiB [## ] 93996301356576
31.6 MiB [## ] 93996287309120
24.8 MiB [# ] 93996405222784
23.0 MiB [# ] 93996315305440
23.0 MiB [# ] 93996283837952
15.5 MiB [# ] 93996319486144
15.5 MiB [# ] 93996405222048
15.5 MiB [# ] 93996309417568

Running my model through (a slightly hacked) pytorch-model-summary:

For TrivialOnR2:

                Layer (type)         Input Shape         Param #     Tr. Param #

=====================================================================================
UNetConvBlock-1 1,856 1,856
PointwiseMaxPoolAntialiased-2 0 0
UNetConvBlock-3 9,856 9,856
PointwiseMaxPoolAntialiased-4 0 0
UNetConvBlock-5 44,288 44,288
PointwiseMaxPoolAntialiased-6 0 0
UNetConvBlock-7 186,880 186,880
PointwiseMaxPoolAntialiased-8 0 0
UNetConvBlock-9 766,976 766,976
UNetUpBlock-10 447,488 447,488
UNetUpBlock-11 125,440 125,440
UNetUpBlock-12 29,952 29,952
UNetUpBlock-13 6,784 6,784
R2Conv-14 208 208

Total params: 1,619,728
Trainable params: 1,619,728
Non-trainable params: 0

For FlipRot(4):
In [4]: s = summary(model, dummy_input, show_input=True, print_summary=True, show_hierarchical=False)

                Layer (type)         Input Shape         Param #     Tr. Param #

=====================================================================================
UNetConvBlock-1 12,608 12,608
PointwiseMaxPoolAntialiased-2 0 0
UNetConvBlock-3 74,368 74,368
PointwiseMaxPoolAntialiased-4 0 0
UNetConvBlock-5 345,344 345,344
PointwiseMaxPoolAntialiased-6 0 0
UNetConvBlock-7 1,477,120 1,477,120
PointwiseMaxPoolAntialiased-8 0 0
UNetConvBlock-9 6,099,968 6,099,968
UNetUpBlock-10 3,558,400 3,558,400
UNetUpBlock-11 992,768 992,768
UNetUpBlock-12 234,240 234,240
UNetUpBlock-13 51,584 51,584
R2Conv-14 208 208

Total params: 12,846,608
Trainable params: 12,846,608
Non-trainable params: 0

There actually ~are 8X more trainable parameters than with TrivialOnR2, so I was probably doing an unfair comparison; My TrivialOnR2 should get 8X more channels.

So the parameters take up 51 MB, leaving 682 MB of unexplained size. Do you have a guess what mistake I might have made to be causing that bloat? Thanks!

pip install error with the legacy_py3.6 branch

Hi developers,

I have set up a python environment with python 3.6.9.

When I install the package,
pip install .

I encounter the following error
ERROR: Command errored out with exit status 1: command: /home/kaikai/anaconda3/envs/python3.6.9/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-iki7taiq cwd: /tmp/pip-req-build-nx2wsig7/ Complete output (2 lines): running egg_info error: Invalid distribution name or version syntax: e2cnn-py3.6--0.1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It turns out that parentheses are invalid characters in the "name" of setuptools.

I have verified if I change the line in e2cnn/__about__.py
from
__title__ = "e2cnn (py3.6)"
to
__title__ = "e2cnn-py3.6",
the installation will work.

Would you help take a look at this?

equivariance in C8 space

Hi,

When I'm testing the equivariance for a sample example in C8 space following the "introduction.ipynb", I've observed it to fail. So, I wanted to check if there is a correct way of evaluating the c8 space equivariance? Further, It would help me if you could point out how could one go about defining an architecture such that equivariance could be observed across all angles of C8 space?

My code:

import torch

from e2cnn import gspaces
from e2cnn import nn


def conv3x3(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=1,
            dilation=1, bias=False):
    """3x3 convolution with padding"""
    return nn.R2Conv(in_type, out_type, 3,
                      stride=stride,
                      padding=padding,
                      dilation=dilation,
                      bias=bias,
                      sigma=None,
                      frequencies_cutoff=lambda r: 3*r,
                      )

def conv1x1(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=0,
            dilation=1, bias=False):
    """1x1 convolution with padding"""
    return nn.R2Conv(in_type, out_type, 1,
                      stride=stride,
                      padding=padding,
                      dilation=dilation,
                      bias=bias,
                      sigma=None,
                      frequencies_cutoff=lambda r: 3*r,
                      )


r2_act = gspaces.Rot2dOnR2(N=8)
feat_type_in = nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = nn.FieldType(r2_act, 3*[r2_act.regular_repr])

conv = conv3x3(feat_type_in, feat_type_out)

x = torch.randn(4, 1, 32, 32)
x = nn.GeometricTensor(x, feat_type_in)

assert isinstance(x.tensor, torch.Tensor)

y = conv(x)

assert y.type == feat_type_out

print(y.tensor.shape)


for g in r2_act.testing_elements:
    # transform the input with the current group element according to the input type
    x_transformed = x.transform(g)
    
    # feed the transformed input in the convolutional layer
    y_from_x_transformed = conv(x_transformed)
    
    # the result should be equivalent to rotating the output produced in the 
    # previous block according to the output type
    y_transformed_from_x = y.transform(g)
    
    assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g

print("success")

result:

Traceback (most recent call last):
  File "e2_test.py", line 62, in <module>
    assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g
AssertionError: 1

Feature Request: Code for generating equivariance visualizations

I really like your visualizations of rotation equivariance (i.e. vectorfield.gif and conventional_cnn.gif
), and would like to generate similar ones for my network. Is the code for generating those something that could be easily added, or should I go ahead write my own code for that?

FieldType.transform doesn't work on GPU tensors

When I transform a tensor that's on a GPU (using FieldType.transform or GeometricTensor.transform), I get TypeError: can't convert cuda:0 device type tensor to numpy.
The issue is that in this line, it would have to be input.detach().cpu().numpy() instead of input.detach().numpy().

Only changing that line is probably bad because it would return a tensor on CPU when a GPU tensor is fed in. You could store the device of the tensor, move the tensor to CPU, and at the end return a new tensor on the original device. Of course, moving the tensor implicitly like that isn't ideal as well because it might be a performance hit without the user being aware of it. Alternatively, maybe just document this and raise a custom Exception ("tensors need to be moved to cpu before transforming")? Not sure which solution is better.

Feature Request: Slicing of GeometricTensors

I'm porting over this function to e2cnn:

    def center_crop(self, layer:torch.Tensor, target_size:List[int]):
        _, _, layer_height, layer_width = layer.size()
        diff_y = (layer_height - target_size[0]) // 2
        diff_x = (layer_width - target_size[1]) // 2
        return layer[
            :, :, diff_y : (diff_y + target_size[0]), diff_x : (diff_x + target_size[1])
        ]

When I pass a GeometricTensor into it, I hit "TypeError: 'GeometricTensor' object is not subscriptable" where the tensor is sliced.

That's probably too much info. I only need to slice along the spatial dimensions... I will experiment with casting back to torch.Tensor, but please let me know if you have a better workaround or update up your sleave. Thanks!

Rotated MNIST example fails if the device is 'cuda'

Fix:
cell 7:
instead of
x_transformed.to(device)
this:
x_transformed = x_transformed.to(device)

cell 9:
instead of:
loss = loss_function(y, t)
this:
loss = loss_function(y, t.to(device=device))

instead of:
correct += (prediction == t).sum().item()
this:
correct += (prediction == t.to(device=device)).sum().item()

Compatibility with python 3.6

Is there a way I can use this library with python 3.6 as well?

Why the judgement is the padding_mode != "zeros" rather than == "zeros"?

e2cnn/e2cnn/nn/modules/r2_conv/r2convolution.py

Line 329 in 48f4976

if self.padding_mode != 'zeros':

Layout of weight tensors sampled from SteerableBasis

Hi admins, Thanks for the nice paper and handy code.

e2cnn/e2cnn/kernels/steerable_basis.py

Line 174 in 2604f67

out = np.zeros((self.shape[0], self.shape[1], self.dim, angles.shape[1]))

The weight tensor return by SteerableBasis.sample has the shape of out_channels x in_channels x dim x len(angles). dim is the summation of basis dimensions of all input/output representation pairs. I feel that this shape is redundant since each basis is only used by its corresponding input/output pair.

e2cnn/e2cnn/kernels/steerable_basis.py

Lines 180 to 192 in 2604f67

	for ii, in_size in enumerate(self.in_sizes):
	out_position = 0
	for oo, out_size in enumerate(self.out_sizes):
	if self.bases[ii][oo] is not None:
	dim = self.bases[ii][oo].dim
	block = out[
	out_position:out_position+out_size,
	in_position:in_position+in_size,
	basis_count:basis_count+dim,
	...
	]
	self.bases[ii][oo].sample(angles, out=block)

The sampling code also implies that the current weight tensor is very sparse by filling a 3D tensor with 2 for loops. I feel that theoretically out_channels x in_channels x 4 x len(angles) should be enough to save all sampled basis as a non-trivial irrep pair usually corresponds to 4 bases.

Could you please let me know if this interpretation is correct? Thanks!

GeometricTensor.split(None) raises exception

Hi,

First off, thanks a lot for this nice package!

I am trying to split a regular field into all of its field components. According to GeometricTensor.split this can be done via x.split(None), which however errors out:

import torch
import e2cnn

r2_act = e2cnn.gspaces.Rot2dOnR2(N=12)

feat_type_in = e2cnn.nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = e2cnn.nn.FieldType(r2_act, [r2_act.regular_repr])

conv = e2cnn.nn.R2Conv(feat_type_in, feat_type_out, kernel_size=3, padding=1)

x = e2cnn.nn.GeometricTensor(torch.randn((1,1,32,32)), feat_type_in)
x = conv(x)

print(x.tensor.shape)

x.split(None)


>>> AssertionError: Error! "breaks" must be an increasing list of positive indexes

Am I doing something wrong here?

equivariant Transformer

hi~i want to ask if i can build rotated-equivariant swin transformer/Transformer anchitectural in E2CNN library?
thank you!!!

Module export

I'm working on creating an equivariant network for classification task. Based on the example available here, it seems that only Equivariant module can be exported to pytorch. As shown below, an assertion error occurs as torch.nn.Linear is not an Equivariant module.

spc = e2cnn.gspaces.Rot2dOnR2(8)
in_ = e2cnn.nn.FieldType(s, [s.trivial_repr])
out_ = e2cnn.nn.FieldType(s, [s.regular_repr]*16)

net = SequentialModule(
    R2Conv(in_, out_, 3, bias=False),
    ReLU(out_, inplace=True),
    PointwiseMaxPool(out_, kernel_size=2, stride=2),
    GroupPooling(out_),
    torch.nn.Linear(in_channels, out_channels)
)

Is there another way in which, the entire net can be exported to pytorch after training?

Learning of kernels

Thank you for this wonderful implementation of equivariant models. I'm currently using them to investigate segmentation problems. I have a question, which I'm really cannot find a clear answer to. What is the difference between the parameters weights and filter in R2Conv?
I'm confused as to how to connect the terms: if a regular square kernel is sampled on a PolarBasis, we obtain rotated versions of the filter. These are in turn used to in R2Conv. Does that mean, they are learnt directly in the back propagation phase?

MaskModule

Hi all,
Thank you very much for your amazing framework!
I would like to ask a few things. Since I am trying to build an auto-encoder for image segmentation based on your framework, I was wondering about how and if I should use your mask module.
I have seen your example and read the documentation for the mask module. My question is, should I use it often? I have many blocks of sequential layers with different operations like convolutions, pooling, upsampling etc.
Do you think it's necessary to add a lot of them to maintain rotation equivariance?

Thank you.

Number of parameter changing when changing dilation

I happened to observe that when I change the dilation value, the number of parameters change. This is not the case with the standard torch.nn.Conv2D module. Is there any specific reason it happens in e2cnn.nn.R2Conv.

If this behaviour is expected, can you please direct me to the right resource.

Environment:

Python=3.7.9
torch=1.7.1
e2cnn=0.1.5

Code to reproduce issue

import torch
import torch.nn as nn
import torch.nn.functional as F

import numpy as np
import math

import e2cnn
import e2cnn.nn as enn
from e2cnn.nn import init
from e2cnn import gspaces   


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        N = 8
        self.gspace = gspaces.Rot2dOnR2(N)
        self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
        self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
        self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
                      stride=1,
                      padding=1,
                      dilation=1,
                      bias=True,
                      )
        self.invariant = enn.GroupPooling(self.out_type)
    def forward(self, x):
        x = enn.GeometricTensor(x, self.in_type)
        out = self.layer(x)
        out = self.invariant(out)
        out = out.tensor
        return out

class ModelDilated(nn.Module):
    def __init__(self):
        super(ModelDilated, self).__init__()
        N = 8
        self.gspace = gspaces.Rot2dOnR2(N)
        self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
        self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
        self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
                      stride=1,
                      padding=2,
                      dilation=2,
                      bias=True,
                      )
        self.invariant = enn.GroupPooling(self.out_type)
    def forward(self, x):
        x = enn.GeometricTensor(x, self.in_type)
        out = self.layer(x)
        out = self.invariant(out)
        out = out.tensor
        return out


if __name__=="__main__":
    m = Model()
    md = ModelDilated()
    ip = torch.randn(1,3,100,100)
    op1 = m(ip)
    op2 = md(ip)

    totalParams = sum(p.numel() for p in m.parameters())
    totalParams2 = sum(p.numel() for p in md.parameters())
    print(totalParams, totalParams2)
    print(op1.shape, op2.shape)

Output

304 400
torch.Size([1, 16, 100, 100]) torch.Size([1, 16, 100, 100])

Counting FLOPs for e2cnn

Hello,

Thank you for sharing this amazing work!

While playing it a little bit, I wonder how many flops the network consumes.

I found that the existing library (e.g., https://github.com/1adrianb/pytorch-estimate-flops) does not work for this as they don't support equivariant modules.

Is there any simple method for this one? If the counting for kernel construction parts would be hard, I just want to know how to translate the model into torch model as in test phase to count using the existing library.

Slow model instantiation

Thank you for a very well-crafted and important package. I have been using this package to test a rotation-equivariant UNet type network. I already see VERY good results; the standard UNet has 92 % accuracy on our dataset, whereas the most naive rotation-equivariant implementation achieves 99 %(!)

I would deploy this model right now if it was faster to instantiate: It currently takes 16 seconds, not including loading parameters and moving the model to GPU (which takes less than a second).

I can make some changes to our existing software to make this work and the accuracy improvements would certainly be worth it; however I am hoping for a way to make instantiation a bit faster.

Thanks,
Jacob

R2Conv breaks with kernel_size=2 (list index out of range)

This looks like a really nice library and I want to test it out, so I am converting an existing neural network to a rotational invariant one.

In one of my layers, I have a convolution with kernel_size=2. This breaks in the R2Conv while kernel_size=3 or more works fine.

Is there a reason why a kernel_size=2 convolution cannot work in the rotational setting or is this a bug?

Minimal example:

    r2_act = gspaces.Rot2dOnR2(N=8)
    in_type = nn.FieldType(r2_act, 3*[r2_act.trivial_repr])
    out_type = nn.FieldType(r2_act, 6*[r2_act.regular_repr])

    conv = nn.R2Conv(in_type, out_type, kernel_size=2, initialize=False, padding=0).cuda()
    x = torch.ones(size=[1,3,190,190]).cuda()
    x = nn.GeometricTensor(x, in_type)
    conv(x)

Error:

list index out of range in basisexpansion_blocks.py in line 327

Error when running model.py, something to do with RandomRotation?

When running model.ipynb, I get the error shown below when training, in box [9]:

When commenting out the RandomRotation in the train_transform function, there are no errors and I able to run the model without errors.

Any idea what could be wrong?

And thanks a lot for this really nice implementation and tutorial!

---

TypeError Traceback (most recent call last)
in
2 for epoch in range(51):
3 model.train()
----> 4 for i, (x, t) in enumerate(train_loader):
5 if i > 20:
6 break

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in next(self)
343
344 def next(self):
--> 345 data = self._next_data()
346 self._num_yielded += 1
347 if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
383 def _next_data(self):
384 index = self._next_index() # may raise StopIteration
--> 385 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
386 if self._pin_memory:
387 data = _utils.pin_memory.pin_memory(data)

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in (.0)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

in getitem(self, index)
21 image = Image.fromarray(image)
22 if self.transform is not None:
---> 23 image = self.transform(image)
24 return image, label
25

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
68 def call(self, img):
69 for t in self.transforms:
---> 70 img = t(img)
71 return img
72

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
1001 angle = self.get_params(self.degrees)
1002
-> 1003 return F.rotate(img, angle, self.resample, self.expand, self.center, self.fill)
1004
1005 def repr(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in rotate(img, angle, resample, expand, center, fill)
727 fill = tuple([fill] * 3)
728
--> 729 return img.rotate(angle, resample, expand, center, fillcolor=fill)
730
731

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in rotate(self, angle, resample, expand, center, translate, fillcolor)
1913
1914 return self.transform((w, h), AFFINE, matrix, resample,
-> 1915 fillcolor=fillcolor)
1916
1917 def save(self, fp, format=None, **params):

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in transform(self, size, method, data, resample, fill, fillcolor)
2203 raise ValueError("missing method data")
2204
-> 2205 im = new(self.mode, size, fillcolor)
2206 if method == MESH:
2207 # list of quads

~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in new(mode, size, color)
2373 color = ImageColor.getcolor(color, mode)
2374
-> 2375 return Image()._new(core.fill(mode, size, color))
2376
2377

TypeError: must be real number, not tuple

Compatibility with Scale Equivariance

Thank you for this great implementation of equivariant CNNs. I noticed that in your paper and in this implementation, while broad groups such as O(2) are mentioned, only transformations of translation/rotation/reflection are explicitly discussed.

In light of new work on scale-equivariant CNNs, such as this one, would scale-equivariance also be covered under O(2), since the origin is preserved under scaling? If so, how could scale-equivariance be incorporated in tandem with the transformations covered in this library?

I would appreciate any tips or thoughts you might have on this subject.

Feature Request: Equivariant Downsampling and Upsampling

I would like to build rotation equivariant networks in the UNet family, but it seems downsampling and upsampling are missing necessary building blocks.

For continuous signals it seems pretty obvious that downsampling and upsampling operators can be defined that are equivariant. i.e. if an image is downscaled by 2, then rotations happen by the same angle, translations happen in the same direction but half as far. Similar story for upscaling but translations are doubled. Both operators are lossless in an idealized infinite continuous 2d plane setting.

For discrete signals it should be possible too up to numerics, although you have to be careful to use downsampling and upsampling kernels that are (approximately) rotation symmetric, aka "radial".

Pytorch already has down and upsampling operators; maybe it's just a matter of wrapping them correctly in your framework.

State dict different in train and eval

Just a small problem I just encountered, currently a convolutional models state_dict in train mode is different to its state_dict in eval mode. Which results in a "Missing Keys" error when loading a state_dict saved during training into a model in eval mode. It is not critical though as loading with "strict=False" and then running model.eval() restores the filter and the expanded_bias.

e2cnn/e2cnn/nn/modules/r2_conv/r2convolution.py

Line 298 in 9792ed9

If ``mode=True``, sets the module in training mode and discards the :attr:`~e2cnn.nn.R2Conv.filter` and

`SO(2)`-equivariant network for ImageNet

Hi Gabriele, thanks again for the great library!

I apologize in advance if the answer to this can be found in the docs/paper/thesis, I couldn't find it there.

My question is rather soft: what is a "reasonable" SO(2)-equivariant architecture? E.g. what would be the "proper" way to generalize the Wide ResNets in examples/e2wrn.py to be (approximately) equivariant wrt SO(2) rather than CN?

My attempt so far is:

• set N=-1 with max_frequency=10 (leaving default value)
• change instances of e2cnn.nn.InnerBatchNorm with e2cnn.nn.NormBatchNorm
  • when making an invariant model and using gspace.trivial_repr for the output of the last layer, I have used InnerBatchNorm for the corresponding BatchNorm
• remove non-linearities (instances of e2cnn.nn.ReLU in this case)
• use field type of the form e2cnn.nn.FieldType(gspace, [gspace.irrep(1)] * a + [gspace.irrep(2)] * b + [gspace.irrep(3)] * c) for some positive integers a, b, and c.

This gist contains my attempt to make a SO(2)-invariant ResNet-18. An equivariance test, however, shows that I am rather off. In particular, the output of

model.cuda()
model.eval()
img_size = 221
repeats = 500
rot_diffs = []
rand_diffs = []

for _ in range(repeats):
    x = torch.randn([1, 3, img_size, img_size]).cuda()
    xrot = torch.rot90(x, k=1, dims=[2, 3])
    xrand = torch.randn([1, 3, img_size, img_size]).cuda()

    with torch.no_grad():
        z, lat = model(x, with_latent=True)
        zrot, latrot = model(xrot, with_latent=True)
        zrand, latrand = model(xrand, with_latent=True)

    rot_diffs.append(torch.norm(z - zrot))
    rand_diffs.append(torch.norm(z - zrand))

print(f'l2 diff between logits of image and its 90-degree rotation: {torch.mean(torch.stack(rot_diffs)).item():.3f}')
print(f'l2 diff between logits of two arbitrary images: {torch.mean(torch.stack(rand_diffs)).item():.3f}')

is

l2 diff between logits of image and its 90-degree rotation: 0.617
l2 diff between logits of two arbitrary images: 0.739

What is going wrong here?

In case this is helpful, this is the model "topology":

MODEL TOPOLOGY:
	0 - 
	1 - conv1
	2 - conv1._basisexpansion
	3 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_1')
	4 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_2')
	5 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_3')
	6 - relu1
	7 - bn1
	8 - maxpool
	9 - layer1
	10 - layer1.0
	11 - layer1.0.conv1
	12 - layer1.0.conv1._basisexpansion
	13 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
	14 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_2')
	15 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_3')
	16 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_1')
	17 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
	18 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_3')
	19 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_1')
	20 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_2')
	21 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
	22 - layer1.0.bn1
	23 - layer1.0.conv2
	24 - layer1.0.conv2._basisexpansion
	25 - layer1.0.bn2
	26 - layer1.1
	27 - layer1.1.conv1
	28 - layer1.1.conv1._basisexpansion
	29 - layer1.1.bn1
	30 - layer1.1.conv2
	31 - layer1.1.conv2._basisexpansion
	32 - layer1.1.bn2
	33 - layer2
	34 - layer2.0
	35 - layer2.0.conv1
	36 - layer2.0.conv1._basisexpansion
	37 - layer2.0.bn1
	38 - layer2.0.conv2
	39 - layer2.0.conv2._basisexpansion
	40 - layer2.0.bn2
	41 - layer2.0.shortcut
	42 - layer2.0.shortcut.0
	43 - layer2.0.shortcut.0._basisexpansion
	44 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
	45 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
	46 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
        ...
        (layers 2.1 to 4.0 only change multiplicities in field types, no 'block_expansion_'s here) 
        ...
	92 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_1', 'irrep_0')
	93 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_2', 'irrep_0')
	94 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_3', 'irrep_0')
	95 - layer4.1.relu2
	96 - layer4.1.bn2
	97 - layer4.1.bn2.batch_norm_[1]
	98 - linear
	99 - avgpool

Thanks in advance!

e2cnn library not working..please re

from e2cnn import gspaces

AttributeError: module 'e2cnn' has no attribute 'group'

this was working the day before but not now.

Equivariance and Image Resolution

Hi Gabriele, thank you very much for the great library!

Is equivariance guaranteed for images of arbitrary sizes/resolutions? It seems like equivariance is satisfied only for certain sizes. As an example, the C4-invariant 10-4 Wide ResNet defined and tested in examples/e2wrn.py seems to only be invariant to 90-degree rotations for images with widths of the form 4k+3.

To exemplify this, the output of this snippet

m = Wide_ResNet(10, 4, 0.3, initial_stride=1, N=4, f=False, r=0, num_classes=10)
m.eval()
for img_size in range(30, 50, 1):
    x = torch.randn([1, 3, 30, img_size])
    xrot = torch.rot90(x, k=1, dims=[2, 3])
    with torch.no_grad():
        z = m(x)
        zrot = m(xrot)
        print(f'Image size is 30x{img_size}. Equivariant:' +\
              ('YES' if torch.allclose(z, zrot, atol=1e-6) else 'NO'))

is

Image size is 30x30. Equivariant: NO
Image size is 30x31. Equivariant: NO
Image size is 30x32. Equivariant: NO
Image size is 30x33. Equivariant: YES
Image size is 30x34. Equivariant: NO
Image size is 30x35. Equivariant: NO
Image size is 30x36. Equivariant: NO
Image size is 30x37. Equivariant: YES
Image size is 30x38. Equivariant: NO
Image size is 30x39. Equivariant: NO
Image size is 30x40. Equivariant: NO
Image size is 30x41. Equivariant: YES
Image size is 30x42. Equivariant: NO
Image size is 30x43. Equivariant: NO
Image size is 30x44. Equivariant: NO
Image size is 30x45. Equivariant: YES
Image size is 30x46. Equivariant: NO
Image size is 30x47. Equivariant: NO
Image size is 30x48. Equivariant: NO
Image size is 30x49. Equivariant: YES.

This seems like an artifact caused by dilation/pooling but I am not sure whether there's any other factor causing this behaviour. Is this the case?

Lastly, would we observe a similar behaviour if we are working with field types composed of irreps of SO(2)? I don't have an intuition on how dilation is going to interact with guaranteeing equivariance there.

Thanks in advance!

Rotation equivariance on a specific angle

Hi, @Gabri95 , thank you for your nice work! And I have got inspiration from your work!
But a question (may be very simple) confused me:
How to generate rotation equivariant features on a specific angle (e.g., 45, 33.3), but not all the 360 degree features.
In other words, I just want the features on a particular orientation.
I guess the FieldType defined in SO(2) can resolve this problem, but I dont know how to use related functions.

Conversion to torchscript or ONNX

I'm working on optimizing my model inference, trying conversion to torchscript as a first step. When I call torch.jit.script() on my model, I hit:

name = '_weights_ranges', item = {('irrep_0,0', 'regular'): (0, 288)}

    def infer_type(name, item):
        # The forward function from Module is special; never use this annotations; we
        # need to infer type directly using JIT.  I originally wanted to write
        # this test as isinstance(class_annotations[name], Callable) but
        # isinstance on typing things doesn't seem to work: isinstance(list, Callable)
        # is also true!
        if name in class_annotations and class_annotations[name] != torch.nn.Module.__annotations__["forward"]:
            attr_type = torch.jit.annotations.ann_to_type(class_annotations[name], _jit_internal.fake_range())
        elif isinstance(item, torch.jit.Attribute):
            attr_type = torch.jit.annotations.ann_to_type(item.type, _jit_internal.fake_range())
        else:
>           attr_type = torch._C._jit_try_infer_type(item)
E           RuntimeError: Cannot create dict for key type '(str, str)', only int, float, Tensor and string keys are supported

This pytorch code resides here:
https://github.com/pytorch/pytorch/blob/22902b9242853a4ce319e7c5c4a1c94bc00ccb7a/torch/jit/_recursive.py#L126

Torch can't trace through a Dict[Tuple[str,str],_], which is used here:

e2cnn/e2cnn/nn/modules/r2_conv/basisexpansion_blocks.py

Line 295 in b2c26e3

coefficients = weights[self._weights_ranges[io_pair][0]:self._weights_ranges[io_pair][1]]

My goal is to get the model to run as fast as possible on NVIDIA hardware, probably using tensorrt. Is there another known-good conversion path?

The above error was with torch 1.6.0, e2cnn v0.1.

Need equivariant ResNet & VGG like architectural networks implemented in E2CNN library

Thank you for this great library for equivariant steerable CNNs. However, I am not able to build an equivariant model for my project through the examples given in this repository. Can you include examples for ResNet & VGG like architectures or can someone tell me how to induce equivariance in these steerable networks. For example, I tried building equivariant model based on WideResNet example given in the repository but I am struggling to induce equivariance and getting only invariance through that.

PointwiseMaxPoolAntialiased wrongly inherits .export() from parent class.

While auditing e2cnn too see if the functions I need actually support .export(), I noticed that:

PointwiseMaxPoolAntialiased:

e2cnn/e2cnn/nn/modules/pooling/pointwise_max.py

Line 143 in 9054d1e

class PointwiseMaxPoolAntialiased(PointwiseMaxPool):

inherits the .export() function from PointwiseMaxPool, whereas it should be exporting ops for the antialiased version.

RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

I believe that the package needs to be updated for new versions of PyTorch. The code works with PyTorch 1.1, but it does not work for PyTorch 1.8.

This issue can be reproduced via the use of another package:

1. git clone https://github.com/AllanYangZhou/metalearning-symmetries.git
2. python generate_synthetic_data.py --problem 2d_rot8

Another issue is the support of machine without CUDA. To reproduce this issue, execute the following two steps on a CPU-only machine:

1. git clone https://github.com/AllanYangZhou/metalearning-symmetries.git
2. python generate_synthetic_data.py --problem 2d_rot8_flip

Legacy 3.6 branch UnicodeDecodeError during installation

When running python setup.py install on the legacy_3.6 branch, I'm getting:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-pv1d_p_r/e2cnn/setup.py", line 19, in <module>
    long_description = f.read()
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 6561: ordinal not in range(128)

If I hotfix this by setting long_description = "", the installation works.

(Python 3.6.9)

About the equivalence of wide_resnet

Dear Author:
I change the wide_resnet by deleting the pooling and linear operation after out=out.tensor:

   def forward(self, x):

        # wrap the input tensor in a GeometricTensor
        x = enn.GeometricTensor(x, self.in_type)
        
        out = self.conv1(x)
        out = self.layer1(out)
        
        out = self.layer2(self.restrict1(out))
        
        out = self.layer3(self.restrict2(out))
        
        out = self.bn(out)
        out = self.relu(out)
        
        # extract the tensor from the GeometricTensor to use the common Pytorch operations
        out = out.tensor
        
        # b, c, w, h = out.shape
        # out = F.avg_pool2d(out, (w, h))
        
        # out = out.view(out.size(0), -1)
        # out = self.linear(out)
        return out

But find that the equivalence does not exist anymore:

    print()
    print('TESTING INVARIANCE:                    ')
    print('REFLECTIONS along the VERTICAL axis:   ' + (str(torch.norm(y-y_fv))))
    print('REFLECTIONS along the HORIZONTAL axis: ' + (str(torch.norm(y-y_fh))))
    print('90 degrees ROTATIONS:                  ' + (str(torch.norm(y-y90))))
    print('REFLECTIONS along the 45 degrees axis: ' + (str(torch.norm(y-y90_fh))))

The output is:

REFLECTIONS along the VERTICAL axis:   tensor(1.9236, grad_fn=<CopyBackwards>)
REFLECTIONS along the HORIZONTAL axis: tensor(2.0343, grad_fn=<CopyBackwards>)
90 degrees ROTATIONS:                  tensor(2.0621, grad_fn=<CopyBackwards>)
REFLECTIONS along the 45 degrees axis: tensor(2.0622, grad_fn=<CopyBackwards>)

May I have a ask about the reason?

Cannot pass weights of R2Conv as a positional argument

Hi there. Many thanks for this amazing library and the whole equivariant framework, it's a very inspiring work!

I think the title of the issue pretty much sums it all up. I am trying to pass 2 positional arguments as input to an R2Conv module, both the input data and also the weight parameters.

My issue is that I get an error saying that 3 positional arguments were given when 2 were expected.
The standard conv2D module of PyTorch allows this, so I was wondering if it something wrong in my implementation or in general this behavior cannot be used for an R2Conv module.
Many thanks in advance,

Should non-trivial irreps support bias or not?

Hi @Gabri95 ,

I noticed the current implementation only supports bias for trivial representation as commented:

e2cnn/e2cnn/nn/modules/r2_conv/r2convolution.py

Lines 184 to 189 in 2604f67

	if bias:
	# bias can be applied only to trivial irreps inside the representation
	# to apply bias to a field we learn a bias for each trivial irreps it contains
	# and, then, we transform it with the change of basis matrix to be able to apply it to the whole field
	# this is equivalent to transform the field to its irreps through the inverse change of basis,
	# sum the bias only to the trivial irrep and then map it back with the change of basis

From a naive intuition, I feel that bias acted on the norm of an irrep seems also equivariant. I didn't find relevant description in the paper. So just a short question for any relevant reference to deny this intuition. Thanks!