gram-ai / capsule-networks Goto Github PK

View Code? Open in Web Editor NEW

1.7K 1.7K 319.0 797 KB

A PyTorch implementation of the NIPS 2017 paper "Dynamic Routing Between Capsules".

Home Page: https://arxiv.org/abs/1710.09829

Python 100.00%

capsule-networks machine-learning mnist pytorch

capsule-networks's People

Contributors

Stargazers

Watchers

Forkers

ryfan-rs zhilangtaosha stevenlol locosoft1986 dendisuhubdy praveenmunagapati laughing-boy ml-lab jfsantos amoliu kjeanclaude cedrickchee o7s8r6 hylihitic cfeitong jingweiz dcherneski mmderakhshani tarrysingh waleedgondal gwding jaiabhayk ivjia indrango xiaomi2008 ykwon0407 gumpfly fehiepsi marclachapelle frosttempest vishbin imnueducn ldkhanh linpingchuan yuxianmeng abi-aryan mutual-ai jacobswan1 alexanderpu dathpham grseb9s atifs rdangovs kingofspace0wzz autohe yechengxi arunrajeie eracah deep-learning-fun rnnccl bnkang guanlongtianzi lsheiba rremani mieitza seemirra zhangxlaurence deep0learning rbunn80110 jianning-li wheatdog yangshiyu89 abhijpes anirudhkashi thelaughingboy eletter-greece tony32769 shaoli-huang romxz birajaghoshal h7474 abduallahmohamed christy-yuan-li caolusg icewwn redhat12345 zhhezhhe gavins13 weeang763162 abitalo emigmo gaoy926 devenlu zywpku shubhampachori12110095 ii0 boazrciasn rwang92 hot-cheeto shyamalschandra raeidsaqur harunpehlivan liuyuuan liyuanyaun unsuthee danilopetrocelli ngatilio bopang1996 mansimane jindongwang

capsule-networks's Issues

Is there any error about the softmax operation?

"The coupling coefficients between capsule i and all the capsules in the layer above sum to 1", softmax should be computed along the channel of capsules, and you computed along the channel of route nodes.

Is squashing used correctly on this line?

capsule-networks/capsule_network.py

Line 78 in bb65115

outputs = self.squash(outputs)

If I understand correctly, squashing should be applied to the feature dimension, whereas here it seems it is applied to the capsule dimension.

How to replace dataset and change dimensions

Hi,
This implementation was one of the easiest to implement as well as understand. I am curious and would want to implement COCO dataset using the same code.

How do I change the dimensions to 3?
How to import COCO dataset into the code? I tried importing from torchvision.datasets but it is throwing an error constantly.
An overview as to how to make it work on COCO dataset.

Could u please help me with this?

OSError: [Errno 16] Device or resource busy: '.nfs000000009886bd5100000235'

This error message keeps appearing during the training process, but the training itself seems to be fine, i.e. the loss is decreasing. I wonder how can I get rid of this..... (btw, how can insert newline in the error message below?)

OSError: [Errno 16] Device or resource busy: '.nfs000000009886bd5100000235' File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/multiprocessing/util.py", line 186, in __call__ res = self._callback(*self._args, **self._kwargs) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers finalizer() Traceback (most recent call last): File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/multiprocessing/util.py", line 186, in __call__ res = self._callback(*self._args, **self._kwargs) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd onerror(os.unlink, fullname, sys.exc_info()) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) Process Process-280: File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd os.unlink(name, dir_fd=topfd) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd onerror(os.unlink, fullname, sys.exc_info()) File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers finalizer() File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd os.unlink(name, dir_fd=topfd) OSError: [Errno 16] Device or resource busy: '.nfs000000009886bd5000000236' File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/multiprocessing/util.py", line 186, in __call__ res = self._callback(*self._args, **self._kwargs) OSError: [Errno 16] Device or resource busy: '.nfs000000009886bd4e00000236' File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) Process Process-278: File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd onerror(os.unlink, fullname, sys.exc_info()) Traceback (most recent call last): File "/home/lijun/anaconda2/envs/pytorch3/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd os.unlink(name, dir_fd=topfd)

Can the 8 32-channel conv be replaced with a single 256-channel conv in PrimaryCapsule?

Thanks for sharing the nicely written code.

I wonder that is there any difference between using 8 32-channel convlayer and a single 256-channel convlayer? It seems like the "In total PrimaryCapsules has [32*6*6] capsule outputs (each output is an 8D vector)" mentioned in Dynamic Routing Between Capsules can be simply implemented as a 256-channel convlayer followed by a reshaping operation.

But I haven't seen anyone using 256-channel conv so far. Am I thinking wrong?

Code for SVHN dataset?

Can you please make the code for SVHN datasets?

Extra softmax layer

Why is there an extra softmax layer https://github.com/gram-ai/capsule-networks/blob/master/capsule_network.py#L106?
Each capsule's norm is already modelling a probability.

Try my implementation

Similar to others, I found many issues with this implementation. So I decided to create my own one. It is bug-free and works very good. You can find it here:

https://github.com/hula-ai/capsule_network_dynamic_routing

Squash function

It's not quite clear how vectors from primary capsules are squashed. As far as I can see from the code, each primary capsule outputs a vector of size 32 * 6 * 6. Then these vectors are stacked and, considering the batch dimension, we get a tensor of the shape

(batch_size, num_nodes_in_capsule = 32 * 6 * 6, num_capsules = 8)

Finally, these vectors are normalized, i.e. their magnitudes are squashed to be in the range from 0 to 1. If I understand correctly, the paper is referring to the magnitude of the (32 * 6 * 6)-dimensional vectors. So if we want to ensure that the length of these vectors is in range [0; 1], we would have to divide each of the (36 * 6 * 6) coordinates by the square root of the sum of squares of these coordinates. Right? In fact, the implementation divides each coordinate by the magnitude of a vector comprised of the coordinates in the same positions of all capsule vectors (dim is set to -1 when calculating squared_norm, i.e. it sums up same coordinates/features, but from different capsules).

Please, consider the following example:

import torch
import numpy as np

def squash(input_tensor):
    '''Squashes an input Tensor so it has a magnitude between 0-1.
        param input_tensor: a stack of capsule inputs, s_j
        return: a stack of normalized, capsule output vectors, v_j
        '''
    squared_norm = (input_tensor ** 2).sum(dim=-1, keepdim=True)    
    scale = squared_norm / (1 + squared_norm) # normalization coeff
    output_tensor = scale * input_tensor / torch.sqrt(squared_norm)    
    return output_tensor

np.random.seed(1)
torch.manual_seed(1)
batch_size = 15
dim=13
n_caps = 7
u = [torch.tensor(np.random.rand(batch_size, dim, 1)) for i in range(n_caps) ] 
#print(u)
u = torch.cat(u, dim=-1)
print("u:", u)

u_squash = squash(u)
print("u_squash:", u_squash)

mag = torch.sqrt( (u_squash **2).sum(dim=-2) )
print("mag: ", mag)

Here I create a randomly filled tensor of shape (batch_size, dim, n_caps), i.e. similar to those produced by the primary capsules, just smaller for illustration. The tensor is squashed by the same squash function. It can be seen from the output that the magnitudes of the vectors exceeds the range [0; 1]:

mag:  tensor([[0.6629, 1.0954, 0.9715, 0.7817, 1.0211, 0.7117, 0.8847],
        [1.0202, 0.9313, 0.8816, 0.8383, 1.0355, 0.9926, 1.0803],
        [0.8864, 1.0694, 0.7617, 0.9194, 0.8355, 0.9432, 1.0051],
        [0.9630, 0.9198, 0.9078, 1.0516, 0.8845, 0.7888, 0.9238],
        [0.6996, 1.0998, 1.1319, 0.6556, 0.8243, 0.9571, 0.9614],
        [0.9705, 0.9879, 0.8915, 0.8308, 1.0063, 1.0607, 0.9306],
        [1.0569, 1.0294, 0.9268, 1.0508, 0.9768, 0.9505, 0.8103],
        [0.9545, 0.9655, 0.9052, 1.0720, 0.7246, 0.9666, 0.9669],
        [1.1237, 0.9768, 0.9749, 0.8128, 0.8935, 0.9216, 0.7607],
        [0.8785, 0.7155, 0.8306, 0.8913, 0.9764, 0.9692, 1.0892],
        [0.9691, 0.8658, 1.0399, 0.9774, 0.9309, 0.8950, 0.8872],
        [0.7124, 1.1386, 0.8535, 1.0913, 0.8478, 0.8779, 0.9850],
        [0.8909, 0.9851, 0.9247, 1.0239, 0.7927, 0.9618, 0.7925],
        [0.8764, 0.9524, 0.9294, 0.8517, 0.8385, 0.9380, 1.0824],
        [1.0076, 0.8668, 1.0051, 0.9030, 1.0067, 0.8850, 0.9519]],
       dtype=torch.float64)

It actually enforces the magnitudes of vectors comprised of particular coordinates from different capsule outputs to be in that range. But is it what was intended?

What are the core code of capsule network?

Base on my understanding, capsule network use vector to replace scalar. What are the core code to implement it?
Thank you and thank you!

Why dim=2 here? I guess the softmax is taken over all capsules in the layer above, so shouldn't that be dim=0?

capsule-networks/capsule_network.py

Line 69 in 1a4edd2

probs = softmax(logits, dim=2)

How should I set num_route_nodes?

According to my understanding, num_route_nodes should be the length of the vote vectors that are used as input to dynamic routing.

In the example code, this value is set according to:

        self.primary_capsules = CapsuleLayer(num_capsules=8, num_route_nodes=-1, in_channels=256, out_channels=32,
                                             kernel_size=9, stride=2)
        self.digit_capsules = CapsuleLayer(num_capsules=NUM_CLASSES, num_route_nodes=32 * 6 * 6, in_channels=8,
                                           out_channels=16)

32 appears to come from the previous out_channels, but I can't tell what 6 * 6 are doing. Is this arbitrary, or is it preventing some kind of dimension mismatch? If I am implementing my own capsule network with new dimensions, do I need to be careful about how I pick this value?

Thanks,
Will

cuda runtime error

Hello,
first of all, thx for ur code~
When i tried to run capsule_network.py, i got a runtime error, do u know how to deal with this?
Here is the error:

parameters: 8215568

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Processing...
Done!
0%| | 1/600 [00:04<42:26, 4.25s/it]THCudaCheck FAIL file=/pytorch/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
File "capsule_network.py", line 259, in
engine.train(processor, get_iterator(True), maxepoch=NUM_EPOCHS, optimizer=optimizer)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torchnet/engine/engine.py", line 39, in train
state['optimizer'].step(closure)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/optim/adam.py", line 39, in step
loss = closure()
File "/home/yellowtown/.local/lib/python3.5/site-packages/torchnet/engine/engine.py", line 31, in closure
loss.backward()
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 156, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/init.py", line 98, in backward
variables, grad_variables, retain_graph)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/function.py", line 91, in apply
return self._forward_cls.backward(self, args)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/_functions/blas.py", line 127, in backward
grad_batch2 = torch.bmm(batch1.transpose(1, 2), grad_output)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 584, in bmm
return self._static_blas(Baddbmm, (output, 0, 1, self, batch), False)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 572, in _static_blas
return cls.apply((args[:1] + args[-2:] + (alpha, beta, inplace)))
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/_functions/blas.py", line 107, in forward
output = _get_output(ctx, add_batch, inplace=inplace)
File "/home/yellowtown/.local/lib/python3.5/site-packages/torch/autograd/_functions/blas.py", line 13, in get_output
return arg.new().resize_as(arg)
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:66

@ operator

hello,how does the operator @ work?

MLP Capsules ?

Great repo. I've been trying to implement an MLP based caps net, where the primary caps analyzes subsets of features in groups, feeding them into MLPs per capsule then pushed to downstream capsules.

For some reason, I am getting terrible results. Do you know what I might be doing wrong?

What are areas that make MLP capsules different from CNN capsules, what should I be aware of? How do I optimize to get good results. I think dynamic routing is essential for establishment of hierarchy, but in the non-image context, I'm unsure of how the affine transformation is helpful.

By any chance could you post an example using MLP capsules?

Just a quick question on the dynamic routing's similarity measurment

capsule-networks/capsule_network.py

Line 73 in 1a4edd2

delta_logits = (priors * outputs).sum(dim=-1, keepdim=True)

I could miss something here, but am just wondering given the original paper, for this line of code on dynamic routing, shouldn't matrix multiplication (e.g. torch.matmul) be used to conduct the similarity measurement of the two (outputs & priors) matrices? instead of the element-wise product as in the current codes above?

For example, I know someone else used torch.matmul for this similarity measurement purpose of dynamic routing in this repo.

I did not spend lots of time reading everything, but just not sure on this... Thanks for the great implementation!

RuntimeError: The size of tensor a (16) must match the size of tensor b (10) at non-singleton dimension 1

I am facing a Runtime error while running the code.

RuntimeError: The size of tensor a (16) must match the size of tensor b (10) at non-singleton dimension 1

No such file or directory

When I run the code on the Google Colab, the error shows in picture, how to solve??

Error during forward pass

Hi! I get the following error stack trace when running the model. I have not made any changes to the code.

I would really appreciate any feedback! Thank you!

Traceback (most recent call last):
File "capsule_network.py", line 264, in
engine.train(processor, get_iterator(True), maxepoch=NUM_EPOCHS, optimizer=optimizer)
File "/home/ubuntu/anaconda3/envs/CapsuleNets/lib/python3.6/site-packages/torchnet/engine/engine.py", line 43, in train
self.hook('on_end_epoch', state)
File "/home/ubuntu/anaconda3/envs/CapsuleNets/lib/python3.6/site-packages/torchnet/engine/engine.py", line 7, in hook
self.hooksname
File "capsule_network.py", line 232, in on_end_epoch
engine.test(processor, get_iterator(False))
File "/home/ubuntu/anaconda3/envs/CapsuleNets/lib/python3.6/site-packages/torchnet/engine/engine.py", line 69, in test
closure()
File "/home/ubuntu/anaconda3/envs/CapsuleNets/lib/python3.6/site-packages/torchnet/engine/engine.py", line 61, in closure
loss, output = state'network'
File "capsule_network.py", line 195, in processor
classes, reconstructions = model(data)
File "/home/ubuntu/anaconda3/envs/CapsuleNets/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "capsule_network.py", line 112, in forward
y = Variable(torch.sparse.torch.eye(NUM_CLASSES)).cuda().index_select(dim=0, index=max_length_indices.data)
RuntimeError: index_select(): argument 'index' must be Variable, not torch.cuda.LongTensor

How to prove it is a real capsule network?

Thank you and thank you.

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

How to cite this repo ?

Hi,

Your implementation is great =)
How to cite this repo ?

Thanks

the visdom window has nothing to show

Hi,thanks for the code and sorry to bother you.
I ran 'sudo python3 -m visdom.server' before 'python3 capsule_network.py' , and the visdom ran correctly , but it showed nothing even i refreshed the page.
Just like this:

And here are the outputs:

Thank you again,please help me.

Technical doubts regarding implementation

Thanks for making this repository, really useful :)

Let me start with the questions:

Is this network specifically done for 28x28 images or would it work for other sizes? The same goes for the number of channels, if self.conv1 = nn.Conv2d(in_channels=3, out_channels=256, kernel_size=9, stride=1) is used instead, will it work for images with 3 channels or is it necessary to make other changes?
In self.digit_capsules = CapsuleLayer(num_capsules=NUM_CLASSES, num_route_nodes=32 * 6 * 6, in_channels=8, out_channels=16), why did you choose 32x6x6 num_route_nodes? I am trying this code with different data and stuck in this step as the image dimensions are different and idk how they relate with this hyperparameter.
By increasing num_iterations will the model's performance increase as well?

visdom exception

Hi,
Sorry for bothering u again :)
When i ran capsule-network.py, the terminal output a list of exception about visdom during every epoch.
Do u have any idea wt happened and how to deal with this problem?
Thx ~

Here are the outputs of the first epoch:

yellowtown@Yellowtown:/media/yellowtown/工作/Github/capsule-networks$ python3.5 capsule_network.py

parameters: 8215568

100%|███████████████████████████████████████| 6000/6000 [05:42<00:00, 17.50it/s]
[Epoch 1] Training Loss: 0.4899 (Accuracy: 95.31%)
Exception in user code:

Traceback (most recent call last):
File "/home/yellowtown/.local/lib/python3.5/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/home/yellowtown/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/home/yellowtown/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused