orkis-research/quaternion-recurrent-neural-networks

This is the code for the paper 'Quaternion Recurrent Neural Networks'.

Quaternion Recurrent Neural Networks

This example trains both a Quaternion-valued RNN and a Real-valued RNN on a simple copy-task, and generates the corresponding plots. By default, the experiments follow the experimental protocol described in the paper : Quaternion Recurrent Neural Networks.

Requirements

A GPU with a running CUDA installation is required.

pip install torch torchvision matplotlib      

Usage

python copy_task.py        

quaternion-recurrent-neural-networks's Issues

Gradients not going backward

Hello,

I am trying a simple MNIST network:
model = nn.Sequential(QuaternionLinear(784, 128), nn.LeakyReLU(), QuaternionLinear(128, 10), nn.LogSoftmax(dim=1))

However, the loss is never decreasing, I could not find any equations for back propagation of the gradients, but assuming the gradients are not going backward, is this expected? if not how to debug? Would be really helpful, if could provide an example on a simple task (MNIST). Thanks!

Using optim.SGD with lr=0.003, momentum=0.9, ran 10 epochs