jbdatascience/ntm-lasagne

Neural Turing Machines library in Theano with Lasagne

NTM-Lasagne

NTM-Lasagne is a library to create Neural Turing Machines (NTMs) in Theano using the Lasagne library. If you want to learn more about NTMs, check out our blog post.

This library features:

• A Neural Turing Machine layer NTMLayer, where all its components (controller, heads, memory) are fully customizable.
• Two types of controllers: a feed-forward DenseController and a "vanilla" recurrent RecurrentController.
• A dashboard to visualize the inner mechanism of the NTM.
• Generators to sample examples from algorithmic tasks.

Getting started

To avoid any conflict with your existing Python setup, and to keep this project self-contained, it is suggested to work in a virtual environment with virtualenv. To install virtualenv:

sudo pip install --upgrade virtualenv

Create a virtual environment called venv, activate it and install the requirements given by requirements.txt. NTM-Lasagne requires the bleeding-edge version, check the Lasagne installation instructions for details. The latest version of Lasagne is included in the requirements.txt.

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pip install .

Example

Here is minimal example to define a NTMLayer

# Neural Turing Machine Layer
memory = Memory((128, 20), memory_init=lasagne.init.Constant(1e-6),
    learn_init=False, name='memory')
controller = DenseController(l_input, memory_shape=(128, 20),
    num_units=100, num_reads=1,
    nonlinearity=lasagne.nonlinearities.rectify,
    name='controller')
heads = [
    WriteHead(controller, num_shifts=3, memory_shape=(128, 20),
        nonlinearity_key=lasagne.nonlinearities.rectify,
        nonlinearity_add=lasagne.nonlinearities.rectify,
        learn_init=False, name='write'),
    ReadHead(controller, num_shifts=3, memory_shape=(128, 20),
        nonlinearity_key=lasagne.nonlinearities.rectify,
        learn_init=False, name='read')
]
l_ntm = NTMLayer(l_input, memory=memory, controller=controller, heads=heads)

For more detailed examples, check the examples folder. If you would like to train a Neural Turing Machine on one of these examples, simply run the corresponding script, like

PYTHONPATH=. python examples/copy-task.py

Tests

This projects has a few basic tests. To run these tests, you can run the py.test on the project folder

venv/bin/py.test ntm -vv

Known issues

Graph optimization is computationally intensive. If you are encountering suspiciously long compilation times (more than a few minutes), you may need to increase the amount of memory allocated (if you run it on a Virtual Machine). Alternatively, turning off the swap may help for debugging (with swapoff/swapon).

Note: Unlucky initialisation of the parameters might lead to a diverging solution (NaN scores).

Paper

Alex Graves, Greg Wayne, Ivo Danihelka, Neural Turing Machines, [arXiv]

Contributing

Please see the Contribution Guidelines.

Copyright

This library is provided by Snips as Open Source software. See LICENSE for more information.