vfleaking/rnnprop

RNNprop

RNNprop

Compatible with TensorFlow 0.12

Training

You can use

python main.py --task rnnprop

to reproduce our RNNprop model, or use

python main.py --task deepmind-lstm-avg

to reproduce the DMoptimizer Andrychowicz et al., 2016 for comparison.

A random 6 digit letter string will be automatically generated as a unique id for each training process, and a folder named <task-name>-<id>_data will be created to place data and logs.

Evaluation

To evaluate the performance of a trained model, use

python main.py --train optimizer_train_optimizee

with other command-line flags:

• task: Must be specified, rnnprop or deepmind-lstm-avg.
• id: Must be specified, the unique 6 digit letter string that represents a trained model.
• eid: Must be specified, the epoch to restore the model.
• n_steps: Steps to train the optimizee. (Default is 100)
• n_epochs: How many times to train the optimizee, 0 means do not stop until keyboard interrupted. (Default is 0)

Optimizees

The optimizees used in all experiments are listed in test_list.py. You can train them with the best traditional optimization algorithm by using

python main.py --train optimizee

with other command-line flags:

• task: Must be specified, a name in test_list.py, e.g., mnist-nn-sigmoid-100.
• n_epochs: How many times to train the optimizee, 0 means do not stop until keyboard interrupted. (Default is 0)

License

MIT License.

rnnprop's Issues

Problem reproducing paper results

Hello! Thank you for the interesting paper!

I am trying to reproduce some results from the paper. As far as I understood all the parameters, which are set by default, are the same as in the paper. However, the number of training epochs is set to 0 i.e. train until keyboard interruption. I've tried to train for about 300 epochs on mnist-nn-sigmoid-100 (at this moment printed average loss of rnnprop became lower than the loss of GD). However, test performance on the same problem wasn't quite the same as in the paper. Sometimes rnnprop even diverged.

Could you please share your thoughts on how to reproduce results from the paper exactly?
Thanks.