zerocurve/lstm-tensorspark

Implementation of a LSTM with TensorFlow and distributed on Apache Spark

LSTM-TensorSpark

Implementation of a LSTM with TensorFlow and distributed on Apache Spark

There are provided three different implementations:

• Distributed on Spark;
• Standalone;
• Standalone with speed improvements;

Detailed explanation here: Distributed implementation of a LSTM on Spark and Tensorflow

Important

Developed for academic purpose

Dependencies

Distributed model needs:

• Python 2.6+
• Pyspark
• TensorFlow
• Numpy
• Argparse

Standalone models need:

• Python 2.6+
• TensorFlow
• Numpy
• Argparse

Usage

Example using Spark:

From src directory

pyspark rnn.py -e 10 -hl 5 -i ../dataset/iris.data -t ../dataset/data.config -p 5 

• -e: number of epochs;
• -hl: number of hidden layer;
• -i: input data;
• -t: class configuration file;
• -p: number of partitions;
• -mb: number of mini batch;

Example without Spark:

From src directory

python rnn-no-spark.py -e 10 -hl 5 -i ../dataset/iris.data -t dataset/data.config

python rnn-speed.py -e 10 -hl 5 -i ../dataset/iris.data -t dataset/data.config

Using minibatch:

data.config

python rnn-no-spark.py -e 10 -hl 5 -i ../dataset/iris.data -t ../dataset/data.config -mb 10

For full documentation use -h option