fferroni/phasedlstm-keras

Keras implementation of Phased LSTM [https://arxiv.org/abs/1610.09513]

PhasedLSTM-Keras

Keras implementation of Phased LSTM [https://arxiv.org/abs/1610.09513], from NIPS 2016.

To install run: `pip install phased_lstm_keras`

Works both with Theano and Tensorflow backend (tested on Theano 0.9 and Tensorflow 1.0).

Classification performance compared to standard Keras LSTM for MNIST dataset:

phasedlstm-keras's Issues

Information: Keras 2.2.0 doesn't work with examples due to a known issue

I rand the random_example.py with the latest Keras 2.2.0 and hit the following error while_loop() got an unexpected keyword argument maximum_iterations. Someone from the tensorflow repository suggested using Keras 2.1.5, so I downgraded my Keras to 2.1.5 and it bypassed this error.

Reference
tensorflow/tensorflow#20075

Error

(plstm_keras) nicole@polarsnow:~/git/PhasedLSTM-Keras/examples(master)$ python random_example.py 
/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Traceback (most recent call last):
  File "random_example.py", line 31, in <module>
    main()
  File "random_example.py", line 13, in main
    model_lstm.add(LSTM(10, input_shape=(100, 2)))
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/engine/sequential.py", line 166, in add
    layer(x)
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 500, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/engine/base_layer.py", line 460, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 2112, in call
    initial_state=initial_state)
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 609, in call
    input_length=timesteps)
  File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2957, in rnn
    maximum_iterations=input_length)
TypeError: while_loop() got an unexpected keyword argument 'maximum_iterations'
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.ops.tensor_array_ops.TensorArray'>):
<tensorflow.python.ops.tensor_array_ops.TensorArray object at 0x7fcaa7c49ba8>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
['File "random_example.py", line 31, in <module>\n    main()', 'File "random_example.py", line 13, in main\n    model_lstm.add(LSTM(10, input_shape=(100, 2)))', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/engine/sequential.py", line 166, in add\n    layer(x)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 500, in __call__\n    return super(RNN, self).__call__(inputs, **kwargs)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/engine/base_layer.py", line 460, in __call__\n    output = self.call(inputs, **kwargs)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 2112, in call\n    initial_state=initial_state)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/layers/recurrent.py", line 609, in call\n    input_length=timesteps)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2877, in rnn\n    input_ta = input_ta.unstack(inputs)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 175, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 413, in unstack\n    indices=math_ops.range(0, num_elements), value=value, name=name)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 175, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 144, in _add_should_use_warning\n    wrapped = TFShouldUseWarningWrapper(x)', 'File "/home/nicole/anaconda3/envs/plstm_keras/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 101, in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']
==================================

License

Dear Francesco,
I apologize for writing to you here, as I could not find any other contact method.

We would like to use your implementation of PhasedLSTM for Keras in our projects. However, we have seen that your license for using it is GNU General Public License v3.0, which obligates that our project also is released under it. Despite planning to release it as open source software, we are still debating on which license is going to be used but we do not want it to be GPL, and yours limits us to just GPL3.

Has it happened this before to you? Is it possible for you to modify the license of your library?

Sorry for any inconvenience and thank you

Which version of Keras and tensorflow are you using to run the code?

I try the combination of different versions, there are always some errors when running the examples. Could you please tell me your version information? Thank you.

FailedPreconditionError: Attempting to use uninitialized value phased_lstm_2/Variable

Trying to implement my first pLSTM powered RNN and ran in to this problem. Can't figure it out.

Code
#create pLSTM network
pLSTMmodel = Sequential()
pLSTMmodel.add(PLSTM(10, input_shape=(1, 10)))
#model.add(Dropout(.5, input_shape=(60,)))
pLSTMmodel.add(Dense(1))
pLSTMmodel.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae', 'acc'])
pLSTMmodel.fit(trainCurrent, trainFutureAQI, epochs=5, batch_size=10, verbose=2)

Full trace back
FailedPreconditionError Traceback (most recent call last)
C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1021 try:
-> 1022 return fn(*args)
1023 except errors.OpError as e:

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1003 feed_dict, fetch_list, target_list,
-> 1004 status, run_metadata)
1005

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\contextlib.py in exit(self, type, value, traceback)
65 try:
---> 66 next(self.gen)
67 except StopIteration:

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:

FailedPreconditionError: Attempting to use uninitialized value phased_lstm_2/Variable
[[Node: phased_lstm_2/Variable/read = IdentityT=DT_FLOAT, _class=["loc:@phased_lstm_2/Variable"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

During handling of the above exception, another exception occurred:

FailedPreconditionError Traceback (most recent call last)
in ()
5 pLSTMmodel.add(Dense(1))
6 pLSTMmodel.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae', 'acc'])
----> 7 pLSTMmodel.fit(trainCurrent, trainFutureAQI, epochs=5, batch_size=10, verbose=2)

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\models.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)
843 class_weight=class_weight,
844 sample_weight=sample_weight,
--> 845 initial_epoch=initial_epoch)
846
847 def evaluate(self, x, y, batch_size=32, verbose=1,

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)
1483 val_f=val_f, val_ins=val_ins, shuffle=shuffle,
1484 callback_metrics=callback_metrics,
-> 1485 initial_epoch=initial_epoch)
1486
1487 def evaluate(self, x, y, batch_size=32, verbose=1, sample_weight=None):

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\engine\training.py in _fit_loop(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch)
1138 batch_logs['size'] = len(batch_ids)
1139 callbacks.on_batch_begin(batch_index, batch_logs)
-> 1140 outs = f(ins_batch)
1141 if not isinstance(outs, list):
1142 outs = [outs]

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2071 value = (indices, sparse_coo.data, sparse_coo.shape)
2072 feed_dict[tensor] = value
-> 2073 session = get_session()
2074 updated = session.run(self.outputs + [self.updates_op],
2075 feed_dict=feed_dict)

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in get_session()
151 session = _SESSION
152 if not _MANUAL_VAR_INIT:
--> 153 _initialize_variables()
154 return session
155

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in _initialize_variables()
304 if uninitialized_variables:
305 sess = get_session()
--> 306 sess.run(tf.variables_initializer(uninitialized_variables))
307
308

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
765 try:
766 result = self._run(None, fetches, feed_dict, options_ptr,
--> 767 run_metadata_ptr)
768 if run_metadata:
769 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
963 if final_fetches or final_targets:
964 results = self._do_run(handle, final_targets, final_fetches,
--> 965 feed_dict_string, options, run_metadata)
966 else:
967 results = []

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1013 if handle is None:
1014 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1015 target_list, options, run_metadata)
1016 else:
1017 return self._do_call(_prun_fn, self._session, handle, feed_dict,

C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1033 except KeyError:
1034 pass
-> 1035 raise type(e)(node_def, op, message)
1036
1037 def _extend_graph(self):

FailedPreconditionError: Attempting to use uninitialized value phased_lstm_2/Variable
[[Node: phased_lstm_2/Variable/read = IdentityT=DT_FLOAT, _class=["loc:@phased_lstm_2/Variable"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op 'phased_lstm_2/Variable/read', defined at:
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel_main.py", line 3, in
app.launch_new_instance()
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 474, in start
ioloop.IOLoop.instance().start()
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 887, in start
handler_func(fd_obj, events)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 390, in execute_request
user_expressions, allow_stdin)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2821, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 3, in
pLSTMmodel.add(PLSTM(10, input_shape=(1, 10)))
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\models.py", line 422, in add
layer(x)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\layers\recurrent.py", line 252, in call
return super(Recurrent, self).call(inputs, **kwargs)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 519, in call
self.build(input_shapes[0])
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\phased_lstm_keras-1.0.1-py3.5.egg\phased_lstm_keras\PhasedLSTM.py", line 245, in build
constraint=self.timegate_constraint)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 355, in add_weight
weight = K.variable(initializer(shape), dtype=K.floatx(), name=name)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\phased_lstm_keras-1.0.1-py3.5.egg\phased_lstm_keras\PhasedLSTM.py", line 66, in _timegate_initializer
dtype=dtype)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 286, in variable
v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variables.py", line 197, in init
expected_shape=expected_shape)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variables.py", line 315, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1490, in identity
result = _op_def_lib.apply_op("Identity", input=input, name=name)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 763, in apply_op
op_def=op_def)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\Christopher\Anaconda2\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1226, in init
self._traceback = _extract_stack()

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value phased_lstm_2/Variable
[[Node: phased_lstm_2/Variable/read = IdentityT=DT_FLOAT, _class=["loc:@phased_lstm_2/Variable"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

Is this compatible with tensorflow 2 ?

Hi, has someone made an update to that implementation to make it compatible with tensorflow 2 ?
I cannot really afford to downgrade my version for other parts of my project

Dropout during inference

Given that A Theoretically Grounded Application of Dropout in Recurrent Neural Networks was metioned in the code I expected that this implementation would allow the use of dropout during inference. But using the same keras, tensorflow and python versions and the same architecture with the default LSTM layer from keras and training=True it works, and with the PhasedLSTM it doesn't work.

I am using:
Keras==2.2.5
Tensorflow==1.15
Python==2.7
On Google Colab

Passing Event Camera Event-Train

First - the implementation works good, but I have a lack of understanding concerning the asynchronous input.

If I have an event train of position X,Y at ts (I ignore the Polarity for now),
how would I pass the stream into the PhasedLSTM?

In the paper : https://arxiv.org/abs/1610.09513 they used kind of an embedding with 40 dimensions (from which the source is not really helpful) and pass the EventMnist into the PLSTM.

In your example you pass the MNIST image as one dimensional input PLSTM(32, input_shape=(28 * 28, 1), implementation=2)
which is ok but I think its not the way the PLSTM should be used. Because what frequency will be learned ?

Can anybody help me?

Add package to pypi

I was unable to find the package on pypi, which would allow a user to install it with pip.
The twine package is a good helper to allow this to go smoothly.

Phased LSTM with attention?

Has anyone seen this done?

with tensorflow backend, why is GPU not used

?????

Regression by PhasedLSTM with a gradient explosion

Hello,

When I used PhasedLSTM (PLSTM) to perform the regression (to find the correlation between an input sequence and an output sequence), I got "nan" in the weight , also the loss in the beginning of the first epoch, even I used gradient clipping.

The generated data for training: (little modified from https://fairyonice.github.io/Extract-weights-from-Keras's-LSTM-and-calcualte-hidden-and-cell-states.html)

The optimizer is as follows:
model.compile(loss="mean_squared_error", sample_weight_mode="temporal", optimizer = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0))

After checked the weights in PLSTM layer, I found the values of timegate-kernel getting larger and larger, then the weights get to "nan". (The first two rows)

I changed to standard LSTM (other settings and learning rate [still 0.01] the same), the loss converges. Therefore, I traced the source code of PLSTM, considering the initialization of timegate_kernel matters, but stuck for a long time, having little progress.

I am wondering if anyone has the similar issue? Any suggestions to find the reason why the gradient get exploded is appreciated. The relevant code is at the link:

https://github.com/hnchang/Regression-with-PhasedLSTM/blob/master/reg_plstm.py

Much thanks,
James

random_example.py: Despite installing, "Unknown layer: PhasedLSTM"

Hi! I'm really interested in trying this out, however, following the instructions to install and then running one of the examples didn't work for me. I got the same error on both Ubuntu 16.04 running Python 3.6, and with MacOS X High Sierra running Python 3.5: ValueError: Unknown layer: PhasedLSTM

Any suggestions for getting this to work?

Below is a log of everything I did, this one for the Mac...

$ pip install phased_lstm_keras
Collecting phased_lstm_keras
Downloading https://files.pythonhosted.org/packages/73/07/20fe5a59c4e11a42c17763219eb5a3de7c827be7b7d9c1afa5834a4e19b1/phased_lstm_keras-1.0.2-py2.py3-none-any.whl
Requirement already satisfied: matplotlib>=2.0.0 in ./anaconda/envs/py35/lib/python3.5/site-packages (from phased_lstm_keras) (2.2.2)
Requirement already satisfied: numpy>=1.12.1 in ./anaconda/envs/py35/lib/python3.5/site-packages (from phased_lstm_keras) (1.14.0)
Requirement already satisfied: keras>=2.0.2 in ./anaconda/envs/py35/lib/python3.5/site-packages (from phased_lstm_keras) (2.1.3)
Requirement already satisfied: cycler>=0.10 in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (2.1.10)
Requirement already satisfied: python-dateutil>=2.1 in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (2.6.1)
Requirement already satisfied: pytz in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (2017.3)
Requirement already satisfied: six>=1.10 in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (1.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./anaconda/envs/py35/lib/python3.5/site-packages (from matplotlib>=2.0.0->phased_lstm_keras) (1.0.1)
Requirement already satisfied: scipy>=0.14 in ./anaconda/envs/py35/lib/python3.5/site-packages (from keras>=2.0.2->phased_lstm_keras) (1.0.0)
Requirement already satisfied: pyyaml in ./anaconda/envs/py35/lib/python3.5/site-packages (from keras>=2.0.2->phased_lstm_keras) (3.12)
Requirement already satisfied: setuptools in ./anaconda/envs/py35/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.0.0->phased_lstm_keras) (39.0.1)
Installing collected packages: phased-lstm-keras
Successfully installed phased-lstm-keras-1.0.2

$ git clone https://github.com/fferroni/PhasedLSTM-Keras.git
Cloning into 'PhasedLSTM-Keras'...
remote: Counting objects: 101, done.
remote: Total 101 (delta 0), reused 0 (delta 0), pack-reused 101
Receiving objects: 100% (101/101), 363.37 KiB | 2.04 MiB/s, done.
Resolving deltas: 100% (40/40), done.

$ cd PhasedLSTM-Keras/examples/
$ python random_example.py
/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
/Users/shawley/anaconda/envs/py35/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
return f(*args, **kwds)
2018-04-22 12:51:22.827292: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA

---

Layer (type) Output Shape Param #

lstm_1 (LSTM) (None, 10) 520

Total params: 520
Trainable params: 520
Non-trainable params: 0

---

Traceback (most recent call last):
File "random_example.py", line 31, in
main()
File "random_example.py", line 23, in main
model_plstm = load_model('model_plstm.h5')
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/models.py", line 243, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/models.py", line 317, in model_from_config
return layer_module.deserialize(config, custom_objects=custom_objects)
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 143, in deserialize_keras_object
list(custom_objects.items())))
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/models.py", line 1352, in from_config
layer = layer_module.deserialize(conf, custom_objects=custom_objects)
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/Users/shawley/anaconda/envs/py35/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 137, in deserialize_keras_object
': ' + class_name)
ValueError: Unknown layer: PhasedLSTM

$ which python
/Users/shawley/anaconda/envs/py35/bin/python
(py35) Scotts-MacBook-Pro-3:examples shawley$ python --version
Python 3.5.3

It seems something wrong in the time gate `k` implementation

Hi fferroni,

Thanks for your great implementation. I found there seems some little thing wrong in this block

        # K.switch not consistent between Theano and Tensorflow backend, so write explicitly.
        up = K.cast(K.lesser(phi, r_on * 0.5), K.floatx()) * 2 * phi / r_on
        mid = K.cast(K.lesser(phi, r_on), K.floatx()) * \
              K.cast(K.greater(phi, r_on * 0.5), K.floatx()) * (
              2 - (2 * phi / r_on))
        end = K.cast(K.greater(phi, r_on * 0.5), K.floatx()) * self.alpha * phi
        k = up + mid + end

Here, don't you think the end should be:

        end = K.cast(K.greater(phi, r_on), K.floatx()) * self.alpha * phi

Well a little bug, but this may make the time gate never closed.

PhasedLSTM can accelerate inference too?

Support for load_model()

keras.models.load_model() does not seem to be supported. Keras throws a ValueError: Unknown layer: PhasedLSTM exception when it tries to deserialize the first PLSTM layer.

_preprocess gates

Thanks a lot for your implementation. This is not an issue but a newbie question:

You applied the time_distributed function in order to compute gates for all timesteps in parallel. Can we expect this is done when using a simple tensorflow BasicLSTMCell, or is this feature of your code an improvement over tensorflow LSTM implementation ?

phased_lstm_keras problem? add_weight () got multiple values for argument 'name'

The below error appeared in https://github.com/abr/neurips2019/blob/master/experiments/psMNIST-phased-lstm.ipynb

This has been reported as a keras issue with a suggested workaround. that may be applied in PhasedLSTM-Keras.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-c56ccd8ad126> in <module>
      4 model.add(PLSTM(200, input_shape=(n_pixels, 1), 
      5                 implementation=0,
----> 6                alpha=0.0001)
      7          )
      8 model.add(Dense(10, activation='softmax'))

~\Miniconda3\envs\lmu\lib\site-packages\keras\engine\sequential.py in add(self, layer)
    164                     # and create the node connecting the current layer
    165                     # to the input layer we just created.
--> 166                     layer(x)
    167                     set_inputs = True
    168             else:

~\Miniconda3\envs\lmu\lib\site-packages\keras\legacy\layers.py in __call__(self, inputs, initial_state, **kwargs)
    511         # modify the input spec to include the state.
    512         if initial_state is None:
--> 513             return super(Recurrent, self).__call__(inputs, **kwargs)
    514 
    515         initial_state = to_list(initial_state, allow_tuple=True)

~\Miniconda3\envs\lmu\lib\site-packages\keras\engine\base_layer.py in __call__(self, inputs, **kwargs)
    461                                          'You can build it manually via: '
    462                                          '`layer.build(batch_input_shape)`')
--> 463                 self.build(unpack_singleton(input_shapes))
    464                 self.built = True
    465 

~\Miniconda3\envs\lmu\lib\site-packages\phased_lstm_keras\PhasedLSTM.py in build(self, input_shape)
    190                                       initializer=self.kernel_initializer,
    191                                       regularizer=self.kernel_regularizer,
--> 192                                       constraint=self.kernel_constraint)
    193 
    194         self.recurrent_kernel = self.add_weight(

TypeError: add_weight() got multiple values for argument 'name'

ValueError: Unkown layer: PhasedLSTM

Hi,
Keras does not recognize the PhasedLSTM layer:
model_plstm = Sequential()
model_plstm.add(PLSTM(10, input_shape=(100, 2)))
model_plstm.compile('rmsprop', 'mse')
model_plstm.save('model_plstm.h5')
model_plstm = load_model('model_plstm.h5')

Weights and biases go to NAN eventually

I've used a PLSTM layer in 3 different configurations on the same data, and after 50-200 epochs it will increasingly have more and more of the weights as NAN until all of them are, and the loss goes to zero.

The input data has no NANs, and has never produced a result like this with plain LSTM or GRU layers.

I suspect that the divisions for the time node are eventually failing, as they don't seem to have any method to prevent overflow.

For irregular time sequences

Hi,
I found out that in your implementation you mentioned for "time related variables, simply add +1 to t for now..."
I wonder if there is a way of handling irregular sequences?

nan loss after a number of steps (epochs)

I am able to train a regression network without error using the base keras LSTM layer but always seem to run into a nan loss (mse) after what starts out as very promising results using this PLSTM layer. I have tried most of the recommendations in these two keras issues #2134 and #1244 but nothing seems to help. Do you have any troubleshooting recommendations concerning this issue using your PLSTM implementation?

edit
I was able to get through 10x as many steps using the adamax optimizer (as opposed to rmsprop or adam), increased layer sizes all around, and an extra dense layer with a tanh activation. Unfortunately, the loss still went to nan. Again, the training loss was significantly better than anything I was able to squeeze out of the plain LSTM approach. 😕

Including asynchronous sequences

Hi!

Really loved your implementation, worked like a charm. This is not really an issue, but a question. If I want to give asynchronous sequences to the Phased LSTM layer, I should just change the step definition in the class right? Like the way you mentioned in the comments.

So my input shape would be (?, seq_len, 1 + nb_features) (first index of axis 2 would be the time difference bw the current and previous sample). Then slice it, adding the first index to t_tm1 instead of just 1, and doing the LSTM calculations in the normal manner with x[1 :]