tomrunia/clockworkrnn

Clockwork RNN in TensorFlow

Clockwork RNN

This is a TensorFlow implementation of the Clockwork RNN proposed by Koutnik et al. (ICML, 2014). In addition to the model itself there is a small data generator for testing the prediction capability of the CW-RNN.

Dependencies

• NumPy
• TensorFlow (tested with v1.0)
• Optionally MatplotLib and Pandas when using the data generator

clockworkrnn's Issues

the issue regrading clockwork_mask (the upper triangular matrix)

According to the [arXiv, https://arxiv.org/abs/1402.3511], My understanding is the mask matrix shape is
[num_group * group_size, num_group * group_size], can also image the mask is constructed by [num_group, num_group] but each element in matrix is a sub-matrix which shape is
[group_size, group_size];

so i think the expected clockwork_mask is upper triangular matrix of [num_group, num_group], but not [num_group * group_size, num_group * group_size];

i also modify the code, change the below code:

self.clockwork_mask = tf.constant(np.triu(np.ones([self.config.num_hidden, self.config.num_hidden])), dtype=tf.float32, name="mask")

to,

        mask = np.zeros((self.config.num_hidden, self.config.num_hidden))
        for i in range(len(self.clockwork_periods)):
            mask[i*self.group_size:(i+1)*self.group_size, i*self.group_size:] = 1
        self.clockwork_mask = tf.constant(mask, name="mask")

Test the model

There seems something wrong in the implement (group_index).

Hi Tom,

I read carefully your code, since I want to implement a network very similar to CWRNN. Your code is really clear, thanks!

However, I found it seems that you didn't correctly implement CWRNN, there is a "little" bug. If you take the hidden_W variable value, for instance I take num_hidden=4, periods = [1, 2], and print the hidden_W change, I got:

...
[[  1.19209290e-07  -1.71363354e-05   0.00000000e+00   0.00000000e+00]
 [  0.00000000e+00  -4.02897596e-04   0.00000000e+00   0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]
 [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]]
...

that's to say your code just use 2 units of hidden_W. I think you forget to set the group_size of each period.

I think it's better to use tf.where rather than group_index, the group_index may cause lots of problem. Check https://github.com/braingineer/ikelos/blob/master/ikelos/layers/cwrnn.py, or later I will give a tensorflow implement.

adding dropout

How can I add dropout to the model?

Test the Model

Hi @tomrunia

I would like to test the model for my own data. Is there any script available wherein i can give an input of my choice and get the segmented output?