对于 LSTM 来说，状态应该是 (c, h) tuple，因为下一步计算需要用到前一步的 c 和 h。但是对于 SRU，下一步只需用到前一步的 c ，<stro

就是说， <div class="snippet-clipboard-content notranslate position-relative overflow-

Why (c, h) tuple? about tensorflow_mnist_sru HOT 1 OPEN

soloice commented on June 16, 2024 2

Why (c, h) tuple?

from tensorflow_mnist_sru.

Comments (1)

soloice commented on June 16, 2024

就是说，

def call_without_highway(self, x, state, scope=None):
        with tf.variable_scope(scope or type(self).__name__):                        
            c, _ = state
            x_size = x.get_shape().as_list()[1]
            
            W_u = tf.get_variable('W_u', [x_size, 3 * self.output_size])
            b_f = tf.get_variable('b_f', [self._num_units])
            b_r = tf.get_variable('b_r', [self._num_units])

            xh = tf.matmul(x, W_u)
            z, f, r = tf.split(xh, 3, 1)            

            f = tf.sigmoid(f + b_f)
            r = tf.sigmoid(r + b_r)            

            new_c = f * c + (1 - f) * z
            new_h = r * tf.tanh(new_c)

            return new_h, (new_c, new_h)

可以改成

def call_without_highway(self, x, state, scope=None):
        with tf.variable_scope(scope or type(self).__name__):                        
            c = state  # first modification
            x_size = x.get_shape().as_list()[1]
            
            W_u = tf.get_variable('W_u', [x_size, 3 * self.output_size])
            b_f = tf.get_variable('b_f', [self._num_units])
            b_r = tf.get_variable('b_r', [self._num_units])

            xh = tf.matmul(x, W_u)
            z, f, r = tf.split(xh, 3, 1)            

            f = tf.sigmoid(f + b_f)
            r = tf.sigmoid(r + b_r)            

            new_c = f * c + (1 - f) * z
            new_h = r * tf.tanh(new_c)

            return new_h, new_c  # second modification

from tensorflow_mnist_sru.

Recommend Projects