Strange finding: When the global seed and @tf.function decorator are used, the random sampling values of the two adjacent periods are equal

[tonyherschel]

Strange finding: When the global seed and @tf.function decorator are used, the random sampling values of the two adjacent periods are equal

[sushreebarsa]

@tonyherschel Could you try to consider explicitly clearing the cache every period.
You could also try to control the caching behavior with the jit_compile option within @tf.function by setting jit_compile to False as follows;

@tf.function(jit_compile=False)
def model(x):
    

Thank you!

[tonyherschel]

Hi, @sushreebarsa, thanks for your reply! I tried setting jit_compile to False in the @tf.function, but the results do not seem to have changed.

import os
import random
import numpy as np
import tensorflow as tf

# set seeds
SEED = 0
random.seed(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
tf.keras.utils.set_random_seed(SEED)
tf.config.experimental.enable_op_determinism()

@tf.function(jit_compile=False)
def model(x):
    err = tf.random.uniform(shape=(1,))
    loss = x + err
    return err, loss

optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
training_periods = 10

# initialize x
x = tf.Variable(tf.random.uniform(shape=(1,)), trainable=True)

for tt in range(training_periods):
    if tt % 2 == 0:
        with tf.GradientTape() as tape:
            err, loss = model(x)
        
        gradients = tape.gradient(loss, [x])  # suppose x need to be optimized
        optimizer.apply_gradients(zip(gradients, [x]))  # update x
        
        print(f"Period: {tt}, err (trained): {err.numpy()}")
        
    else:
        err, loss = model(x)
        
        print(f"Period: {tt}, err (not trained): {err.numpy()}")

log outputs

Period: 0, err (trained): [0.01975703]
Period: 1, err (not trained): [0.01975703]
Period: 2, err (trained): [0.5400312]
Period: 3, err (not trained): [0.5400312]
Period: 4, err (trained): [0.51667833]
Period: 5, err (not trained): [0.51667833]
Period: 6, err (trained): [0.4683528]
Period: 7, err (not trained): [0.4683528]
Period: 8, err (trained): [0.14856052]
Period: 9, err (not trained): [0.14856052]

[sushreebarsa]

@SuryanarayanaY I was able to replicate the issue reported here. Thank you!