Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence when call some methods of `tf.data`

[x0w3n]

Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence when call some methods of `tf.data`

[tilakrayal]

@x0w3n,
I tried to execute the mentioned code on the tf-nightly GPU environment and haven't found out_of_range issues. Kindly find the gist of it here and let me know if it is working as expected. Thank you!

[x0w3n]

@x0w3n, I tried to execute the mentioned code on the tf-nightly GPU environment and haven't found out_of_range issues. Kindly find the gist of it here and let me know if it is working as expected. Thank you!

It seems that colab is not outputting debug messages, I can see the debug messages output when I run it with the idea tool on a physical machine. As the last line of the debug message in the image shows:

[YUHSINCHENG1230]

i have the same problem!!

[tilakrayal]

@x0w3n,
I tried to execute the mentioned code on the tensorflow 2.16 GPU environment and haven't faced the issue. Also I observed that the mentioned OUT_OF_RANGE is the warning which might not effect the execution of the code and the output.

Thank you!

[x0w3n]

I think this output may be caused by gpu unavailability. Although it does not affect the execution of the code, the reason for this output is still unknown. Thank you!

[tilakrayal]

@x0w3n,
The above code was executed in the GPU environment, and the output was also expected. In the screenshot it was available. Could you please provide the GPU details which you are trying. I suspect it might the GPU issue for the different output in your machine. Thank you!

[dryglicki]

I'm going to add some color here, since I'm running into the same issue.
Here is my code to load my TFRecords dataset (a dummy dataset I created for another issue):

        TFRECDIR = Path('tfrecords')
        fileList = tf.data.Dataset.list_files(str(TFRECDIR / '*tfrecords'), shuffle = True)
        ds = tf.data.TFRecordDataset(
                fileList, compression_type = 'GZIP').map(
                        _parse_function,
                        num_parallel_calls = tf.data.AUTOTUNE
                    ).batch(
                        4, drop_remainder = True
                    ).prefetch(
                        tf.data.AUTOTUNE)

...

model.fit(ds, epochs = 10)

Since I am using Keras, the first epoch looks like this:

8/Unknown 6s 531ms/step - loss: 25.4941 - mae: 4.4993 - mse: 25.49412024-06-14 14:08:00.703161: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence

For the second Epoch, it then says the correct number of steps (8), but at the end of the epoch, I get the same warning thrown. So this raises a few questions.

1. How do I prescribe the number of files in my dataset before I get to fit?
2. What do I have to do to stop this warning?
3. Are TFRecords files not meant to be one-at-a-time? My true dataset has many larger values, with files around 30 MB/each.

Before we get to the GPU issue, I'd point out
TF v2.16.1
Keras v3.3.3

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.