Grain vs. `tf.data` Input Pipeline

[google]

Grain vs. `tf.data` Input Pipeline

[rwitten]

@aireenmei to give context.

Mostly runtime performance is that both are fast enough to not be your bottleneck but Grain gives determinism guarantees!

[aireenmei]

Hi @leandrolcampos,

The biggest benefit of grain is determinism. No matter how your run gets interrupted, you can always recover the same data order from your checkpoint, so your loss curve is reproducible, debuggable.
Other benefits includes:

• minimal dependency, especially if you don't want to depends on TensorFlow
• Close to a global shuffle, while the shuffle in tf.data is limited by memory
• For some transformations, it may be easier to implement in numpy

And like I mentioned with you in another thread under grain repo, I just put up some grain docs: https://github.com/google/grain/tree/main/docs

[leandrolcampos]

Hi @rwitten and @aireenmei,

Thanks for your answers. They helped me to better understand the PyGrain value proposition. I'm looking forward to use it in my next research projects.

All the best,