adlfs support

[google]

adlfs support

[Conchylicultor]

Good for me if that works for you!

[Conchylicultor]

Thank you for the suggestion. With the fsspec backend, it should be quite straightforward (using Azure extension for fsspec: https://github.com/fsspec/adlfs)

There's only 3 places to change:

• Add a condition here to load the adlfs fsspec backend:
  

  etils/etils/epath/backend.py

  Line 349 in 470e159

  elif path.startswith('s3://'):

• Add GCS as an accepted URI scheme:

  etils/etils/epath/gpath.py

  Line 38 in 470e159

  _URI_PREFIXES = ('gs://', 's3://')

• Currently, for backward compatibility, if TF is installed, the tf.io.gfile backend will be used instead of fsspec. You need to exclude az:// from this condition that force the TF backend:

  etils/etils/epath/gpath.py

  Line 110 in 470e159

  if _is_tf_installed() and self._uri_scheme is not None:

Don't hesitate to send a PR. Happy to review it.

[daniel-character]

Thanks @Conchylicultor. I'm ready to send you a PR. I think I need you to give me the right permissions for that.

[Conchylicultor]

Thanks @Conchylicultor. I'm ready to send you a PR. I think I need you to give me the right permissions for that.

Thanks for the contribution. I'm not sure I understand which permission you need. Everyone is able to send a PR on any repository: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request

[daniel-character]

You're right, I just needed a fork. In any case, I just realized something:

So my actual end goal is to make tfds work on az:// paths, and because I'm using tfds via tf.data (instead of using say datasource), tfds actually will read files from the given dataset path via two different paths: etils and tensorflow_io. Unfortunately tensorflow_io expects the az:// path to start with the azure account name, while fsspec does not (it gets the account name from e.g., an env var).

That probably means that if we want tfds to work with az://, I can do all the pieces you suggested excepted this part:

"Currently, for backward compatibility, if TF is installed, the tf.io.gfile backend will be used instead of fsspec. You need to exclude az:// from this condition that force the TF backend".

That way tfds should theoretically be able to work with az whether you use it with tensorflow or not.

What do you think?

[Conchylicultor]

#470 was merged. Closing this! Thanks again for the contribution