Cannot read file from Azure Blob Path

[angel-hernandez91]

Cannot read file from Azure Blob Path

[angel-hernandez91]

If you put another file type (e.g. parquet) in the same blob store, are you able to read it with that config set?

Yes I am. So it turns out the credentials needed to be set differently for the SAS reader since it's using the Spark Context as opposed to the Spark Session variable.

I was able to set the credentials using: sc._jsc.hadoopConfiguration().set("fs.azure.account.key.{}.blob.core.windows.net".format(account_name), account_key)

I imagine the same would work from an S3 path.

So I guess this can be marked as resolved. It would be great to add this tidbit of information regarding the credentials to the README docs.

[thesuperzapper]

If you put another file type (e.g. parquet) in the same blob store, are you able to read it with that config set?

[Antalares]

Hi,

I'm facing the same issue.
The command "sc._jsc.hadoopConfiguration().set("fs.azure.account.key.{}.blob.core.windows.net".format(account_name), account_key)" can resolve this issue but this implies storing credential on cluster and not only for spark session.

There is a workaround byusing following command ?
spark.conf.set( "fs.azure.account.key.{}.blob.core.windows.net".format(account_name), account_key )

Thank you