Distcp OAuth Access

[googleclouddataproc]

Distcp OAuth Access

[dennishuo]

Where are you putting the settings? Is it in core-site.xml on all the nodes? Do you also have distcp-default.xml in the conf directories?

[mrares]

I want to have both SA and OAuth based authentication, so I populated both the SA related ones (the json keyfile and all) and the Oauth client_id and client_secret. In my core-site.xml I am setting google.cloud.auth.service.account.enable=true but then at runtime I execute:

hadoop fs -Dgoogle.cloud.auth.service.account.enable=false ... -ls gs://.../

and the above works. What doesn't work is:

hadoop distcp -Dgoogle.cloud.auth.service.account.enable=false ... hdfs://.../ gs://.../

Presumably because the settings and OAuth credentials aren't propagated to the mappers (distcp being distributed as the name implies through MR, the fs commands are local)