Permission denied when trying to download NCDC data from S3

[brunitto]

Permission denied when trying to download NCDC data from S3

[brunitto]

Tried within an EC2 instance in us-east-1 region.

[aliyevnb]

tried it with emr cluster in us-east-1, looks like permission issue. aws cli command return following (snippet)
..
download failed: s3://hadoopbook/ncdc/all/2001.gz to ../../mnt/2001.gz An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
..

[aliyevnb]

It's a wilde guess but there is a most likely access is restricted to a AWS us-east-1 subnets on S3 GetObjects, because I can list objects with no issues. I assume bucket policy has restrictions on us-east-1 ip range. Latest us-east-1 public ip range available here:

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.service | contains("EC2")) | select(.region | contains("us-east-1"))'

[dahejin]

Remove me from this list. From: Bruno Moyle <[email protected]> Sent: Saturday, July 11, 2020 12:50 PM To: tomwhite/hadoop-book <[email protected]> Cc: Subscribed <[email protected]> Subject: Re: [tomwhite/hadoop-book] Permission denied when trying to download NCDC data from S3 (#30) Closed #30 <#30> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRQLMKOGKT2K2VMVZFL66DR3C65XANCNFSM4DH6NENA> . <https://github.com/notifications/beacon/AFRQLMMBKJ3ONZTAMGS6K43R3C65XA5CNFSM4DH6NENKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGO2LFXQXQ.gif>