aarande/nexradaws

This python package is designed to provide query and download capabilities to the NEXRAD archive available on Amazon Web Services (AWS). https://aws.amazon.com/public-datasets/nexrad/

nexradaws

This module is designed to allow you to query and download Nexrad radar files from Amazon Web Services S3 Storage. The real-time feed and full historical archive of original resolution (Level II) NEXRAD data, from June 1991 to present, is now freely available on Amazon S3 for anyone to use. More information can be found here https://aws.amazon.com/public-datasets/nexrad/.

nexradaws supports Python 2.7 and Python 3.6.

Github - https://github.com/aarande/nexradaws

PyPi - https://pypi.python.org/pypi/nexradaws

Docs - http://nexradaws.readthedocs.io/en/latest/

Required dependencies

• boto3
• pytz
• six

Optional dependencies

• pyart

Install with pip::

pip install nexradaws

New in version 1.1:

• Bug fix for varying filename extensions over the years (.gz .V06 etc). Thanks Nick Guy for the PR!

nexradaws's Issues

OSError: unknown compression record

Hi,

I am getting an error similar to ARM-DOE/pyart#499

nexradaws 1.1 pypi_0 pypi
arm_pyart 1.11.0 py37h3b54f70_0 conda-forge

I have 4 cases where I am trying to get and plot radar data. I think it's more likely to be a nexradaws issue rather than a pyart issue, but I am not sure. I'm looping through 4 cases where data is downloaded and plotted for each scan. It works for 3 of the 4 cases but it does not work for the June 7th, 2014 12z - June 9th, 2014 6z case from KDOX.

###########

Downloaded KDOX20140607_121639_V07.gz
Downloaded KDOX20140607_124600_V07.gz
Downloaded KDOX20140607_123613_V07.gz
Downloaded KDOX20140607_120654_V07.gz
Downloaded KDOX20140607_122626_V07.gz
Downloaded KDOX20140607_125546_V07.gz
6 out of 6 files downloaded...0 errors
KDOX volume scan time 2014-06-07 12:06:54+00:00
KDOX volume scan time 2014-06-07 12:16:39+00:00
KDOX volume scan time 2014-06-07 12:26:26+00:00
KDOX volume scan time 2014-06-07 12:36:13+00:00
KDOX volume scan time 2014-06-07 12:46:00+00:00
KDOX volume scan time 2014-06-07 12:55:46+00:00
0 downloads failed.
[]
2014-06-07_12:00 /Users/allenea/Documents/Eric_Allen/observation_data/Python_Programs/plots/Radar/Delaware/all_scans/pyCase_20140607/
Traceback (most recent call last):

File "", line 1, in
runfile('/Users/allenea/Documents/Eric_Allen/observation_data/Python_Programs/NEXRAD_Radar_Plotting.py', wdir='/Users/allenea/Documents/Eric_Allen/observation_data/Python_Programs')

File "/Users/allenea/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "/Users/allenea/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "/Users/allenea/Documents/Eric_Allen/observation_data/Python_Programs/NEXRAD_Radar_Plotting.py", line 141, in
plot_radar_data(RADAR_ID, OUTDIR, CASESTUDY_TIME, RUN_TIME_DAY, RUN_TIME_HOUR)

File "/Users/allenea/Documents/Eric_Allen/observation_data/Python_Programs/NEXRAD_Radar_Plotting.py", line 80, in plot_radar_data
radar = scan.open_pyart()

File "/Users/allenea/anaconda3/lib/python3.7/site-packages/nexradaws/resources/localnexradfile.py", line 56, in open_pyart
return pyart.io.read_nexrad_archive(self.filepath)

File "/Users/allenea/anaconda3/lib/python3.7/site-packages/pyart/io/nexrad_archive.py", line 101, in read_nexrad_archive
nfile = NEXRADLevel2File(prepare_for_read(filename))

File "/Users/allenea/anaconda3/lib/python3.7/site-packages/pyart/io/nexrad_level2.py", line 135, in init
raise IOError('unknown compression record')

OSError: unknown compression record

Function get_avail_scans should check for existence of resp['Contents']

If no data at all is available for a given date, then resp['Contents'] does not exist in the function get_avail_scans. This raises an error, while more desired behaviour is to return an empty list I think.

I found this issue because I wanted to retrieve data for a given time range for one day, for which there indeed is data. For the next date there however there is no data, and it appears that the function get_avail_scans_in_range includes by default the next date in its attempt to obtain available scans, if the time span is nonzero. The error that I mentioned above caused get_avail_scans_in_range to return an error, instead of a list with scans that were in fact available.

Here is a simple fix:
if 'Contents' in resp: for scan in resp.get('Contents'): match = self._scan_re.match(scan.get('Key')) if match is not None: scans.append(AwsNexradFile(scan))

Despite having the module installed fails to import (Ubuntu 16.04.5 LTS)

When I pip install nexradaws, it gives no errors or warnings. However, when I run python, this happens:

Python 2.7.12 (default, Nov 12 2018, 14:36:49)                                                                                                                                                                                   
[GCC 5.4.0 20160609] on linux2                                                                                                                                                                                                       
Type "help", "copyright", "credits" or "license" for more information.                                                                                                                                                             
>>> import nexradaws                                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                                     
File "<stdin>", line 1, in <module>                                                                                                                                                                                                
ImportError: No module named nexradaws                                                                                                                                                                                               
>>>

Function get_avail_scans_in_range fails when attempting to get current UTC day data.

When attempting to retrieve a range of data for the UTC current day, the function get_avail_scans_in_range fails. It is works correctly if you input days prior to current. Furthermore, get_avail_scans shows available files that can be downloaded for the available range.

Here is the error thrown.

  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/nexradaws/nexradawsinterface.py", line 195, in get_avail_scans_in_range
    radar.upper())
  File "/usr/local/lib/python2.7/site-packages/nexradaws/nexradawsinterface.py", line 160, in get_avail_scans
    for scan in resp.get('Contents'):
TypeError: 'NoneType' object is not iterable

To reproduce.

>>> from datetime import datetime
>>> import nexradaws
>>> conn = nexradaws.NexradAwsInterface()

Look for available scans for current UTC day. In this case 05/03.
>>> scans = conn.get_avail_scans('2019', '05', '03', 'KTLX')
>>> print scans
[<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_000406_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_001346_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_002326_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_003305_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_004245_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_005225_V06>, 
<AwsNexradFile object - 2019/05/03/KTLX/KTLX20190503_010204_V06>]

Data is available for download.
>>> results = conn.download(scans[0:7], tempdir)
Downloaded KTLX20190503_000406_V06
Downloaded KTLX20190503_001346_V06
Downloaded KTLX20190503_005225_V06
Downloaded KTLX20190503_003305_V06
Downloaded KTLX20190503_002326_V06
Downloaded KTLX20190503_004245_V06
Downloaded KTLX20190503_010204_V06
7 out of 7 files downloaded...0 errors

Set time in UTC for range that you want to download. In this case 05/03 at hour 00.
>>> start = datetime(2019,5,3,0,0)
>>> print start
2019-05-03 00:00:00
>>> end = datetime(2019,5,3,1,0)
>>> print end
2019-05-03 01:00:00

Attempt to get scans in the range start and end times.
>>> scans = conn.get_avail_scans_in_range(start, end, 'KTLX')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/nexradaws/nexradawsinterface.py", line 195, in get_avail_scans_in_range
    radar.upper())
  File "/usr/local/lib/python2.7/site-packages/nexradaws/nexradawsinterface.py", line 160, in get_avail_scans
  for scan in resp.get('Contents'):
TypeError: 'NoneType' object is not iterable

TypeError: 'NoneType' object is not iterable

Error when getting available scans for a timeframe and radar.

Full Traceback

Traceback (most recent call last):
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\flask\app.py", line 2528, in wsgi_app
   response = self.full_dispatch_request()
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\flask\app.py", line 1825, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\flask\app.py", line 1823, in full_dispatch_request
   rv = self.dispatch_request()
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\flask\app.py", line 1799, in dispatch_request
   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
 File "C:\Users\Aareon\Desktop\NEXRADAWSDataAccess\webapp\app.py", line 51, in getData
   return getDataAWS(radarSite, ssYear, ssDay, ssMonth, ssTimeFromHR, ssTimeFromMIN, ssTimeToHR, ssTimeToMIN)
 File "C:\Users\Aareon\Desktop\NEXRADAWSDataAccess\webapp\app.py", line 21, in getDataAWS
   scans = conn.get_avail_scans_in_range(start, end, radar)
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\nexradaws\nexradawsinterface.py", line 192, in get_avail_scans_in_range 
   availscans = self.get_avail_scans('{0:0>2}'.format(day.year),
 File "C:\Users\Aareon\.virtualenvs\NEXRADAWSDataAccess-Wr4jm0uy\lib\site-packages\nexradaws\nexradawsinterface.py", line 160, in get_avail_scans
   for scan in resp.get('Contents'):
TypeError: 'NoneType' object is not iterable

TypeError: 'NoneType' object is not iterable when trying to get available scans in range for current day

Hello,

I've been working with your package to retrieve a list of files for a specific radar from an AWS bucket, and ran into a problem with retrieving a list of radar scans for the current day.

Example code that throws an error:

import nexradaws as nexr
from datetime import datetime, timedelta

start_time = datetime.now().replace(hour=10, minute=0, second=0, microsecond=0)
end_time = start_time + timedelta(hours=2)
radar_id = "KEAX"

conn = nexr.NexradAwsInterface()
file_list = conn.get_avail_scans_in_range(start_time, end_time, radar_id)

>>> TypeError: 'NoneType' object is not iterable

We have linked this error to the _datetime_range() function within get_avail_scans_in_range(), which is defined as:

    def _datetime_range(self, start=None, end=None):
        span = end - start
        if span.seconds > 0:
            numdays = span.days + 1
        else:
            numdays = span.days
        for i in range(0, numdays + 1):
            yield start + timedelta(days=i)

The issue is with the +1s in the above function, and the datetime range it returns includes the current and following day to what is defined as end_time.
Therefore, if the end_time is defined as the current day, it is trying to look for files from the following day, which do not exist, and returns the TypeError. The function performs as expected for all other days that are not the current day.

Thank you for looking into this.

Add Realtime AWS Feature

Would love to see an option to query for the latest NEXRAD data available.

SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed

I’ve been trying to get one of the NEXRAD on AWS examples from https://nexradaws.readthedocs.io/en/latest/index.html to run in a python script.

conn = nexradaws.NexradAwsInterface()
years = conn.get_avail_years()
print( years )
days = conn.get_avail_days( '2013', '05')
print( days )    

and I get this error

SSL validation failed for https://noaa-nexrad-level2.s3.amazonaws.com/?delimiter=%2F&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)

google has some suggestions, but none of them helps me.

I was wondering anyone has a solution.

Thanks