katjensen/cygnss-wetlands

A repo dedicated to investigating global inundation patterns with NASA CYGNSS observations and other supporting data sources.

cygnss-wetlands's Issues

File Fragments Lead to Unreadable Data

Description
File fragments lead to unreadable data

I found that when utilizing the downloader via commands such as cygnss download --product_level=L1 --start_date=2020-04-18 --end_date=2020-04-18 --dest_dir=/Volumes/Apricorn/cygnss_data occasionally I would receive file sizes that are ~95kb. See SS below. This would happen for long stretches (Almost all of 2021)

To Reproduce
So far unable to reproduce.

Steps to reproduce the behavior:

1. Setup environment
2. Run script cygnss download --product_level=L1 --start_date=2020-04-01 --end_date=2020-06-30 --dest_dir=/Volumes/Apricorn/cygnss_data
3. Wait...

Expected behavior
A clear and concise description of what you expected to happen.
Do not accept downloads that are not valid & delete invalid downloads when rerunning the download script

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.
This may be related to errors such as Could not download file: cyg02.ddmi.s20200414-000000-e20200414-235959.l1.power-brcs.a31.d32.nc, error: HTTP Error 404: Not Found

HTTP404 Error when Downloading CYGNSS Data

Description
Receive an HTTP 404 error when attempting to download file utilizing this command cygnss download --product_level=L1 --start_date=2020-01-01 --end_date=2020-01-31 --dest_dir=/path/to/cygnss/data

Specifically happens for cyg04.ddmi.s20200130-000000-e20200130-235959.l1.power-brcs.a31.d32.nc

To Reproduce
Steps to reproduce the behavior:

1. In your terminal, enter the following command with your respective destination directory cygnss download --product_level=L1 --start_date=2020-01-30 --end_date=2020-01-30 --dest_dir=/path/to/data
2. Wait for the download of the first 3 spacecrafts to complete
3. Notice HTTP 404 Error terminates program, seen below.

Expected behavior
If file does not exist, handle exception.

Screenshots
N/A, See error text below

Additional context
The command cygnss download --product_level=L1 --start_date=2020-01-01 --end_date=2020-01-31 --dest_dir=/path/to/cygnss/data is referenced in the README

Automated Testing

• An automated test has been created to check for this bug in the future

Error Text

  File "/Users/sberndt/anaconda3/bin/cygnss", line 8, in <module>
    sys.exit(entry())
             ^^^^^^^
  File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/cli.py", line 116, in entry
    main(obj={})
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/cli.py", line 109, in download
    filelist = http_download_by_date(product_level, date, dest_subdir, overwrite)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/download.py", line 108, in http_download_by_date
    with urllib.request.urlopen(request) as response, open(complete_filepath, "wb") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1056, in http_error_401
    response = self.http_error_auth_reqed('www-authenticate',
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1005, in http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1020, in retry_http_basic_auth
    return self.parent.open(req, timeout=req.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found```