katjensen / cygnss-wetlands Goto Github PK
View Code? Open in Web Editor NEWA repo dedicated to investigating global inundation patterns with NASA CYGNSS observations and other supporting data sources.
A repo dedicated to investigating global inundation patterns with NASA CYGNSS observations and other supporting data sources.
Description
File fragments lead to unreadable data
I found that when utilizing the downloader via commands such as cygnss download --product_level=L1 --start_date=2020-04-18 --end_date=2020-04-18 --dest_dir=/Volumes/Apricorn/cygnss_data
occasionally I would receive file sizes that are ~95kb. See SS below. This would happen for long stretches (Almost all of 2021)
To Reproduce
So far unable to reproduce.
Steps to reproduce the behavior:
cygnss download --product_level=L1 --start_date=2020-04-01 --end_date=2020-06-30 --dest_dir=/Volumes/Apricorn/cygnss_data
Expected behavior
A clear and concise description of what you expected to happen.
Do not accept downloads that are not valid & delete invalid downloads when rerunning the download script
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
This may be related to errors such as Could not download file: cyg02.ddmi.s20200414-000000-e20200414-235959.l1.power-brcs.a31.d32.nc, error: HTTP Error 404: Not Found
Description
Receive an HTTP 404 error when attempting to download file utilizing this command cygnss download --product_level=L1 --start_date=2020-01-01 --end_date=2020-01-31 --dest_dir=/path/to/cygnss/data
Specifically happens for cyg04.ddmi.s20200130-000000-e20200130-235959.l1.power-brcs.a31.d32.nc
To Reproduce
Steps to reproduce the behavior:
cygnss download --product_level=L1 --start_date=2020-01-30 --end_date=2020-01-30 --dest_dir=/path/to/data
Expected behavior
If file does not exist, handle exception.
Screenshots
N/A, See error text below
Additional context
The command cygnss download --product_level=L1 --start_date=2020-01-01 --end_date=2020-01-31 --dest_dir=/path/to/cygnss/data
is referenced in the README
Automated Testing
Error Text
File "/Users/sberndt/anaconda3/bin/cygnss", line 8, in <module>
sys.exit(entry())
^^^^^^^
File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/cli.py", line 116, in entry
main(obj={})
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/cli.py", line 109, in download
filelist = http_download_by_date(product_level, date, dest_subdir, overwrite)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/Documents/cygnss-wetlands/cygnss_wetlands/cygnss/download.py", line 108, in http_download_by_date
with urllib.request.urlopen(request) as response, open(complete_filepath, "wb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
result = self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
return self.parent.open(new, timeout=req.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
result = self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1056, in http_error_401
response = self.http_error_auth_reqed('www-authenticate',
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1005, in http_error_auth_reqed
return self.retry_http_basic_auth(host, req, realm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 1020, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
result = self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
return self.parent.open(new, timeout=req.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 557, in error
result = self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 749, in http_error_302
return self.parent.open(new, timeout=req.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 563, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Users/sberndt/anaconda3/lib/python3.11/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found```
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.