nrel / nsrdb Goto Github PK
View Code? Open in Web Editor NEWThis repository contains all of the methods for the NSRDB data processing pipeline.
Home Page: https://nrel.github.io/nsrdb/
License: BSD 3-Clause "New" or "Revised" License
This repository contains all of the methods for the NSRDB data processing pipeline.
Home Page: https://nrel.github.io/nsrdb/
License: BSD 3-Clause "New" or "Revised" License
Why this feature is necessary:
It would be super nice to be able to submit an NSRDB pipeline job but only actually submit 1 out of the 365 jobs to slurm so you can debug on a single job instead of a full production run. The feature would still have to enter the other jobs into the status file but only a single job would actually get submitted on the hpc.
Current alternative is probably to submit manually from python.
Why this feature is necessary:
Need a docker container for lambda
Need a container to be lambda compliant:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html
The container must have a handler wrapper as the entry point:
https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html
Example handler files:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/python/example_code/lambda/boto_client_examples/lambda_handler_scheduled.py
Charge code
Puerto Rico
Urgency / Timeframe
ASAP
Bug Description
Several ancillary variables are retrieved and interpolated from MERRA on a daily basis. Data at the end of the day is forward-filled resulting in some timesteps with constant values. Correct behavior would be to either linearly extrapolate or (ideally) retrieve the MERRA timestep around the current data to interpolate to.
Charge code
SETP 10304 71.01.01
Why this feature is necessary:
When running the albedo module it would be nice to have modis data automatically downloaded if it is available and to include verification checks. It seems that as of 2/7/22 modis data is not available later than 2017 but downloading directly from usgs will return files (just not complete ones).
Implement here
Line 236 in 2999477
Charge code
SETP > -- 10304 - 71.01.01
Bug Description
For the Puerto Rico project we need to dockerize NSRDB and you CANNOT install pyhdf from pypi (despite the presence of the package in pypi!).
As a fall back I've moved the pyhdf import into the MODIS class and removed it from the requirements.txt, that said HDF4 is a this point so archaic that we want to remove pyhdf entirely and instead pre-convert the files to HDF5 using this tool:
nsrdb/nsrdb/utilities/file_utils.py
Line 128 in 0453c2f
Charge code
Puerto Rico: WDHS 11822 00.01.10
@mikebannis I can't seem to install pyhdf which will make pip installing NSRDB a pain. Any way we can remove pyhdf and instead access modis data by:
nsrdb/nsrdb/utilities/file_utils.py
Line 167 in 930fcd0
Why this feature is necessary:
The file spec for cloud data and other data model inputs is pretty custom/hacky right now. It would be nice to be able to do source_pattern instead of source_directory and put in a linux style filepath with w*ldcards.
Charge code
None available right now and NSRDB is overspent.
Urgency / Timeframe
Not urgent but would be nice when we do more dev.
Bug Description
AOD goes above the previous limit (1.5?) during fires. We should up the AOD limit to 10. Make sure we check the bitrange.
Bug Description
The current system for mapping IMS snow data to the MODIS snow free albedo to create a composite albedo has a flaw that sometimes results in the erroneous assignment of "snowy" or "dry" along snow/no-snow boundaries. These artifacts resemble triangles or shark's teeth and are most noticeable above roughly 60N latitude.
A nearest-neighbor approach is used to find the IMS data that is closest to a given MODIS raster cell. A consequence of the polar projection is each raster cells has completely unique lat and longs, requiring a large KD tree to map IMS data to the MODIS data, which is stored as a more common WGS-84 based raster format. Making a KD tree from the full IMS data set is impractical from both a memory and time perspective. Fortunately, for the purposes of making the composite albedo, we don't care which IMS point is nearest but only if it's snowy or dry. Further, there are typically large snowy and dry areas in the IMS data. We only need to know if a modis point is in a snowy or dry area. The existence of large snowy and dry areas can be used to dramatically reduce the number of IMS points needed to classify any MODIS cell as snowy or dry.
The IMS snow data is saved in a polar projection as shown below from day 190 of 1998. The north pole is in the center of the image, with the Himalayas towards the top in yellow, and the Alaskan and Canadian pacific coast towards the bottom left. Yellow cells are snow, orange is sea ice, and blue is snow and ice free. For the NSRDB, snow and sea ice are considered equivalent.
Below is the same image zoomed and centered on Greenland.
The scipy.ndimage.morphology.binary_dilation()
function is used to find the boundaries of the snow and dry regions. Example output is shown below, blue indicates the boundary.
The edge layer indicates the boundary between snowy and dry, but does not provide enough information to classify an arbitrary location as snowy or dry. To include more information, the edge is buffered using the binary_dilation()
function. This expands the edge from one pixel wide to three as shown below, yellow indicates the boundary pixels (please ignore the change in color scheme).
In theory, the buffered boundary should contain enough information to accurately classify any location as snowy or dry. Viewing the composite data shows a number of "shark's teeth" at higher latitudes as shown in the image below of Greenland. White indicates snow from the IMS data. Black or gray areas are considered dry and show the higher resolution data from MODIS. Examples of shark's teeth are circled.
This occurs due to the mismatch between the two projections. When "unwrapped", it becomes apparent that the IMS cell density near the north pole is much less than that near the equator. The image below shows the composite albedo overlayed with the IMS points, with shape and color indicating snow, sea ice, or dry. Note the low point density near the north pole.
The image below shows the selected IMS points in the polar projection after the buffer. Yellow are snowy points, maroon are dry points, and blue points are considered unnecessary for building the KD tree. Looking at the selected points in a polar projection is appears there is enough information to define arbitrary point as snowy or dry.
The below image shows the selected points unwrapped to WGS 84 and overlaid on the composite albedo. Note that at locations of the shark's teeth there appear to be missing yellow IMS snow dots, effectively "exposing" the pink dry dots, and creating the shark's teeth. While the selected points for the KD tree appears adequate in the polar projection, when converted to WGS 84 it is apparent that more points are needed to accurately define the snow/no-snow boundary. This is generally only a probably at higher latitudes. At lower latitudes the density and general orientation of points is more consistent between the two projections.
Why this feature is necessary:
Makes things easier when sending NSRDB data to HSDS
@grantbuster The aggregation tests are failing due to to many NANs, no idea why this is happening, mind taking a look?
If the data model crashes while writing a dataset output the file could look complete but would not have all the actual data. Better practice is to write the file to temp scratch then shutil copy the file to the output dir.
Why this feature is necessary:
Coordinates datasets are often added to final NSRDB data. Might as well have that generated when running nsrdb pipeline.
Using a frequency of "10m" instead of "10min" results in the error ValueError: zero-size array to reduction operation maximum which has no identity due using a 10 month frequency for interpolation. Possible verify that "h", "min", or "t" is in the freq string?
Might be nice to have flags just for pc, shading, and remap_pc instead of having to specify files also for when running python -m nsrdb.cli direct data-model
?
Would need to add flags to cli.py and pass them to the data-model routine.
Not urgent at all though.
Why this feature is necessary:
Need "climatological" average as a backup for aerosol data.
A possible solution is:
Pull average aerosols from MERRA2, downscale to 5min/2km Puerto Rico
Charge code
WDHS 11822 00.01.10
Urgency / Timeframe
ASAP
REST2:
FARMS:
There is from nsrdb.all_sky import SZA_LIM
in extract_surfrad.py but there is no import in nsrdb/all_sky/__init__.py
Can the NSRDB requirements be relaxed to allow for any pandas version? reV requires pandas>1 but NSRDB has previously only wanted pandas<1. Pytest pass on eagle with pandas v1.2.1. To close this, lets test pandas 1.2.1 using the nsrdb pipeline (next time we run it) then relax our nsrdb requirements file.
Why this feature is necessary:
NaN values in string columns should be converted to str("None"), Timezone should be integer, Elevation should be float32 or int32. These fixes will make export to HSDS easier.
os.path.basename throws run ending error when file is missing (nan for file name)
nsrdb/nsrdb/data_model/clouds.py
Line 1439 in 19d8f7a
proposed solution:
nsrdb/nsrdb/data_model/clouds.py
Lines 1439 to 1440 in 5860c32
Malik discovered an edge case here (same code in sup3r): If row_loc
or col_loc
is [0]
then not any()
will be True even though a location was found. Should we change the not any()
to not len() > 0
?
@mikebannis, shitty ask, but we've updated our linters and it's failing on your albedo files, mind taking a look and cleaning up?
https://github.com/NREL/nsrdb/runs/2231772202?check_suite_focus=true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.