Giter Site home page Giter Site logo

grib.jl's Introduction

GRIB.jl

Documentation Build Status

The Gridded Binary (GRIB) format is a format commonly used in meteorology. A GRIB file is a collection of independent records that store 2D data. This package is an interface to the ECMWF ecCodes library. In ecCodes, each GRIB file is composed of a series of messages, and a message is an object with keys and values. Each message has many keys. Some are actually stored in the data, while others are computed by ecCodes on access. Some commonly used keys include:

key value
name long name of the quantity
shortName standard abbreviation of name
latitudes array of latitudes
longitudes array of longitudes
values array of data values
units units of quantity
date date in YYYYmmdd format
typeOfLevel kind of vertical level
level value of vertical level

Installation

You can install this package through the normal methods: Pkg.add("GRIB") or ]add GRIB.

This package currently doesn't work on Windows. Any help in getting it to work would be greatly appreciated!

GribFile

A GribFile functions similarly to a Julia IOStream, except that instead of working as a stream of bytes, GribFile works as a stream of messages. Basic access looks like

GribFile(filename) do f
    # Do things with f
end

Using the do-block construct guarantees that the resources are released after exiting the do-block. The style

f = GribFile(filename)
# Do things with f
destroy(f)

is also valid, but be sure to call destroy when finished with the file.

A GribFile is an iterable, and it defines seek, skip, and seekfirst to aid in navigating the file.

Message

The Message type represents a single GRIB record. It works like a dictionary where the indices are of type String. Creating a Message looks like

GribFile(filename) do f
    # Get the first message from f
    msg = Message(f)

    # Get the second and third messages from f
    msgs = read(f, 2)

    # Work on the rest of the messages in f
    for message in f
        # Do things with message
    end
end

An important thing to note is that creating a Message moves the position of the GribFile. This means that in order to access a message that has already passed, the seek function must be used to change the position of the GribFile.

The Message type behaves like a dictionary in that data is accessed like value = msg[key]. The most important keys are listed above, and the other keys are discoverable using the keys function. The keys function returns an iterable, but this iterable does not define length, so the best way to get a list of all keys in the message is to do

keylist = Vector{String}()
for key in keys(message)
    push!(keylist, key)
end

Another important function for the Message type is data. This function returns a tuple of longitudes, latitudes, and values for each point in the message. The following is true

lons, lats, values = data(message)
valuesfromkey = data["values"]
valuesfromkey == values

The eachpoint function returns an iterable that iterates through the points returning a longitude, latitude, and value for each point.

GRIB messages often have missing values. The value that represents missing can be discovered using the missingvalue function. There is also a convenience function maskedvalues that returns the values with the missing values replaced with missing.

Calling print or println on a Message returns the hexidecimal representation of the message, since those functions are meant return a string that can be used to recreate the object. Use display instead for an informative summary of the Message.

Index

The Index type is a way to reduce the size of a file so that only messages with specific key-value pairs are included. A typical use-case looks like this:

Index(filename, "shortName", "typeOfLevel", "level") do index
    select!(index, "shortName", "t")
    select!(index, "typeOfLevel", "isobaricInhPa")
    select!(index, "level", 500)
    for msg in index
        # Do things with msg
    end
end

This example selects all messages that are temperature at the 500 hPa level. Indexes are invaluable for reducing the complexity of the file before retreiving data from it. There are a few important things to note:

  • Only keys passed to the Index when it is created can be select!ed.
  • All keys passed to Index must be select!ed before accessing any messages.
  • Like with GribFile, retreiving a message from an Index advances the Index.
  • Only the latest value select!ed per key is kept in the Index.
  • Files with multi-field messages cannot be used with Index. This includes most files created by NCEP.

Other notes

This package has support for multi-field messages on by default. If you are not working with files with multi-field messages, you may turn off support with a call to the function nomultisupport.

Bug Reporting

This package has been tested mainly with well-behaved GRIB files, but some files exist that push the boundaries of the format. If you encounter any issues, please file an issue. A good issue has

  • A full stack trace. The error can't be understood without knowing where it comes from.
  • The file causing the issue, if possible.
  • A hypothesis of what you think is going wrong.

Future plans (good places to help out)

  • Add support for BUFR files
  • Add examples
  • Fix the bug in eachpoint that occasionally causes Julia to segfault (probably something with the GC).

grib.jl's People

Contributors

rustyconover avatar weech avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

grib.jl's Issues

Array Interface?

Hi, I was pleased to find this package existed!

At some stage I want to add a GRIB wrapper to GeoData.jl, leveraging this package. It will give a common interface and conversion to NetCDF, GeoTiff, grd and other formats.

It's much easier to write a reliable interface, and do a lot of other things if there is already an Array interface wrapping access to the underlying data as if it is a regular julia array. This can still happen inside a safe function, dispatching on the type that is passed into the do block/anonymous function.

NCDatasets.jl does this quite well for their Variable type:
https://github.com/Alexander-Barth/NCDatasets.jl/blob/master/src/NCDatasets.jl#L1063-L1130

Anyway, just a thought if you are doing any more work on this package

Cheers

Edit: in case its not clear, using indexing woudl be to allow loading subsets of the data from disk without loading the whole file, which can be much quicker when you have to load a lot of files.

Missing message names from HRRR GRIB2 file.

Hi there,

I'm using GRIB.jl to parse a HRRR forecast Grib2 file i.e.

https://pando-rgw01.chpc.utah.edu/hrrr/sfc/20200911/hrrr.t00z.wrfsfcf00.grib2

But it seems that the file isn't fully understandable as I'm missing some message names vs wgrib2.

Table of messages using GRIB.jl

Maximum/Composite radar reflectivity
unknown
Vertically-integrated liquid
Visibility
Derived radar reflectivity
Derived radar reflectivity
Derived radar reflectivity
Wind speed (gust)
U component of wind
V component of wind
U component of wind
V component of wind
Geopotential Height
Temperature
Dew point temperature
U component of wind
V component of wind
Geopotential Height
Temperature
Dew point temperature
U component of wind
V component of wind
Geopotential Height
Temperature
Dew point temperature
U component of wind
V component of wind
Temperature
Dew point temperature
U component of wind
V component of wind
Temperature
Dew point temperature
U component of wind
V component of wind
unknown
unknown
Geometric vertical velocity
MSLP (MAPS System Reduction)
Geopotential Height
unknown
Derived radar reflectivity
unknown
unknown
unknown
unknown
unknown
unknown
Vorticity (relative)
Vorticity (relative)
Hail
Hail
unknown
Lightning
U component of wind
V component of wind
Surface pressure
Orography
Temperature
Total snowfall
Moisture availability
Plant canopy surface water
Water equivalent of accumulated snow depth (deprecated)
Snow cover
Snow depth
2 metre temperature
Potential temperature
Specific humidity
2 metre dewpoint temperature
2 metre relative humidity
10 metre U wind component
10 metre V wind component
10 metre wind speed
unknown
unknown
Percent frozen precipitation
Precipitation rate
Total Precipitation
Water equivalent of accumulated snow depth (deprecated)
unknown
Freezing Rain
Storm surface runoff
Baseflow-groundwater runoff
Categorical snow
Categorical ice pellets
Categorical freezing rain
Categorical rain
Surface roughness
Frictional velocity
Sensible heat net flux
Latent heat net flux
Ground heat flux
Vegetation Type
Surface lifted index
Convective available potential energy
Convective inhibition
Precipitable water
Low cloud cover
Medium cloud cover
High cloud cover
Total Cloud Cover
Pressure
Geopotential Height
Geopotential Height
Pressure
Geopotential Height
Upward long-wave radiation flux
Downward short-wave radiation flux
Downward long-wave radiation flux
Upward short-wave radiation flux
Upward long-wave radiation flux
Visible Beam Downward Solar Flux
Visible Diffuse Downward Solar Flux
Upward short-wave radiation flux
Storm relative helicity
Storm relative helicity
U-component storm motion
V-component storm motion
Vertical u-component shear
Vertical v-component shear
Vertical u-component shear
Vertical v-component shear
Geopotential Height
Relative humidity
Pressure
Geopotential Height
Relative humidity
Pressure
Geopotential Height
Geopotential Height
Best (4-layer) lifted index
Convective available potential energy
Convective inhibition
Planetary boundary layer height
Geopotential Height
Convective available potential energy
Convective inhibition
Convective available potential energy
Convective inhibition
Geopotential Height
Pressure of level from which parcel was lifted
unknown
Land-sea mask
Sea ice area fraction
unknown
unknown
unknown
unknown

When using wgrib2 I can decode the entire file and every message has a name.

Table of contents from wgrib2.

1:0:D=20200911000000:REFC Composite reflectivity [dB]:entire atmosphere:anl:
2:569171:D=20200911000000:RETOP Echo Top [m]:cloud top:anl:
3:831923:D=20200911000000:VIL Vertically-Integrated Liquid Water [kg/m^2]:entire atmosphere:anl:
4:1214910:D=20200911000000:VIS Visibility [m]:surface:anl:
5:2405765:D=20200911000000:REFD Reflectivity [dB]:1000 m above ground:anl:
6:2613416:D=20200911000000:REFD Reflectivity [dB]:4000 m above ground:anl:
7:2809924:D=20200911000000:REFD Reflectivity [dB]:263 K level:anl:
8:3000776:D=20200911000000:GUST Wind Speed (Gust) [m/s]:surface:anl:
9:4039149:D=20200911000000:UGRD U-Component of Wind [m/s]:250 mb:anl:
10:4674949:D=20200911000000:VGRD V-Component of Wind [m/s]:250 mb:anl:
11:5285812:D=20200911000000:UGRD U-Component of Wind [m/s]:300 mb:anl:
12:5936591:D=20200911000000:VGRD V-Component of Wind [m/s]:300 mb:anl:
13:6559735:D=20200911000000:HGT Geopotential Height [gpm]:500 mb:anl:
14:7236238:D=20200911000000:TMP Temperature [K]:500 mb:anl:
15:7771059:D=20200911000000:DPT Dew Point Temperature [K]:500 mb:anl:
16:8628233:D=20200911000000:UGRD U-Component of Wind [m/s]:500 mb:anl:
17:9207521:D=20200911000000:VGRD V-Component of Wind [m/s]:500 mb:anl:
18:9784266:D=20200911000000:HGT Geopotential Height [gpm]:700 mb:anl:
19:10462982:D=20200911000000:TMP Temperature [K]:700 mb:anl:
20:11020633:D=20200911000000:DPT Dew Point Temperature [K]:700 mb:anl:
21:11977993:D=20200911000000:UGRD U-Component of Wind [m/s]:700 mb:anl:
22:12573691:D=20200911000000:VGRD V-Component of Wind [m/s]:700 mb:anl:
23:13159372:D=20200911000000:HGT Geopotential Height [gpm]:850 mb:anl:
24:13865160:D=20200911000000:TMP Temperature [K]:850 mb:anl:
25:14444833:D=20200911000000:DPT Dew Point Temperature [K]:850 mb:anl:
26:15547693:D=20200911000000:UGRD U-Component of Wind [m/s]:850 mb:anl:
27:16162963:D=20200911000000:VGRD V-Component of Wind [m/s]:850 mb:anl:
28:16766183:D=20200911000000:TMP Temperature [K]:925 mb:anl:
29:17357438:D=20200911000000:DPT Dew Point Temperature [K]:925 mb:anl:
30:18425304:D=20200911000000:UGRD U-Component of Wind [m/s]:925 mb:anl:
31:19044084:D=20200911000000:VGRD V-Component of Wind [m/s]:925 mb:anl:
32:19649770:D=20200911000000:TMP Temperature [K]:1000 mb:anl:
33:20254323:D=20200911000000:DPT Dew Point Temperature [K]:1000 mb:anl:
34:21294820:D=20200911000000:UGRD U-Component of Wind [m/s]:1000 mb:anl:
35:21908929:D=20200911000000:VGRD V-Component of Wind [m/s]:1000 mb:anl:
36:22512567:D=20200911000000:MAXUVV Hourly Maximum of Upward Vertical Velocity in the lowest 400hPa [m/s]:100-1000 mb above ground:0-0 day max fcst:
37:22761604:D=20200911000000:MAXDVV Hourly Maximum of Downward Vertical Velocity in the lowest 400hPa [m/s]:100-1000 mb above ground:0-0 day max fcst:
38:22989103:D=20200911000000:DZDT Vertical Velocity (Geometric) [m/s]:0.5-0.8 sigma layer:0-0 day ave fcst:
39:23190170:D=20200911000000:MSLMA MSLP (MAPS System Reduction) [Pa]:mean sea level:anl:
40:23810793:D=20200911000000:HGT Geopotential Height [gpm]:1000 mb:anl:
41:24520595:D=20200911000000:MAXREF Hourly Maximum of Simulated Reflectivity at 1 km AGL [dB]:1000 m above ground:0-0 day max fcst:
42:24768927:D=20200911000000:REFD Reflectivity [dB]:263 K level:0-0 day max fcst:
43:24953892:D=20200911000000:MXUPHL Hourly Maximum of Updraft Helicity over Layer 2km to 5 km AGL [m^2/s^2]:5000-2000 m above ground:0-0 day max fcst:
44:24954104:D=20200911000000:MNUPHL Hourly Minimum of Updraft Helicity [m^2/s^2]:5000-2000 m above ground:0-0 day min fcst:
45:24954316:D=20200911000000:MXUPHL Hourly Maximum of Updraft Helicity over Layer 2km to 5 km AGL [m^2/s^2]:2000-0 m above ground:0-0 day max fcst:
46:24954528:D=20200911000000:MNUPHL Hourly Minimum of Updraft Helicity [m^2/s^2]:2000-0 m above ground:0-0 day min fcst:
47:24954740:D=20200911000000:MXUPHL Hourly Maximum of Updraft Helicity over Layer 2km to 5 km AGL [m^2/s^2]:3000-0 m above ground:0-0 day max fcst:
48:24954952:D=20200911000000:MNUPHL Hourly Minimum of Updraft Helicity [m^2/s^2]:3000-0 m above ground:0-0 day min fcst:
49:24955164:D=20200911000000:RELV Relative Vorticity [1/s]:2000-0 m above ground:0-0 day max fcst:
50:24955376:D=20200911000000:RELV Relative Vorticity [1/s]:1000-0 m above ground:0-0 day max fcst:
51:24955588:D=20200911000000:HAIL Hail [m]:entire atmosphere:0-0 day max fcst:
52:25022089:D=20200911000000:HAIL Hail [m]:0.1 sigma level:0-0 day max fcst:
53:25028360:D=20200911000000:TCOLG Total Column Integrate Graupel [kg/m^2]:entire atmosphere (considered as a single layer):0-0 day max fcst:
54:25039588:D=20200911000000:LTNG Lightning [non-dim]:entire atmosphere:anl:
55:25039776:D=20200911000000:UGRD U-Component of Wind [m/s]:80 m above ground:anl:
56:26026614:D=20200911000000:VGRD V-Component of Wind [m/s]:80 m above ground:anl:
57:26980281:D=20200911000000:PRES Pressure [Pa]:surface:anl:
58:28557752:D=20200911000000:HGT Geopotential Height [gpm]:surface:anl:
59:30792529:D=20200911000000:TMP Temperature [K]:surface:anl:
60:32062137:D=20200911000000:ASNOW Total Snowfall [m]:surface:0-0 day acc fcst:
61:32062349:D=20200911000000:MSTAV Moisture Availability [%]:0 m underground:anl:
62:33486538:D=20200911000000:CNWAT Plant Canopy Surface Water [kg/m^2]:surface:anl:
63:33627333:D=20200911000000:WEASD Water Equivalent of Accumulated Snow Depth [kg/m^2]:surface:anl:
64:33665038:D=20200911000000:SNOWC Snow Cover [%]:surface:anl:
65:33689314:D=20200911000000:SNOD Snow Depth [m]:surface:anl:
66:33719835:D=20200911000000:TMP Temperature [K]:2 m above ground:anl:
67:34918028:D=20200911000000:POT Potential Temperature [K]:2 m above ground:anl:
68:35911572:D=20200911000000:SPFH Specific Humidity [kg/kg]:2 m above ground:anl:
69:37223834:D=20200911000000:DPT Dew Point Temperature [K]:2 m above ground:anl:
70:38281206:D=20200911000000:RH Relative Humidity [%]:2 m above ground:anl:
71:39672738:D=20200911000000:UGRD U-Component of Wind [m/s]:10 m above ground:anl:
72:40679141:D=20200911000000:VGRD V-Component of Wind [m/s]:10 m above ground:anl:
73:41660035:D=20200911000000:WIND Wind Speed [m/s]:10 m above ground:0-0 day max fcst:
74:42680300:D=20200911000000:MAXUW U Component of Hourly Maximum 10m Wind Speed [m/s]:10 m above ground:0-0 day max fcst:
75:43686727:D=20200911000000:MAXVW V Component of Hourly Maximum 10m Wind Speed [m/s]:10 m above ground:0-0 day max fcst:
76:44667645:D=20200911000000:CPOFP Percent frozen precipitation [%]:surface:anl:
77:44667833:D=20200911000000:PRATE Precipitation Rate [kg/m^2/s]:surface:anl:
78:44668021:D=20200911000000:APCP Total Precipitation [kg/m^2]:surface:0-0 day acc fcst:
79:44668233:D=20200911000000:WEASD Water Equivalent of Accumulated Snow Depth [kg/m^2]:surface:0-0 day acc fcst:
80:44668445:D=20200911000000:FROZR Frozen Rain [kg/m^2]:surface:0-0 day acc fcst:
81:44668657:D=20200911000000:FRZR Freezing Rain [kg/m^2]:surface:0-0 day acc fcst:
82:44676422:D=20200911000000:SSRUN Storm Surface Runoff [kg/m^2]:surface:0-0 day acc fcst:
83:44676634:D=20200911000000:BGRUN Baseflow-Groundwater Runoff [kg/m^2]:surface:0-0 day acc fcst:
84:44676846:D=20200911000000:CSNOW Categorical Snow [-]:surface:anl:
85:44677034:D=20200911000000:CICEP Categorical Ice Pellets [-]:surface:anl:
86:44677222:D=20200911000000:CFRZR Categorical Freezing Rain [-]:surface:anl:
87:44677410:D=20200911000000:CRAIN Categorical Rain [-]:surface:anl:
88:44677598:D=20200911000000:SFCR Surface Roughness [m]:surface:anl:
89:46534705:D=20200911000000:FRICV Frictional Velocity [m/s]:surface:anl:
90:47439768:D=20200911000000:SHTFL Sensible Heat Net Flux [W/m^2]:surface:anl:
91:48553481:D=20200911000000:LHTFL Latent Heat Net Flux [W/m^2]:surface:anl:
92:49206767:D=20200911000000:GFLUX Ground Heat Flux [W/m^2]:surface:anl:
93:49778966:D=20200911000000:VGTYP Vegetation Type [Integer(0-13)]:surface:anl:
94:50560145:D=20200911000000:LFTX Surface Lifted Index [K]:500-1000 mb:anl:
95:51308486:D=20200911000000:CAPE Convective Available Potential Energy [J/kg]:surface:anl:
96:51851212:D=20200911000000:CIN Convective Inhibition [J/kg]:surface:anl:
97:52084788:D=20200911000000:PWAT Precipitable Water [kg/m^2]:entire atmosphere (considered as a single layer):anl:
98:53106726:D=20200911000000:LCDC Low Cloud Cover [%]:low cloud layer:anl:
99:53797506:D=20200911000000:MCDC Medium Cloud Cover [%]:middle cloud layer:anl:
100:54044999:D=20200911000000:HCDC High Cloud Cover [%]:high cloud layer:anl:
101:54189941:D=20200911000000:TCDC Total Cloud Cover [%]:entire atmosphere:anl:
102:54904589:D=20200911000000:PRES Pressure [Pa]:cloud base:anl:
103:55676372:D=20200911000000:HGT Geopotential Height [gpm]:cloud base:anl:
104:56934084:D=20200911000000:HGT Geopotential Height [gpm]:cloud ceiling:anl:
105:58281385:D=20200911000000:PRES Pressure [Pa]:cloud top:anl:
106:59088630:D=20200911000000:HGT Geopotential Height [gpm]:cloud top:anl:
107:60355709:D=20200911000000:ULWRF Upward Long-Wave Rad. Flux [W/m^2]:top of atmosphere:anl:
108:62012936:D=20200911000000:DSWRF Downward Short-Wave Radiation Flux [W/m^2]:surface:anl:
109:63159919:D=20200911000000:DLWRF Downward Long-Wave Rad. Flux [W/m^2]:surface:anl:
110:64939723:D=20200911000000:USWRF Upward Short-Wave Radiation Flux [W/m^2]:surface:anl:
111:65769636:D=20200911000000:ULWRF Upward Long-Wave Rad. Flux [W/m^2]:surface:anl:
112:67414596:D=20200911000000:VBDSF Visible Beam Downward Solar Flux [W/m^2]:surface:anl:
113:68105037:D=20200911000000:VDDSF Visible Diffuse Downward Solar Flux [W/m^2]:surface:anl:
114:69133047:D=20200911000000:USWRF Upward Short-Wave Radiation Flux [W/m^2]:top of atmosphere:anl:
115:70175888:D=20200911000000:HLCY Storm Relative Helicity [m^2/s^2]:3000-0 m above ground:anl:
116:71874960:D=20200911000000:HLCY Storm Relative Helicity [m^2/s^2]:1000-0 m above ground:anl:
117:73565110:D=20200911000000:USTM U-Component Storm Motion [m/s]:0-6000 m above ground:anl:
118:74515624:D=20200911000000:VSTM V-Component Storm Motion [m/s]:0-6000 m above ground:anl:
119:75470567:D=20200911000000:VUCSH Vertical U-Component Shear [1/s]:0-1000 m above ground:anl:
120:76574414:D=20200911000000:VVCSH Vertical V-Component Shear [1/s]:0-1000 m above ground:anl:
121:77654426:D=20200911000000:VUCSH Vertical U-Component Shear [1/s]:0-6000 m above ground:anl:
122:78655270:D=20200911000000:VVCSH Vertical V-Component Shear [1/s]:0-6000 m above ground:anl:
123:79658071:D=20200911000000:HGT Geopotential Height [gpm]:0C isotherm:anl:
124:81701692:D=20200911000000:RH Relative Humidity [%]:0C isotherm:anl:
125:82357827:D=20200911000000:PRES Pressure [Pa]:0C isotherm:anl:
126:83045604:D=20200911000000:HGT Geopotential Height [gpm]:highest tropospheric freezing level:anl:
127:83748981:D=20200911000000:RH Relative Humidity [%]:highest tropospheric freezing level:anl:
128:84402237:D=20200911000000:PRES Pressure [Pa]:highest tropospheric freezing level:anl:
129:85073856:D=20200911000000:HGT Geopotential Height [gpm]:263 K level:anl:
130:85702622:D=20200911000000:HGT Geopotential Height [gpm]:253 K level:anl:
131:86287577:D=20200911000000:4LFTX Best (4 layer) Lifted Index [K]:180-0 mb above ground:anl:
132:87239308:D=20200911000000:CAPE Convective Available Potential Energy [J/kg]:180-0 mb above ground:anl:
133:87865366:D=20200911000000:CIN Convective Inhibition [J/kg]:180-0 mb above ground:anl:
134:88198479:D=20200911000000:HPBL Planetary Boundary Layer Height [m]:surface:anl:
135:91069323:D=20200911000000:HGT Geopotential Height [gpm]:level of adiabatic condensation from sfc:anl:
136:93769410:D=20200911000000:CAPE Convective Available Potential Energy [J/kg]:90-0 mb above ground:anl:
137:94268232:D=20200911000000:CIN Convective Inhibition [J/kg]:90-0 mb above ground:anl:
138:94482824:D=20200911000000:CAPE Convective Available Potential Energy [J/kg]:255-0 mb above ground:anl:
139:95134732:D=20200911000000:CIN Convective Inhibition [J/kg]:255-0 mb above ground:anl:
140:95484215:D=20200911000000:HGT Geopotential Height [gpm]:equilibrium level:anl:
141:97807227:D=20200911000000:PLPL Pressure of level from which parcel was lifted [Pa]:255-0 mb above ground:anl:
142:98911623:D=20200911000000:RHPW Relative Humidity with Respect to Precipitable Water [%]:entire atmosphere:anl:
143:100093729:D=20200911000000:LAND Land Cover (0=sea, 1=land) [Proportion]:surface:anl:
144:100144205:D=20200911000000:ICEC Ice Cover [Proportion]:surface:anl:
145:100144393:D=20200911000000:SBT123 Simulated Brightness Temperature for GOES 12, Channel 3 [K]:top of atmosphere:anl:
146:101602413:D=20200911000000:SBT124 Simulated Brightness Temperature for GOES 12, Channel 4 [K]:top of atmosphere:anl:
147:103272606:D=20200911000000:SBT113 Simulated Brightness Temperature for GOES 11, Channel 3 [K]:top of atmosphere:anl:
148:104599363:D=20200911000000:SBT114 Simulated Brightness Temperature for GOES 11, Channel 4 [K]:top of atmosphere:anl:

Thank you for writing your module, can you please let me know what you recommend to get the expected message names?

Rusty

Thread safety

Hi,

GRIB.jl doesn't appear to be thread safe, when parsing multiple files on different threads this error is thrown:

fatal flex scanner internal error--end of buffer missed

Rusty

Key not found when parsing a message

Hi @weech,

I'm continuing on my HRRR workflow but it seems I'm missing a field from the parsed messages:

Some wgrib2 output for table of contents:

72:40679141:d=2020091100:VGRD V-Component of Wind [m/s]:10 m above ground:anl:
73:41660035:d=2020091100:WIND Wind Speed [m/s]:10 m above ground:0-0 day max fcst:
74:42680300:d=2020091100:MAXUW U Component of Hourly Maximum 10m Wind Speed [m/s]:10 m above ground:0-0 day max fcst:
75:43686727:d=2020091100:MAXVW V Component of Hourly Maximum 10m Wind Speed [m/s]:10 m above ground:0-0 day max fcst:

Do you see that field that contains "anl' or "0-0 day max fcst" parsing out all of the fields from a messages that doesn't appear to be included. Here are the values that I did find omitting a few large values:

7777 7777
alternativeRowScanning 0
angleSubdivisions 1000000
average 11.78039109366541
backgroundProcess 0
binaryScaleFactor -4
bitMapIndicator 255
bitsPerValue 10
bottomLevel 250
centre kwbc
centreDescription US National Weather Service - NCEP 
cfName eastward_wind
cfNameECMF eastward_wind
cfVarName u
cfVarNameECMF u
changeDecimalPrecision 0
dataDate 20200911
dataRepresentationTemplateNumber 3
dataTime 0
day 11
decimalPrecision 0
decimalScaleFactor 0
deleteCalendarId 0
deleteLocalDefinition 0
deletePV 1
discipline 0
Dx 3000000
DxInMetres 3000.0
Dy 3000000
DyInMetres 3000.0
editionNumber 2
endStep 0
forecastTime 0
generatingProcessIdentifier 83
genVertHeightCoords 0
getNumberOfValues 1905141
globalDomain g
grib2divider 1000000
grib2LocalSectionPresent 0
GRIBEditionNumber 2
gridDefinitionDescription Lambert Conformal can be secant or tangent, conical or bipolar
gridDefinitionTemplateNumber 30
gridDescriptionSectionPresent 1
gridType lambert
groupSplittingMethodUsed 1
hour 0
hoursAfterDataCutoff 0
identifier GRIB
iDirectionIncrementGiven 0
ieeeFloats 1
ifsParam 131
ijDirectionIncrementGiven 0
indicatorOfUnitOfTimeRange 1
interpretationOfNumberOfPoints 0
is_aerosol 0
is_aerosol_optical 0
is_chemical 0
is_chemical_distfn 0
is_efas 0
is_uerra 0
iScansNegatively 0
iScansPositively 1
isConstant 0.0
isHindcast 0
jDirectionIncrementGiven 0
jPointsAreConsecutive 0
jScansPositively 1
julianDay 2.4591035e6
kurtosis 0.8099556435428337
LaD 38500000
LaDInDegrees 38.5
Latin1 38500000
Latin1InDegrees 38.5
Latin2 38500000
Latin2InDegrees 38.5
latitudeOfFirstGridPoint 21138123
latitudeOfFirstGridPointInDegrees 21.138123
latitudeOfSouthernPole 0
latitudeOfSouthernPoleInDegrees 0.0
lengthIncrementForTheGroupLengths 1
lengthOfHeaders 136
level 250
localDir grib2/tables/local/[centre]/[localTablesVersion]
localTablesVersion 1
longitudeOfFirstGridPoint 237280472
longitudeOfFirstGridPointInDegrees 237.280472
longitudeOfSouthernPole 0
longitudeOfSouthernPoleInDegrees 0.0
LoV 262500000
LoVInDegrees 262.5
masterDir grib2/tables/[tablesVersion]
maximum 66.19209671020508
md5Headers 005af0e7d154499c1609d525546854ea
md5Section1 189d5f7438ba6a4f10c886fd7f5894ae
md5Section3 78367561440d7c7b608b8532a02e4780
md5Section4 59e73b254567ff9ee7e06a4b9757a230
md5Section5 3b99042055a1d5f7d4e2bda677133b6f
md5Section6 01522c0fd43e4befb347e7995baa0167
md5Section7 a2d80a3012d6586a0eb4e978ba24a042
minimum -21.370403289794922
minute 0
minutesAfterDataCutoff 0
missingValue 9999
missingValueManagementUsed 0
missingValuesPresent [0, 0]
modelName unknown
month 9
name U component of wind
nameECMF U component of wind
nameOfFirstFixedSurface Isobaric surface 
nameOfSecondFixedSurface Missing
neitherPresent 0
numberOfBitsForScaledGroupLengths 7
numberOfBitsUsedForTheGroupWidths 4
numberOfDataPoints 1905141
numberOfGroupsOfDataValues 62618
numberOfMissing 0
numberOfOctectsForNumberOfPoints 0
numberOfOctetsExtraDescriptors 2
numberOfSection [1, 3, 4, 5, 6, 7]
numberOfValues 1905141
NV 0
Nx 1799
Ny 1059
offsetValuesBy 0.0
optimizeScaleFactor 0
orderOfSpatialDifferencing 2
packingType grid_complex_spatial_differencing
parameterCategory 2
parameterName u-component of wind 
parameterNumber 2
parameterUnits m s-1
paramId 131
paramIdECMF 131
PLPresent 0
pressureUnits hPa
primaryMissingValueSubstitute 0
productDefinitionTemplateNumber 0
productionStatusOfProcessedData 0
productType unknown
projectionCentreFlag 0
PVPresent 0
radius 6371229
referenceForGroupLengths 1
referenceForGroupWidths 0
referenceValue -21.370403289794922
referenceValueError 1.9073486328125e-6
resolutionAndComponentFlags 8
resolutionAndComponentFlags1 0
resolutionAndComponentFlags2 0
resolutionAndComponentFlags6 0
resolutionAndComponentFlags7 0
resolutionAndComponentFlags8 0
scaledValueOfEarthMajorAxis 0
scaledValueOfEarthMinorAxis 0
scaledValueOfFirstFixedSurface 25000
scaledValueOfRadiusOfSphericalEarth 0
scaledValueOfSecondFixedSurface 0
scaleFactorOfEarthMajorAxis 0
scaleFactorOfEarthMinorAxis 0
scaleFactorOfFirstFixedSurface 0
scaleFactorOfRadiusOfSphericalEarth 0
scaleFactorOfSecondFixedSurface 0
scaleValuesBy 1.0
scanningMode 64
scanningMode5 0
scanningMode6 0
scanningMode7 0
scanningMode8 0
second 0
secondaryMissingValueSubstitute 0
section0Length 16
section1Length 21
section3Length 81
section3Padding UInt8[]
section4Length 34
section5Length 49
section6Length 6
section7Length 635589
section8Length 4
sectionNumber [1, 3, 3, 4, 5, 6, 7]
selectStepTemplateInstant 1
selectStepTemplateInterval 1
setBitsPerValue 10
setCalendarId 0
shapeOfTheEarth 6
shortName u
shortNameECMF u
significanceOfReferenceTime 1
skewness 1.0168379773898042
sourceOfGridDefinition 0
standardDeviation 16.274422744167317
startStep 0
stepRange 0
stepType instant
stepTypeInternal instant
stepUnits 1
subCentre 0
tablesVersion 2
tablesVersionLatest 25
tempPressureUnits hPa
topLevel 250
totalLength 635800
trueLengthOfLastGroup 27
typeOfFirstFixedSurface pl
typeOfGeneratingProcess 2
typeOfLevel isobaricInhPa
typeOfOriginalFieldValues 0
typeOfProcessedData fc
typeOfSecondFixedSurface 255
units m s**-1
unitsECMF m s**-1
unitsOfFirstFixedSurface Pa
unitsOfSecondFixedSurface unknown
uvRelativeToGrid 1
validityDate 20200911
validityTime 0
year 2020

Error when reading data: Function not yet implemented

I installed GRIB in a liunx environment, can open and view grib file, but can not extract data from the file. I need some help. The following is the error message. Thank you.
data(msg)
ERROR: Error: Function not yet implemented
Stacktrace:
[1] errorcheck
@ ~/.julia/packages/GRIB/y8dyK/src/GRIB.jl:119 [inlined]
[2] data(handle::Message)
@ GRIB ~/.julia/packages/GRIB/y8dyK/src/message.jl:348
[3] top-level scope
@ REPL[29]:1

Fix hour field for ERA5-Land hourly data

Hi!

First of all, thank you very much for your work! This library is completely useful!

However, I've noticed a kind of bug when reading data from ERA5-Land given in hourly temporal resolution. The problem is that the hour field is not properly read, instead, every hour of each day count as the same "date" field, as the hour filed is always "0".

I hope it won't be a hard problem to solve!

Alex

Register :)

Install tagbot to automatically tag your versions:

https://github.com/marketplace/actions/julia-tagbot

Install julia registrator:

https://github.com/apps/juliateam-registrator/installations/new

  • Make sure all your dependencies in Package.toml have upper and lower bounds in the compat section. (see https://julialang.github.io/Pkg.jl/v1/toml-files/)
  • set the version to 0.1.0 in Package.toml
  • comment @JuliaRegistrator register in a comment box on the latest/supported commit

All things going to plant your package should be registered.

More details on registrator here:
https://github.com/JuliaRegistries/Registrator.jl

Reduced Gaussian Grid

The is a problem with the interface when using msg["values"] - see further below at the bottom.
However I can get the values like this:
julia> msg["latLonValues"]
19795656-element Array{Float64,1}:
89.94618771566562
180.0
0.9720458984375
...

In grib_api (if I am not mistaken) the are functions to get the closest grid point values for a list a geolocations - with the same interface no matter if the data is stored on lat/lon grid or reduced gaussian grid. Does GRIB.jl support something like that?

julia> msg = Message(f)
date gridType stepRange typeOfLevel level shortName name
20080223 reduced_gg 3 surface 0 ci Sea ice area fraction

julia> msg["values"]
ERROR: DimensionMismatch("new dimensions (2147483647, 2560) must be consistent with array size 6598552")
Stacktrace:
[1] (::Base.var"#throw_dmrsa#213")(::Tuple{Int64,Int64}, ::Int64) at ./reshapedarray.jl:41
[2] reshape at ./reshapedarray.jl:45 [inlined]
[3] reshape at ./reshapedarray.jl:116 [inlined]
[4] getindex(::Message, ::String) at /tcenas/home/hultberg/.julia/packages/GRIB/wj6Fn/src/message.jl:219
[5] top-level scope at REPL[8]:1

error retrieving values from era-20c

If you have an idea how I would do this I'd appreciate it:

I have downloaded a dataset of 50 daily global t2m values. I can access the file just fine.

f = GribFile(filename)

works and

each m=Message(f) retrieves a new day's data.

m["longitudes"] gives me an array of longitudes data (length=88838)

however

m["values"] gives me:

ERROR: DimensionMismatch("new dimensions (2147483647, 256) must be consistent with array size 88838")
Stacktrace:
[1] (::Base.var"#throw_dmrsa#213")(::Tuple{Int64,Int64}, ::Int64) at ./reshapedarray.jl:41
[2] reshape at ./reshapedarray.jl:45 [inlined]
[3] reshape at ./reshapedarray.jl:116 [inlined]
[4] getindex(::Message, ::String) at /Users/jonnorberg/.julia/packages/GRIB/wj6Fn/src/message.jl:219
[5] top-level scope at REPL[41]:1

Would you know what I am doing wrong here?

Many thanks, Jon

Co-operation on high-level GRIB interface

Hi @weech,

Thanks for your great work in building a Julia wrapper around ecCodes. We at ECMWF are working with @RobertRosca on a package called cfgrib (https://github.com/RobertRosca/cfgrib.jl) which also makes use of GRIB.jl. We would like to work with you together to build a testing chain to ensure future ecCodes change do not cause any regressions.

First of all we would like to know if there is anything we can help you with maintaining and developing grib.jl. Perhaps then we can discuss what we try to achieve with cfgrib.jl.

Looking forward working with you
Cheers
Stephan

Define Custom Exception Type

Currently we throw ErrorException for any error raised by ecCodes. It might be worthwhile to change this to a custom error type that is a subtype of Exception. This would be a breaking change, so it'd go in 0.4.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Support for Windows

We noticed that GRIB.jl does not support Windows yet.

Can you comment on why this is the case? Any plans to get it fixed?

Create definition files for NCEP local codes

NCEP commonly shares GRIB files that include local codes. As this library uses ecCodes, an ECMWF product, it has an out of date version of that code table. However, ecCodes is designed so that anyone can create definition files for local codes and include them transparently. It would be great if we had up-to-date definition files for NCEP files since they are used so often (see #4 for example for where this causes confusion). This web page describes the syntax.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.