nilsmagnus/grib

Golang GRIB2 parser

grib's Issues

export float data to json

Here is my script to get wind data from NOAA

#!/bin/bash
DATE=`date +%Y%m%d`
echo "GET ${DATE}"
curl "http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_1p00.pl?file=gfs.t00z.pgrb2.1p00.f000&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&dir=%2Fgfs.${DATE}00" -o gfs.t00z.pgrb2.1p00.f000

With this command I get integers instead of floats
./grib -file gfs.t00z.pgrb2.1p00.f000 -latMin -90000000 -latMax 90000000 -longMin 0 -longMax 359000000 -export 3

Corrupt message "runtime error: index out of range"

Hello,

I apologize in advance, I'm falling into the grib files recently, to replace an old Python import which I understand nothing. So I try to understand but I admit that it's a bit complicated ^^

I downloaded a grb file from the NOAA FTP site, but when I try to export to json, I have these following errors.

I am trying to extract meteorological data from grib files for a weather site so I am trying to do a complete redesign.

Thanks for your help

$ go version
go version go1.6.2 linux/amd64

$ grib -export 3 -file pgb174.grb
Input parameters : griblib.Options{Operation:"parse", Discipline:-1, DataExport:true, Category:-1, Filepath:"pgb174.grb", ReduceFilePath:"reduced.grib2", ExportType:3, MaximumNumberOfMessages:2147483647, GeoFilter:griblib.GeoFilter{MinLat:-90000000, MaxLat:90000000, MinLong:0, MaxLong:360000000}, Surface:griblib.Surface{Type:0xff, Scale:0x0, Value:0x0}} 
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Corrupt message "runtime error: index out of range"
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
Using GeoFilter %#V {-90000000 90000000 0 360000000}
panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x56cea0, 0xc82000e0c0)
        /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
github.com/nilsmagnus/grib/griblib.filterValuesFromGeoFilter(0xc82f3d9a00, 0x55d4a80faa2b580, 0x15752a0000000000, 0x35, 0x0, 0x0)
        /home/jewome62/go/src/github.com/nilsmagnus/grib/griblib/filters.go:97 +0x2b7
github.com/nilsmagnus/grib/griblib.Filter(0xc8ccba1000, 0x1a1, 0x200, 0x5a1dc8, 0x5, 0xffffffffffffffff, 0x1, 0xffffffffffffffff, 0x7fff9a5faff8, 0xa, ...)
        /home/jewome62/go/src/github.com/nilsmagnus/grib/griblib/filters.go:41 +0x47d
main.parse(0x7f65fa73d770, 0xc820090078, 0x5a1dc8, 0x5, 0xffffffffffffffff, 0x1, 0xffffffffffffffff, 0x7fff9a5faff8, 0xa, 0x5a8cc0, ...)
        /home/jewome62/go/src/github.com/nilsmagnus/grib/main.go:121 +0x1e4
main.main()
        /home/jewome62/go/src/github.com/nilsmagnus/grib/main.go:72 +0x490

Fix data3 parsing

Seems like the data in Data3 is parsed incorrectly. Have a look at how it is done in https://github.com/Unidata/thredds/blob/5.0.0/grib/src/main/java/ucar/nc2/grib/grib2/Grib2DataReader.java#L374

and re-implement the parsing.

Unexpected grid0 message length

I'm attempting to decode the attached grib file, which is grid0 with an Ni = 1440 and Nj = 721. I've dealt with similar grib files using griblib in the past, but I'm seeing this particular file return a single message with a Section7.Data length of 468056 where I would typically expect 1038240 (Ni * Nj).

I've opened the grib file with an independent viewer and the data is listed there as expected (Grid: 1440x721=1038240 points). I've also confirmed that the Section5.DataTemplateNumber is 3, which I would expect.

Is this a decoding issue, or some grib peculiarity that I'm missing?

Push release tag

Can you please create a new release, so that the latest version will be taken into account when using dep (https://golang.github.io/dep/docs/daily-dep.html) ?

Thanks

grib2influx panics with out of range

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/nilsmagnus/grib/griblib.ParseData3(0x848a20, 0xc42908a060, 0x22eb, 0xc42004d080, 0x5, 0xc42d3c1438, 0x7000022f0)
/home/larsgard/go/src/github.com/nilsmagnus/grib/griblib/data.go:215 +0x10fd
github.com/nilsmagnus/grib/griblib.ReadSection7(0x848a20, 0xc42908a060, 0x22eb, 0x40000000, 0x6258d19a01010004, 0x6deffffffff, 0x100000300, 0x6a00000001, 0x10208, 0x0, ...)
/home/larsgard/go/src/github.com/nilsmagnus/grib/griblib/sections.go:341 +0x4d
github.com/nilsmagnus/grib/griblib.readMessage(0x848a20, 0xc42908a060, 0x202000047524942, 0x23ba, 0x202000047524942, 0x23ba, 0x1010200000007, 0xc0b0c07e1, 0x100, 0x0, ...)
/home/larsgard/go/src/github.com/nilsmagnus/grib/griblib/sections.go:117 +0x537
github.com/nilsmagnus/grib/griblib.ReadMessage(0x849720, 0xc42000e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/larsgard/go/src/github.com/nilsmagnus/grib/griblib/sections.go:86 +0x2d9
github.com/nilsmagnus/grib/griblib.ReadMessages(0x849720, 0xc42000e040, 0x0, 0x0, 0x0, 0x1f96, 0x7ffdf9daecfc)
/home/larsgard/go/src/github.com/nilsmagnus/grib/griblib/sections.go:51 +0xa9
main.main()
/home/larsgard/go/src/github.com/nilsmagnus/grib2influx/main.go:47 +0x17a

Reduce ends up with invalid grid

When reducing file, the grid is left untouched even if the filter-options for coordinates is used.

Fix: update latitude/longitude start/stop when reducing on coordinates.

Section 5 JPEG2000 support

I'm working with the library to decode JPEG2000 data as spec'd here. I'd like to put together a pull request with my work but am unsure how the maintainers wish to handle decoding of the JPEG2000 format.

It doesn't appear that anyone has put together a native Go JPEG2000 decoder and as such I've been making use of gographic's Imagick bindings to convert the data into a native Image instance.

Would this work still be of value to the project given that it introduces two dependencies (the Imagick binding and the imagemagick binary suite)?

godoc

add godoc to github

Eror while parsing data3

data3 is not parsed correctly

You can check with : http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_1p00.pl?bottomlat=-90&dir=%2Fgfs.2019010612&file=gfs.t12z.pgrb2.1p00.f006&leftlon=0&lev_10_m_above_ground=on&rightlon=360&toplat=90&var_UGRD=on&var_VGRD=on

on the second message, data should be:
-1.09462
-1.03462
-0.964617
-0.904617
-0.834617
-0.774617
-0.714617
-0.644617
-0.574617
-0.514617
-0.444617

But I get:
-1.094617
-1.034617
14.255383
14.235383
14.255383
14.235383
14.245383
14.255383
14.245383
14.235383
14.255383

Clean up bitreader

Seems like parts of bitreader.go is copy-paste from the golang bzip2 package. Because of this, license and creditation needs to be updated.

There is duplicate code, fix this by using 1 of the implementations.

Crash while reading file

http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_1p00.pl?bottomlat=-90&dir=%2Fgfs.2019020512&file=gfs.t12z.pgrb2.1p00.f003&leftlon=0&lev_10_m_above_ground=on&rightlon=360&toplat=90&var_UGRD=on&var_VGRD=on

panic: runtime error: makeslice: len out of range

goroutine 1 [running]:
github.com/nilsmagnus/grib/griblib.ReadMessage(0x177be20, 0xc420336060, 0x0, 0x17e6fc8, 0x0)
github.com/nilsmagnus/grib/griblib/sections.go:80 +0xb4
github.com/nilsmagnus/grib/griblib.ReadMessages(0x177be20, 0xc420336060, 0xc42004f520, 0x55ec6e, 0x7fa8dea7a160, 0xc420382060, 0xc42004f520)
github.com/nilsmagnus/grib/griblib/sections.go:58 +0x9b

when support grib1?

how about grib1

Template number not supported: 41 error

I downloaded the gribb file from weather.gov and I get this error when
Input parameters : {"operation":"parse","discipline":-1,"dataExport":true,"category":-1,"filePath":"latest.grib2","reduceFilePath":"reduced.grib2","exportType":3,"maximumNumberOfMessages":2147483647,"geoFilter":{"minLat":0,"maxLat":36000000,"minLong":-9000000,"maxLong":9000000},"surfaceFilter":{"type":0,"scale":0,"value":0}}
Error when parsing a message, Template number not supported: 41
[
]