Comments (27)
This is awesome guys!
I took the fix and threw a bunch of refactoring on top to hopefully arrive at the same functionality you're getting now, but with the addition of a products
configurable in the configuration.
Some relevant commits:
- 878326a renames
product
toorigin
to make room for a newproducts
configurable that's used for the actual L2 product names. - 980d169 adds the
products
field and indeed extracts the product name from the filename as @eminence suggested earlier in this thread. I didn't find any other place where it is encoded. I would have expected it to sit in the ancillary text header tbqh. - 647f8ac adds support for negative matching. This means we don't have to list all L2 products in a
products
field for the handler to pick it up (and to differentiate between CMIP and non-CMIP handlers), but instead can have one withproducts = [ "cmip" ]
and another withproducts = [ "^cmip" ]
. Of course you can still list them out if you want to use different directories, filenames, etc. - 24c7fcb uses the product name as lookup key in the gradient map, so that all these awesome gradient maps work out of the box.
I ran the following configuration against a few sample L2 LRITs and it looks like it works:
[[handler]]
type = "image"
origin = "goes16"
products = [ "^cmip" ]
directory = "./goes16/non-cmip/{product:short|lower}/{time:%Y-%m-%d}"
filename = "GOES16_{product:short}_{region:short}_{time:%Y%m%dT%H%M%SZ}"
format = "jpg"
[[handler.map]]
path = "/usr/local/share/goestools/ne/ne_50m_admin_0_countries_lakes.json"
color = "#FFFF00"
[[handler.map]]
path = "/usr/local/share/goestools/ne/ne_50m_admin_1_states_provinces_lakes.json"
color = "#FFFF00"
from goestools.
I'm working on the lookups now, here's a pair of examplers for RRQPEF and LSTF:
https://imgur.com/a/peEYjTE
from goestools.
Looks like this file name gave it issues too:
what(): Assertion `rv == 2` failed at /root/Builds/goestools/src/goesproc/handler_goesr.cc:29, Expected to
extract mode and channel from file name (OR_ABI-L2-ACHTF-M6_G16_s20191981810396_e20191981820104_c20191981821149.lrit)
from goestools.
According to docs here, the channel part of the filename is used for "single band CMI and DMW products only". In these two cases, we have different products (RRQPEF
--> Rainfall Rate/QPE FullDisc; ACHTF
--> Cloud Top Temperature FullDisc).
It looks like the product type is only encoded in the filename. So maybe we have to start parsing the filename to handle products that don't have a channel?
from goestools.
The two parts that would be of importance would probably be the following:
goestools/src/goesproc/handler_goesr.cc
Lines 21 to 31 in b0700de
goestools/src/goesproc/handler_goesr.cc
Lines 69 to 84 in b0700de
The filename builder takes in the channel structs in order to stick them in the right folder, I haven't looked at the contents of the products yet, so I'm not sure if they'd need their own color lookup tables as well.
from goestools.
It looks like @pietern has already addressed this commit before, but after running lritdump, it appears that the non CMIP products have an ancillary text header now.
/root/OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit:
Primary (0):
File type: 0 (Image Data)
Header header length (bits/bytes): 34080/4260
Data length (bits/bytes): 235358208/29419776
Total length (bits/bytes): 235392288/29424036
Image structure (1):
Bits per pixel: 8
Columns: 5424
Lines: 5424
Compression: 1
Image navigation (2):
Projection name: geos(-75.0) z▒7z▒7▒
Column scaling: 20425338
Line scaling: 20425338
Column offset: 2711
Line offset: 2711
Image data function (3):
Data:
$HALFTONE:=8
_NAME:=rainfall_rate
_UNIT:=mm h-1
0:=0.0000
<-- Rest of Lookup table -->
255:=99.6186
Annotation (4):
Text: OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit
Time stamp (5):
Time stamp: 2019-07-17 23:01:04
Ancillary text (6):
Text: Time of frame start = 2019-07-17T22:50:39.8Z;Satellite = G16;Instrument = FM1;Channel = N/A;Imaging Mode = ABI Mode 6;Region = Full Disk;Reso lution = 2km at nadir;Segmented = no
NOAA LRIT (129):
Agency signature: NOAA
Product ID: 16
Product SubID: 0
Parameter: 0
NOAA-specific compression: 1
Rice compression (131):
Flags: 177
Pixels per block: 32
Scan lines per packet: 1
from goestools.
So maybe we should start supporting these products.
Here's an example of what a FD rainfall rate image looks like: https://0x0.st/z9Du.jpg (with associated data)
from goestools.
There is definitely some interesting data here. Here's TPW (Total Precipitable Water)
And SST (Sea Surface Skin Temperature)
from goestools.
I suppose we could create some lookup tables to make the images a bit more pleasing to the eye as well, the problem is we'd need a way to implement a way to distinguish these images, as currently the LUTs are applied based on the channel iirc.
from goestools.
Did you guys check the json files? With most of these products there's a table with floats included that seem to be some min->max gradient for whatever's in the images. Most of those that i've seen had that.
Perhaps that could be used for making things look prettier...
Also i have a quick fix for unknown products in my fork. I just don't want to do a PR at this point, i feel this can be implemented much nicer than i did.
from goestools.
Mostly a work in progress, this is what I have so far based on the max/min values:
https://0x0.st/z956.txt
from goestools.
And here's some for SSTF, DSIF; it seems that clouds prevent the imager from seeing parts of the image, which I suppose makes sense, like in the sea surface temperature where the clouds make the ocean appear as if it's frigid.
Also, just as an aside, it must be a formatting issue of some sort, but goesproc refuses to apply gradients if I include the new handler in my existing goesproc.conf, so as it stands I have to have a separate instance handling these new products. https://0x0.st/zpsD.conf
from goestools.
I don't know if this is helpful for calibrating the LUTs, but here's a noaa-colorized image of sea-surface temps (I have no idea if its generated from this same GOES product, though. The descriptions are pretty far over my head)
from goestools.
Ah I just noticed I didn't actually upload the images in my last post, here they are:
https://imgur.com/a/e8uN8r7
As for that image you posted, I could adjust the colors to look like that, it is quite a bit more pleasing to the eye.
from goestools.
@eminence, I followed the NESDIS link on the page you just sent and found this image:
https://www.ospo.noaa.gov/data/sst/contour/global.cf.gif
This ended up being pretty useful and I was able to make a new gradient with it:
[handler.gradient.SSTF]
points = [
{ units = 271, color = "#cc04ae" },
{ units = 273.25, color = "#b005c6" },
{ units = 275.5, color = "#8f06e4" },
{ units = 277.75, color = "#6c1ff7" },
{ units = 280, color = "#486afa" },
{ units = 282.25, color = "#24b4fc" },
{ units = 284.5, color = "#00ffff" },
{ units = 287.25, color = "#00e696" },
{ units = 289, color = "#00d54b" },
{ units = 291.3, color = "#0cc600" },
{ units = 293.6, color = "#59d800" },
{ units = 295.85, color = "#a5ea00" },
{ units = 298.1, color = "#f2fc00" },
{ units = 300.35, color = "#ffc900" },
{ units = 302.6, color = "#ff9400" },
{ units = 304.85, color = "#ff5500" },
{ units = 307.1, color = "#ff1500" },
{ units = 340, color = "#ffffff" }
]
Here's a resulting image:
https://imgur.com/a/D0YNC00
from goestools.
That looks pretty awesome! The purple between South America and Africa (and in the Gulf of Mexico) doesn't seem quite right, though.
BTW, what code are you running locally? The code from #58, or did you come up with your own fix?
from goestools.
I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc
It makes sense since the cloud tops are going to be much, much colder than the ocean below.
from goestools.
Ahh, very interesting. Good point. I suppose this would have been accounted for in global.cf.gif
from goestools.
Here's a better table for LSTF, based off of this, the color scale for the Aqua and Terra sats
https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTD_CLIM_M
[handler.gradient.LSTF]
points = [
{ units = 248, color = "#beeaea" },
{ units = 256.75, color = "#3692e6" },
{ units = 265.5, color = "#3251cd" },
{ units = 274.5, color = "#631ebf" },
{ units = 283, color = "#c6019d" },
{ units = 291.75, color = "#e30d00" },
{ units = 300.5, color = "#ec6200" },
{ units = 309.25, color = "#f1c900" },
{ units = 318, color = "#f6ec79" },
{ units = 353.2, color = "#ffffff" },
{ units = 354, color = "#000000" }
]
Result: https://imgur.com/a/KQpK6n6
from goestools.
New TPWF from NOAA, and a better DSIF from CIMSS
Here's the color tables:
[handler.gradient.DSIF]
points = [
{ units = 0, color = "#543e20" },
{ units = 500, color = "#a58154" },
{ units = 750, color = "#cfa370" },
{ units = 1000, color = "#c3afc2" },
{ units = 1500, color = "#8181c5" },
{ units = 1750, color = "#6363a3" },
{ units = 2000, color = "#969656" },
{ units = 2500, color = "#fffc02" },
{ units = 2750, color = "#ffd327" },
{ units = 3000, color = "#ff9063" },
{ units = 3500, color = "#ff231e" },
{ units = 4000, color = "#b10032" },
{ units = 4250, color = "#680063" },
{ units = 4500, color = "#8d0090" },
{ units = 4961.3914, color = "#ea00ef" },
{ units = 4980.9245, color = "#000000" }
]
[handler.gradient.TPWF]
points = [
{ units = 0, color = "#3a290b" },
{ units = 11, color = "#d1a777" },
{ units = 12, color = "#8999ee" },
{ units = 22, color = "#584c96" },
{ units = 23, color = "#9a6a5c" },
{ units = 32, color = "#72a429" },
{ units = 33, color = "#97c03c" },
{ units = 43, color = "#f4fe00" },
{ units = 44, color = "#fd775e" },
{ units = 53, color = "#982a19" },
{ units = 54, color = "#6f0069" },
{ units = 65, color = "#fb00ee" },
{ units = 66, color = "#ffc8ff" },
{ units = 69, color = "#f0bbef" },
{ units = 99, color = "#f069ef" },
{ units = 99.6187, color = "#ffffff" },
]
Examples: https://imgur.com/a/dThFOkY
All that's left is RRQPEF, which I should finish Soon™
from goestools.
Here's my latest goesproc.conf: https://0x0.st/zpbk.conf
It's a modified goesproc-goesr.conf.in, so it should process all the products from GOES-16, just remember to replace ${CMAKE_INSTALL_PREFIX} with whatever you're using on your system.
I don't have any ACHTF LRIT files, so I didn't have a chance to test that gradient.
from goestools.
I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc
It makes sense since the cloud tops are going to be much, much colder than the ocean below.
First off, awesome work @eminence with the quick patch and @tmbates12 with the gradients for all of these Level II products!
https://www.goes-r.gov/products/baseline-SST.html
As seen above, the SSTF product would use the ABI cloud and ice masks to blank out those regions. This data product does not currently seem to use those masks. That said, any temperature below 270.65K (-2.5C) is highly likely to be indicating ice of some form or another -- either sea ice or ice clouds. I'm currently using a slightly modified version of @tmbates12 's SSTF gradient, with a couple of extra points on the cold end to blank out ice and cloud tops.
[handler.gradient.SSTF]
points = [
{ units = 180, color = "#ffffff" },
{ units = 270.65, color = "#ffffff" },
{ units = 273.25, color = "#b005c6" },
{ units = 275.5, color = "#8f06e4" },
<...>
The resulting image looks like this: https://imgur.com/a/Pqyd3mh
There's still some lower clouds that are cluttering the output, but we lack the masks to deal with them.
from goestools.
Looks great @pietern -- been running for about 24 hours and things look solid. Closing this issue as fixed!
And thanks again to @tmbates12 for working on these gradients. A small note to anyone trying to copy them with the latest fixes from @pietern -- The product names don't end with the letter F
, so instead of [handler.gradient.SSTF]
, you should use [handler.gradient.SST]
from goestools.
@eminence Excellent! Thank you for the confirmation. You can also use the lowercase product names by the way. The handler ensures all comparisons are not case sensitive.
from goestools.
@tmbates12 Would you be interested in adding the non-CMIP gradients to the default configuration files?
from goestools.
Of course! I can get the pull done today.
from goestools.
@tmbates12 That's awesome! Thank you very much!
from goestools.
Related Issues (20)
- Error: no configuration file specified HOT 1
- Cannot start goesrecv (std::runtime_error)
- Unstable frequency HOT 1
- Raspberry pi3 install not working HOT 1
- Can not build on Ubuntu 22 HOT 2
- fatal error: proj_api.h: No such file or directory HOT 3
- Himawari 9 Product ID has been agreed to remain as 43 HOT 1
- Illegal instruction HOT 6
- goestools configuration utility
- Reading packets from a publisher (how to) HOT 1
- Linux Raspbian os for x86 epoch time is off for emwin
- EMWIN date always 12 -31 -1969 HOT 7
- fatal error: proj_api.h: No such file or directory Ubuntu server 22.04 LTS HOT 3
- Interest in a DAMSNT service being added to the project? HOT 1
- Receiving Images from GOES-16 while pointed at GOES-18
- Receiving Images from GOES-16 while pointed at GOES-18 HOT 2
- Cannot build with PROJ 8 HOT 1
- Goestools / Raspbian memory usage HOT 1
- Not receiving Dsi in non-cmip
- High Vit after RPI rebuild.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goestools.