assertion failure: Expected to extract mode and channel from file name

[eminence]

assertion failure: Expected to extract mode and channel from file name

[pietern]

This is awesome guys!

I took the fix and threw a bunch of refactoring on top to hopefully arrive at the same functionality you're getting now, but with the addition of a products configurable in the configuration.

Some relevant commits:

• 878326a renames product to origin to make room for a new products configurable that's used for the actual L2 product names.
• 980d169 adds the products field and indeed extracts the product name from the filename as @eminence suggested earlier in this thread. I didn't find any other place where it is encoded. I would have expected it to sit in the ancillary text header tbqh.
• 647f8ac adds support for negative matching. This means we don't have to list all L2 products in a products field for the handler to pick it up (and to differentiate between CMIP and non-CMIP handlers), but instead can have one with products = [ "cmip" ] and another with products = [ "^cmip" ]. Of course you can still list them out if you want to use different directories, filenames, etc.
• 24c7fcb uses the product name as lookup key in the gradient map, so that all these awesome gradient maps work out of the box.

I ran the following configuration against a few sample L2 LRITs and it looks like it works:

[[handler]]
type = "image"
origin = "goes16"
products = [ "^cmip" ]
directory = "./goes16/non-cmip/{product:short|lower}/{time:%Y-%m-%d}"
filename = "GOES16_{product:short}_{region:short}_{time:%Y%m%dT%H%M%SZ}"
format = "jpg"

  [[handler.map]]
  path = "/usr/local/share/goestools/ne/ne_50m_admin_0_countries_lakes.json"
  color = "#FFFF00"

  [[handler.map]]
  path = "/usr/local/share/goestools/ne/ne_50m_admin_1_states_provinces_lakes.json"
  color = "#FFFF00"

[tmbates12]

I'm working on the lookups now, here's a pair of examplers for RRQPEF and LSTF:
https://imgur.com/a/peEYjTE

[tmbates12]

Looks like this file name gave it issues too:

what():  Assertion `rv == 2` failed at /root/Builds/goestools/src/goesproc/handler_goesr.cc:29, Expected to
 extract mode and channel from file name (OR_ABI-L2-ACHTF-M6_G16_s20191981810396_e20191981820104_c20191981821149.lrit)

[eminence]

According to docs here, the channel part of the filename is used for "single band CMI and DMW products only". In these two cases, we have different products (RRQPEF --> Rainfall Rate/QPE FullDisc; ACHTF --> Cloud Top Temperature FullDisc).

It looks like the product type is only encoded in the filename. So maybe we have to start parsing the filename to handle products that don't have a channel?

[tmbates12]

The two parts that would be of importance would probably be the following:

goestools/src/goesproc/handler_goesr.cc

Lines 21 to 31 in b0700de

	int getChannelFromFileName(const std::string& fileName) {
	const auto parts = split(fileName, '-');
	ASSERT(parts.size() >= 4);
	int mode = -1;
	int channel = -1;
	auto rv = sscanf(parts[3].c_str(), "M%dC%02d", &mode, &channel);
	ASSERTM(
	rv == 2,
	"Expected to extract mode and channel from file name (", fileName, ")");
	return channel;
	}

goestools/src/goesproc/handler_goesr.cc

Lines 69 to 84 in b0700de

	if (key == "Channel") {
	std::array<char, 32> buf;
	size_t len;
	int num = -1;
	try {
	num = std::stoi(value);
	} catch(std::invalid_argument &e) {
	num = getChannelFromFileName(fileName);
	}
	ASSERTM(num >= 1 && num <= 16, "num = ", num);
	len = snprintf(buf.data(), buf.size(), "CH%02d", num);
	details.channel.nameShort = std::string(buf.data(), len);
	len = snprintf(buf.data(), buf.size(), "Channel %d", num);
	details.channel.nameLong = std::string(buf.data(), len);
	continue;
	}

The filename builder takes in the channel structs in order to stick them in the right folder, I haven't looked at the contents of the products yet, so I'm not sure if they'd need their own color lookup tables as well.

[tmbates12]

It looks like @pietern has already addressed this commit before, but after running lritdump, it appears that the non CMIP products have an ancillary text header now.

/root/OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit:
Primary (0):
  File type: 0 (Image Data)
  Header header length (bits/bytes): 34080/4260
  Data length (bits/bytes): 235358208/29419776
  Total length (bits/bytes): 235392288/29424036
Image structure (1):
  Bits per pixel: 8
  Columns: 5424
  Lines: 5424
  Compression: 1
Image navigation (2):
  Projection name: geos(-75.0)                     z▒7z▒7▒

  Column scaling: 20425338
  Line scaling: 20425338
  Column offset: 2711
  Line offset: 2711
Image data function (3):
  Data:
    $HALFTONE:=8
    _NAME:=rainfall_rate
    _UNIT:=mm h-1
    0:=0.0000
<-- Rest of Lookup table -->
    255:=99.6186
Annotation (4):
  Text: OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit
Time stamp (5):
  Time stamp: 2019-07-17 23:01:04
Ancillary text (6):
  Text: Time of frame start = 2019-07-17T22:50:39.8Z;Satellite = G16;Instrument = FM1;Channel = N/A;Imaging Mode = ABI Mode 6;Region = Full Disk;Reso                                        lution = 2km at nadir;Segmented = no
NOAA LRIT (129):
  Agency signature: NOAA
  Product ID: 16
  Product SubID: 0
  Parameter: 0
  NOAA-specific compression: 1
Rice compression (131):
  Flags: 177
  Pixels per block: 32
  Scan lines per packet: 1

[eminence]

So maybe we should start supporting these products.

Here's an example of what a FD rainfall rate image looks like: https://0x0.st/z9Du.jpg (with associated data)

[eminence]

There is definitely some interesting data here. Here's TPW (Total Precipitable Water)

And SST (Sea Surface Skin Temperature)

[tmbates12]

I suppose we could create some lookup tables to make the images a bit more pleasing to the eye as well, the problem is we'd need a way to implement a way to distinguish these images, as currently the LUTs are applied based on the channel iirc.

[k5123]

Did you guys check the json files? With most of these products there's a table with floats included that seem to be some min->max gradient for whatever's in the images. Most of those that i've seen had that.
Perhaps that could be used for making things look prettier...

Also i have a quick fix for unknown products in my fork. I just don't want to do a PR at this point, i feel this can be implemented much nicer than i did.

[tmbates12]

Mostly a work in progress, this is what I have so far based on the max/min values:
https://0x0.st/z956.txt

[tmbates12]

And here's some for SSTF, DSIF; it seems that clouds prevent the imager from seeing parts of the image, which I suppose makes sense, like in the sea surface temperature where the clouds make the ocean appear as if it's frigid.

Also, just as an aside, it must be a formatting issue of some sort, but goesproc refuses to apply gradients if I include the new handler in my existing goesproc.conf, so as it stands I have to have a separate instance handling these new products. https://0x0.st/zpsD.conf

[eminence]

I don't know if this is helpful for calibrating the LUTs, but here's a noaa-colorized image of sea-surface temps (I have no idea if its generated from this same GOES product, though. The descriptions are pretty far over my head)

https://www.nhc.noaa.gov/sst/

[tmbates12]

Ah I just noticed I didn't actually upload the images in my last post, here they are:
https://imgur.com/a/e8uN8r7
As for that image you posted, I could adjust the colors to look like that, it is quite a bit more pleasing to the eye.

[tmbates12]

@eminence, I followed the NESDIS link on the page you just sent and found this image:
https://www.ospo.noaa.gov/data/sst/contour/global.cf.gif
This ended up being pretty useful and I was able to make a new gradient with it:

[handler.gradient.SSTF]
  points = [
    { units = 271, color = "#cc04ae" },
    { units = 273.25, color = "#b005c6" },
    { units = 275.5, color = "#8f06e4" },
    { units = 277.75, color = "#6c1ff7" },
    { units = 280, color = "#486afa" },
    { units = 282.25, color = "#24b4fc" },
    { units = 284.5, color = "#00ffff" },
    { units = 287.25, color = "#00e696" },
    { units = 289, color = "#00d54b" },
    { units = 291.3, color = "#0cc600" },
    { units = 293.6, color = "#59d800" },
    { units = 295.85, color = "#a5ea00" },
    { units = 298.1, color = "#f2fc00" },
    { units = 300.35, color = "#ffc900" },
    { units = 302.6, color = "#ff9400" },
    { units = 304.85, color = "#ff5500" },
    { units = 307.1, color = "#ff1500" },
    { units = 340, color = "#ffffff" }
  ]

Here's a resulting image:
https://imgur.com/a/D0YNC00

[eminence]

That looks pretty awesome! The purple between South America and Africa (and in the Gulf of Mexico) doesn't seem quite right, though.

BTW, what code are you running locally? The code from #58, or did you come up with your own fix?

[tmbates12]

I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc
It makes sense since the cloud tops are going to be much, much colder than the ocean below.

[eminence]

Ahh, very interesting. Good point. I suppose this would have been accounted for in global.cf.gif

[tmbates12]

Here's a better table for LSTF, based off of this, the color scale for the Aqua and Terra sats
https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTD_CLIM_M

  [handler.gradient.LSTF]
  points = [
    { units = 248, color = "#beeaea" },
    { units = 256.75, color = "#3692e6" },
    { units = 265.5, color = "#3251cd" },
    { units = 274.5, color = "#631ebf" },
    { units = 283, color = "#c6019d" },
    { units = 291.75, color = "#e30d00" },
    { units = 300.5, color = "#ec6200" },
    { units = 309.25, color = "#f1c900" },
    { units = 318, color = "#f6ec79" },
    { units = 353.2, color = "#ffffff" },
    { units = 354, color = "#000000" }
  ] 

Result: https://imgur.com/a/KQpK6n6

[tmbates12]

New TPWF from NOAA, and a better DSIF from CIMSS
Here's the color tables:

[handler.gradient.DSIF]
  points = [
    { units = 0, color = "#543e20" },
    { units = 500, color = "#a58154" },
    { units = 750, color = "#cfa370" },
    { units = 1000, color = "#c3afc2" },
    { units = 1500, color = "#8181c5" },
    { units = 1750, color = "#6363a3" },
    { units = 2000, color = "#969656" },
    { units = 2500, color = "#fffc02" },
    { units = 2750, color = "#ffd327" },
    { units = 3000, color = "#ff9063" },
    { units = 3500, color = "#ff231e" },
    { units = 4000, color = "#b10032" },
    { units = 4250, color = "#680063" },
    { units = 4500, color = "#8d0090" },
    { units = 4961.3914, color = "#ea00ef" },
    { units = 4980.9245, color = "#000000" }
  ]
  [handler.gradient.TPWF]
  points = [
    { units = 0, color = "#3a290b" },
    { units = 11, color = "#d1a777" },
    { units = 12, color = "#8999ee" },
    { units = 22, color = "#584c96" },
    { units = 23, color = "#9a6a5c" },
    { units = 32, color = "#72a429" },
    { units = 33, color = "#97c03c" },
    { units = 43, color = "#f4fe00" },
    { units = 44, color = "#fd775e" },
    { units = 53, color = "#982a19" },
    { units = 54, color = "#6f0069" },
    { units = 65, color = "#fb00ee" },
    { units = 66, color = "#ffc8ff" },
    { units = 69, color = "#f0bbef" },
    { units = 99, color = "#f069ef" },
    { units = 99.6187, color = "#ffffff" },
  ]

Examples: https://imgur.com/a/dThFOkY
All that's left is RRQPEF, which I should finish Soon™

[tmbates12]

Here's my latest goesproc.conf: https://0x0.st/zpbk.conf
It's a modified goesproc-goesr.conf.in, so it should process all the products from GOES-16, just remember to replace ${CMAKE_INSTALL_PREFIX} with whatever you're using on your system.
I don't have any ACHTF LRIT files, so I didn't have a chance to test that gradient.

[ahobgood]

I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc
It makes sense since the cloud tops are going to be much, much colder than the ocean below.

First off, awesome work @eminence with the quick patch and @tmbates12 with the gradients for all of these Level II products!

https://www.goes-r.gov/products/baseline-SST.html

As seen above, the SSTF product would use the ABI cloud and ice masks to blank out those regions. This data product does not currently seem to use those masks. That said, any temperature below 270.65K (-2.5C) is highly likely to be indicating ice of some form or another -- either sea ice or ice clouds. I'm currently using a slightly modified version of @tmbates12 's SSTF gradient, with a couple of extra points on the cold end to blank out ice and cloud tops.

  [handler.gradient.SSTF]
  points = [
    { units = 180, color = "#ffffff" },
    { units = 270.65, color = "#ffffff" },
    { units = 273.25, color = "#b005c6" },
    { units = 275.5, color = "#8f06e4" },
         <...>

The resulting image looks like this: https://imgur.com/a/Pqyd3mh

There's still some lower clouds that are cluttering the output, but we lack the masks to deal with them.

[eminence]

Looks great @pietern -- been running for about 24 hours and things look solid. Closing this issue as fixed!

And thanks again to @tmbates12 for working on these gradients. A small note to anyone trying to copy them with the latest fixes from @pietern -- The product names don't end with the letter F, so instead of [handler.gradient.SSTF], you should use [handler.gradient.SST]

[pietern]

@eminence Excellent! Thank you for the confirmation. You can also use the lowercase product names by the way. The handler ensures all comparisons are not case sensitive.

[pietern]

@tmbates12 Would you be interested in adding the non-CMIP gradients to the default configuration files?

[tmbates12]

Of course! I can get the pull done today.

[pietern]

@tmbates12 That's awesome! Thank you very much!