ncar/obs2ioda

Converter developed by @jamiebresch to transform conventional and remote sensing observations in different formats to the IODA format needed in JEDI

obs2ioda

To convert NCEP PREPBUFR/BUFR files and Himawari Standard Data FLDK files to ioda-v1 or ioda-v2 format.

To convert GOES-ABI netCDF to ioda-v1 format.

File types that obs2ioda can handle:

• prepbufr.gdas.YYYYMMDD.tHHz.nr
• gdas.satwnd.tHHz.YYYYMMDD.bufr
• gdas.gpsro.tHHz.YYYYMMDD.bufr
• gdas.1bamua.tHHz.YYYYMMDD.bufr
• gdas.1bmhs.tHHz.YYYYMMDD.bufr
• gdas.airsev.tHHz.YYYYMMDD.bufr
• gdas.cris.tHHz.YYYYMMDD.bufr
• gdas.crisf4.tHHz.YYYYMMDD.bufr
• gdas.mtiasi.tHHz.YYYYMMDD.bufr
• HS_H08_YYYYMMDD_HH00_BNN_FLDK_R20_S0210.DAT
• OR_ABI-L1b-Rad nc files

See obs2ioda-v1/README.md for compilation and usage.

See obs2ioda-v2/README.md for compilation and usage.

See goes_abi/README.md for compilation and usage.

Observation data sources:

• https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/
• https://rda.ucar.edu/datasets/ds337.0/
• https://rda.ucar.edu/datasets/ds735.0/
• https://registry.opendata.aws/noaa-goes/
• https://registry.opendata.aws/noaa-himawari/

Contributors

• Primary original developer: Jamie Bresch (@jamiebresch)

Original repository can be found at https://github.com/jamiebresch/obs2ioda

obs2ioda's Issues

obs2ioda-v2.x replaces output files for sub-hourly HSD data

It has recently been found that the obs2ioda-v2.x uses the following naming convention for output files generated for HSD data:
"ahi_himawari8_obs_yyyymmddhh.h5"

Therefore, the use of sub-hourly HSD data will lead to automatic file replacement. For example, if we use obs2ioda-v2.x to convert HSD data for the following times:

HS_H08_20180101_0000_Bxx_FLDK_R20_S0101.DAT
HS_H08_20180101_0010_Bxx_FLDK_R20_S0101.DAT
HS_H08_20180101_0020_Bxx_FLDK_R20_S0101.DAT
HS_H08_20180101_0030_Bxx_FLDK_R20_S0101.DAT
HS_H08_20180101_0040_Bxx_FLDK_R20_S0101.DAT
HS_H08_20180101_0050_Bxx_FLDK_R20_S0101.DAT

all the output files will have the same output filename: "ahi_himawari8_obs_2018010100.h5". If they were written in the same directory, output files for 00:00, 00:10, 00:20, 00:30, and 00:40 will be replaced by the output file at 00:50.

Unable to convert AHI data to IODA format

Hello! I am trying to convert AHI data from AWS cloud https://registry.opendata.aws/noaa-himawari/ but I am not able to successfully convert the data to the IODA format. I tried the Full Disk sample data from https://www.data.jma.go.jp/mscweb/en/himawari89/space_segment/spsg_sample.html as well as the files from HS_H08_20210119_1600_B09_FLDK_R20_S0510.DAT and from the period during 15APR 00Z, 2018 to 14MAY 18Z, 2018 for but without success yet. Some comments on the issues found:

1. Found an error when reading files with segment number different than 10, as in the case of data from 15APR 00Z, 2018 to 14MAY 18Z, 2018. In the code, the segment in the file name is 'S', isegm, nsegm, where nsegm is fixed as 10, but for these data is 01. I changed this value and then it can read the files.
2. Once I am able to successfully read the files, there is an issue reading the data from the headers down to the needed information. For some reason (I am not sure why), it does not find any data in the files, for example:
   header%navicorr%correctNum= 0
   then, I get the following error message:

forrtl: severe (408): fort: (8): Attempt to fetch from allocatable variable LINENO when it is not allocated

Image              PC                Routine            Line        Source             
libifcoremt.so.5   00002B90702C0F12  for_emit_diagnost     Unknown  Unknown
obs2ioda.x         000000000041D28E  ahi_hsd_mod_mp_re         324  hsd.f90
obs2ioda.x         0000000000484A0B  MAIN__                    263  main.f90
obs2ioda.x         0000000000402C92  Unknown               Unknown  Unknown
libc-2.22.so       00002B9072360A35  __libc_start_main     Unknown  Unknown
obs2ioda.x         0000000000402B99  Unknown               Unknown  Unknown

This error makes reference to a variable that is not allocated, but it's only allocated if there are observations in the file (header%navicorr%correctNum> 0), so it's related to the same problem.

In addition, I tried the sample source code in C Programming Language https://www.data.jma.go.jp/mscweb/en/himawari89/space_segment/hsd_sample/sample_code_netcdf121.zip with which I was able to convert the data to netcdf (not IODA) successfully, for the same data (e.g., HS_H08_20210119_1600_B09_FLDK_R20_S0510.DAT).

Original issue: Issue jamiebresch#3

bug syntax error

On line 306 of hsd.f90 , there is a syntax error in the comparison of two logicals. The fortran standard dictates these must be compared with .eqv. not ==

if (fexist(ij) == .false.) then

obs2ioda fails to process himawari HSD data

I tried to read and convert HSD binary data for himawari 8 (available here: /glade/derecho/scratch/mrislam/work/pandac/radiance_data_processing/obs_ioda/work/input) but it did not work as expected.

I used the following command: "./obs2ioda-v2.x -i input/ -ahi -t 201801010000", as suggested by the readme file: https://github.com/jamiebresch/obs2ioda/blob/main/obs2ioda-v2/README.md.

I tried both the obs2ioda.x (from obs2ioda-v1) and obs2ioda-v2.x (from obs2ioda-v2) built in the following directory (/glade/derecho/scratch/mrislam/work/pandac/radiance_data_processing/obs2ioda) gave me the exact same error message.

forrtl: severe (71): integer divide by zero
Image PC Routine Line Source
libpthread-2.31.s 000014FA32DF28C0 Unknown Unknown Unknown
obs2ioda-v2.x 0000000000411605 Unknown Unknown Unknown
obs2ioda-v2.x 000000000041F659 Unknown Unknown Unknown
obs2ioda-v2.x 0000000000402C8D Unknown Unknown Unknown
libc-2.31.so 000014FA32A1C29D __libc_start_main Unknown Unknown
obs2ioda-v2.x 0000000000402BBA Unknown Unknown Unknown

Screenshot:

Remove debugging build flags

obs2ioda/goes_abi/src/Makefile

Line 14 in 6d65ac6

FFLAGS = -mcmodel medium -g -traceback -debug all -check all # needed for intel error message "failed to convert GOTPCREL relocation"

After the ABI code is working and stable, the traceback and debug flags can be removed: -g -traceback -debug all