noaa-emc/upp

Unified Post Processor (UPP)

The Unified Post Processor (UPP) software package is a software package designed to generate useful products from raw model output.

The UPP is currently used in operations with the Global Forecast System (GFS), GFS Ensemble Forecast System (GEFS), North American Mesoscale (NAM), Rapid Refresh (RAP), High-Resolution Rapid Refresh (HRRR), Short Range Ensemble Forecast (SREF), and Hurricane WRF (HWRF) applications. It is also used in the Unified Forecast System (UFS), including the Rapid Refresh Forecast System (RRFS), Hurricane Analysis and Forecast System (HAFS), and the Medium-Range Weather (MRW) and Short- Range Weather (SRW) Applications.

The UPP provides the capability to compute a variety of diagnostic fields and interpolate to pressure levels or other vertical coordinates.

UPP also incorporates the Joint Center for Satellite Data Assimilation (JCSDA) Community Radiative Transfer Model (CRTM) to compute model-derived brightness temperature (TB) for various instruments and channels. This additional feature enables the generation of a number of simulated satellite products including GOES products.

Output from the UPP is in National Weather Service (NWS) and World Meteorological Organization (WMO) GRIB2 format and can be used directly by visualization, plotting, or verification packages or for further downstream post-processing, e.g., statistical post-processing techniques.

Examples of UPP products include:

• T, Z, humidity, wind, cloud water, cloud ice, rain, and snow on pressure levels
• SLP, shelter level T, humidity, and wind fields
• Precipitation-related fields
• PBL-related fields
• Severe weather products (e.g. CAPE, Vorticity, Wind shear)
• Radiative/Surface fluxes
• Cloud related fields
• Aviation products
• Radar reflectivity products
• Satellite look-alike products

User Support

Support for the UFS UPP is provided through GitHub Discussions.

Documentation

User Guide for latest standalone public release: https://upp.readthedocs.io/en/latest/.

Technical code-level documentation: https://noaa-emc.github.io/UPP/.

Developer Information

Please review the wiki

Authors

NCEP/EMC Developers

Code Managers: Wen Meng, Huiya Chuang, Fernando Andrade-Maldonado

Prerequisites

The UPP requires certain NCEPLIBS packages to be installed via the spack-stack project. For instructions on installing these packages as a bundle via spack-stack, see: https://spack-stack.readthedocs.io/en/latest/. The UPP/modulefiles directory indicates which package versions are used and supported on Level 1 systems.

Required NCEPLIBS packages:

• NCEPLIBS-g2
• NCEPLIBS-g2tmpl
• NCEPLIBS-sp
• NCEPLIBS-ip
• NCEPLIBS-bacio
• NCEPLIBS-w3emc
• CRTM

Also required to build NCEPpost executable (cmake option BUILD_POSTEXEC):

• NCEPLIBS-sigio
• NCEPLIBS-sfcio
• NCEPLIBS-nemsio

The NCEPLIBS-wrf_io library is required to build with NCEPpost with WRF-IO library (cmake option BUILD_WITH_WRFIO).

The following third-party libraries are required:

• netcdf
• netcdf-c
• netcdf-fortran
• Jasper
• libpng
• zlib
• hdf5

Building

Builds include:

• Inline post (UPP library): Currently only supported for the GFS, RRFS, HAFS, and the UFS-MRW Application.

• Offline post (UPP executable): Supported for regional applications including SRW, RRFS, HAFS, and standalone applications of UPP.

CMake is used to manage all builds of the UPP. The script UPP/tests/compile_upp.sh can be used to automatically build UPP on fully supported platforms where HPC-stack is supported. Details in this script can be used to build on new platforms.

Disclaimer

The United States Department of Commerce (DOC) GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use. DOC has relinquished control of the information and no longer has responsibility to protect the integrity, confidentiality, or availability of the information. Any claims against the Department of Commerce stemming from the use of its GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.

UPP Terms of Use Notice

The UPP Terms of Use Notice is available at: https://github.com/NOAA-EMC/UPP/wiki/UPP-Terms-of-Use-Notice

upp's Issues

Cloud ceiling failure in in-line post

The PR #136 which was merged to branch release/gfs_v16 output cloud ceiling from GFS V16. This commit causes in-line post hung in processing total cloud fraction. It was figured out that the GFS
total cloud fraction computation call UPP subroutine AllGETHERV which uses MPI_COMM_WORLD.
The MPI_COMM_WORLD will make all tasks including forecasting syncing each other. That makes the whole FV3 job hung.

Update README with better build instructions

There is a long discussion of building UPP in this issue #212 , and some of that information needs to be in the README. I will update and submit a PR.

Subscript #1 of the array ID has value 36 which is greater than the upper bound of 25

/scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/sufs/simple-ufs-aero/src/post/sorc/ncep_post.fd/MDL2P.f(2095): error #5560: Subscript #1 of the array ID has value 36 which is greater than the upper bound of 25
               ID(36)=2                                                          
---------------^                                                                 
compilation aborted for /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/sufs/simple-ufs-aero/src/post/sorc/ncep_post.fd/MDL2P.f (code 1)
make[2]: *** [sorc/ncep_post.fd/CMakeFiles/nceppost.dir/MDL2P.f.o] Error 1       
make[2]: *** Waiting for unfinished jobs....

This should be fixed:

diff --git a/sorc/ncep_post.fd/RQSTFLD.F b/sorc/ncep_post.fd/RQSTFLD.F
index 50410d1..313c509 100644
--- a/sorc/ncep_post.fd/RQSTFLD.F
+++ b/sorc/ncep_post.fd/RQSTFLD.F
@@ -41,7 +41,7 @@
                  IQ(MXFLD),IS(MXFLD),ISMSTG(MXFLD),               &
                  ISMFUL(MXFLD),ISMOUT(MXFLD),LVLS(MXLVL,MXFLD),   &
                  IDENT(MXFLD),IFILV(MXFLD),IAVBLFLD(MXFLD),       &
-                 ID(25),IGDS(18)
+                 ID(36),IGDS(18)
       real    :: DEC(MXFLD)
       integer :: num_post_afld
       integer,allocatable :: LVLSXML(:,:)

Add 3D instantaneous cloud fraction

FV3 LAM will output a 3D instantaneous cloud fraction for any microphysics scheme. The cld_amt array in gfs_dyn will no longer be used for the GFDL microphysics. The cldfra array will be read in from the gfs_phy file instead.

release/public-v1: Update cmake submodule to dbee713

Update cmake submodule to NOAA-EMC/CMakeModules@dbee713

Rename UFS release branch (again)

Further discussion and decisions have been made to use EMC's version numbers for ufs branches/tags. Need to rename release/public-v4 to release/public-v8. I can do this after current PRs are complete #76 #77 #78 .
@climbfuji Please let me know when ready for new branch to be created. Updates to submodule pointers will be needed as well. No immediate rush as I will not be available to make the new branch until this evening.

UPP naming conventions

There is inconsistency in how UPP is named, referred to, tagged for releases, etc. For example, community releases and support call it UPP and the executable is named unipost.exe, and in repository tags it uses "post". Operational applications may tag the release code named with upp, but the executable is called ncep_post. The in-line post library version is installed as ncep_post. The repository is called EMC_post. This inconsistency is simply a product of historical evolution of the package, but now is a good time to rally around a consistent naming convention.

Things to consider:

• Rename repository
• Set a standard procedures for names to tag releases (relates to #144 as well)
• Unify the executable and library names

include install directory destination is incorrect in cmake.

module_dir variable for Fortran module files in L189 is defined as:

set(module_dir "${CMAKE_CURRENT_BINARY_DIR}/include")

During install, these files are copied into the destination include as seen in L221 reproduced here as:

install(DIRECTORY ${module_dir} DESTINATION include)

This means that the final installation directory containing these Fortran module files is:

${CMAKE_INSTALL_PREFIX}/include/include

The correct L221 should be

install(DIRECTORY ${module_dir} DESTINATION ${CMAKE_INSTALL_PREFIX})

release/public-v8: Delete old makefiles

Public release branch is using cmake. Old makefiles should be deleted.

makefile
makefile_dtc
makefile_lib
makefile_LINUX
makefile_Linux_gnu
makefile_Linux_intel
makefile_Linux_pgi
makefile_module
makefile_nems
makefile.sh
makefile_wcoss

Update wrfio v1.1.1 path on theia

The wrfio v1.1.1 was built at /scratch3/NCEPDEV/nwprod/NCEPLIBS on theia. Update WRFIO_LIB
to /scratch3/NCEPDEV/nwprod/NCEPLIBS/compilers/intel/15.1.133/lib/libwrfio_v1.1.1.a.

Output new fields in gfs v16 product

Per the MEG team's request, new fields are added in gfs v16 product:

1. output cloud ceiling height and instant total cloud fraction.
2. output instant cloud fraction at low/mid/high cloud layer.
3. correct grib2 names of time averaged cloud fraction fraction at low/mid/high cloud layer from "TCDC" into "LCDC, MCDC, HCDC".
4. output radar reflectivity at 1/4 km above ground and model layer 1/2.
5. output mixed layer CAPE/CIN.
   Total is 9 fields(11 records) are added in forecast master and pgrb2 files.

release/public-v8: Post executable should be linked with "_4" version of g2tmpl

Post executable should be linked with "_4" version of g2tmpl ( not "_d" ), like all other ncep libraries.

https://github.com/NOAA-EMC/EMC_post/blob/d4a38debfa57b6b515f3ff5dff99f1e68a5bab92/sorc/ncep_post.fd/CMakeLists.txt#L201

Post executable is not compiled in double precision.

Merge release/public-v1 into develop

UFS MRW v1.1.0 was released on October 6, 2020. The release/public-v1 branch used for this release needs to be merged with develop.

Issue: This branch is quite behind develop. Need to carefully compare and take only needed changes.

• Which cmake mods to merge, if any?
• Are there docs/*.rst mods that need to be merged?
• What's changed between v1.0 and v1.1 release for upp branch?

add doxygen documentation

The NCEPLIBS team is adding doxygen based documentation to all NCEPLIBS libraries. I believe EMC_post should also add doxygen.

Doxygen is the world leader in generating documentation from code, used by many of the libraries we rely on, including netCDF and HDF5. It is widely understood and supported on all platforms.

To see the kind of documentation we have generated so far, take a look here: https://noaa-emc.github.io/NCEPLIBS/

The documentation for NCEPLIBS is still in first draft form, but here's a project with a reasonably complete example: https://noaa-emc.github.io/NCEPLIBS-sp/

If it is agreed, then the first step will be to add the docs directory, the Doxyfile.in file, and the CMake support for building doxygen docs. Then would come a series of PRs in which existing Fortran files doxygenated, converting exisisting comments into doxygen comments. For a good example of what this looks like, see this open PR: ufs-community/UFS_UTILS#198

Note that this means all programmers on EMC_post would have to respect and maintain the doxygen documentation. It is not hard to do, but I'm not signing up to be your tech writer - I have important coding to do as well. ;-) But I will get you started if you like.

The format of the doxygen comments should roughly match the rest of the NCEPLIBS documentation, since they will all be the same documentation set for the users. There is a long discussion of doxygen and how it may be used on our codes here: NOAA-EMC/NCEPLIBS#121

3DRTMA specific change

The output of 10m wind from UPP is a direct copy of the 10m wind fields in a given WRF NetCDF file. The GSI analysis does not update the 10m wind field. So using current UPP for 3DRTMA will output background 10m wind, which is not desirable. This update added a 'SUBMODELNAME="RTMA" ' logic so that UPP will update 10m wind using the wind fields at lowest HRRR model level (z=8m). It does not affect other UPP functions. #172

Small fixes for HREFv3 UPP code

The release branch "release/href_v3" was created from 3eaca89 for Matt adding small fixes for HREFv3 UPP code.

Updates in branch release/gfs_v16

The branch release/gfs_v16 was created for GFS V16 implementation. Since 08/2020, all GFS V16 related changes will be documented in this issue.

release/public-v8: Out of Bound Exception in GFSPOST.F (tpause subroutine)

When I run post compiled with a flag that checks out-of-bound exceptions, I get the following runtime error:

At line 459 of file /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/GFSPOST.F
Fortran runtime error: Index '65' of dimension 1 of array 'u' above upper bound of 64

Error termination. Backtrace:
#0  0x902e2c in tpause_
        at /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/GFSPOST.F:460
#1  0xa87d7f in miscln_
        at /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/MISCLN.f:443
#2  0x7453cc in process_
        at /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/PROCESS.f:111
#3  0x4951f3 in wrfpost
        at /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/WRFPOST.f:869
#4  0x495bcd in main
        at /home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/WRFPOST.f:134

release/public-v8: post job crashes on Cray (gaea) in MPI layer

Running the ncep_post executable of the release/public-v8 branch on gaea using the cray-intel compiler 18.0.3.222 and cray-mpich/7.4.0 crashes in grib2_module.f when trying to open the file:

forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
ncep_post          0000000001591DAE  Unknown               Unknown  Unknown
ncep_post          0000000001343190  Unknown               Unknown  Unknown
ncep_post          00000000013C36ED  Unknown               Unknown  Unknown
ncep_post          0000000001472847  Unknown               Unknown  Unknown
ncep_post          000000000147F41F  Unknown               Unknown  Unknown
ncep_post          000000000136F23B  Unknown               Unknown  Unknown
ncep_post          000000000136F96B  Unknown               Unknown  Unknown
ncep_post          000000000140C721  Unknown               Unknown  Unknown
ncep_post          0000000001418CC7  Unknown               Unknown  Unknown
ncep_post          000000000136669A  Unknown               Unknown  Unknown
ncep_post          0000000001360A7E  Unknown               Unknown  Unknown
ncep_post          00000000004C7111  grib2_module_mp_g         430  grib2_module.f
ncep_post          0000000000466A33  MAIN__                    877  WRFPOST.f

This line is

     call mpi_file_open(mpi_comm_comp,trim(post_fname),                       &
          mpi_mode_create+MPI_MODE_WRONLY,MPI_INFO_NULL,fh,ierr)

Obviously the Unknowns are coming from the MPI layer - hence not much more I can do in terms of turning on debug flags. If anyone has seen this and knows a solution, please let me know.

The compiler flags are:

  set(fortran_flags "-free -O3 -convert big_endian -traceback -g -fp-model source ${OpenMP_Fortran_FLAGS}")
  set(c_flags "-DLINUX -Dfunder -DFortranByte=char -DFortranInt=int -DFortranLlong='long long'")

The OpenMP flags are empty (OpenMP is turned off).

@GeorgeVandenberghe-NOAA @GeorgeGayno-NOAA @DusanJovic-NOAA @mark-a-potts @kgerheiser

Update documentation to reflect changes to GFS RH with refactored release

The first phase of refactored UPP code is scheduled to be merged into develop branch by end of calendar year 2020.
One major variable change is GFS RH. Need to update documentation to reflect this change.

Warning: Extension: Unary operator following arithmetic operator

[ 86%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/nceppost.dir/INITPOST_GFS_NEMS_MPIIO.f.o
/home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/INITPOST_GFS_NEMS_MPIIO.f:1159:44:

 1159 |             npass = npass + nint(0.5*(jtem--j))
      |                                            1
Warning: Extension: Unary operator following arithmetic operator (use parentheses) at (1)

EMC_post_refactor to EMC_post/develop

Phase 1 of refactor of remove goto remove grib1

Capability of detect gaussian grid scanning mode north2south vs. south2north

Add a a fix from Dusan Javic for scanning mode in POST for gaussian grid to detect north2south vs. south2north based on latstart vs latlast.
The change in sorc/ncep_post.fd/grib2_module.f as:

@@ -1389,6 +1389,11 @@
ifield3(16) = lonlast
ifield3(17) = NINT(360./(IM)*1000000.)
ifield3(18) = NINT(JM/2.0)

•   if( latstart < latlast ) then
  

•    ifield3(19) = 64      !for SN scan  
  

•   else
  

•    ifield3(19) = 0       !for NS scan
  

•   endif
  

!
!** Latlon grid
ELSE IF(MAPTYPE == 0 ) THEN

This change will be committed to branch "ufs_public_release".

Build fails on Jet due to removal of /lfs3 soft links for NCEP libraries

UPP runs slow on HERA -- staskfarm and multi-prog options

UPP runs slow on HERA because "srun" does not support MPMD or CFP. Wen Meng suggested to use staskfarm, which is equivalent to MPMD/CFP. Offline tests showed staskfarm appears to be working. However, there are two issues. First, it always creates a directory .taskfarm_job_$LSF_JOB_ID under the running directory, it contains many small job scripts to be run on each task. $LSF_JOB_ID is uniquely determined by the system and cannot be changed by users. fv3gfs_downstream_nems.sh has nset=2, and submits staskfarm twice. In the 2nd submission, jobs that have been run when nset=1 are catted to the same small job scripts and are executed one more time. I have not been able to found a solution to overcome this without restructuring the UPP script. Secondly, when staskfarm is submitted from the global-workflow by the Rocoto job scheduler, the program always fails with "slurmstepd: error: execve(): : No such file or directory". It appears environmental variables are not properly broadcasted to each task. I searched the internet and tested various options to load the modules at different locations, but could not find a solution.

Add CI with github actions?

It would benefit the NCEPLIBS team, and presumably also the UPP team, to add continuous integration to this repository with github actions.

GitHub actions are a free CI service offered by GitHub. We use it for most of the NCEPLIBS projects. A good example can be seen here:
NOAA-EMC/NCEPLIBS-bufr#34

Note that the tests are automatically run on each PR.

This is accomplished by adding one or more build files, in a special directory .github/workflows. This requires no changes to any of your codes or build system. A sample may be seen here: https://github.com/NOAA-EMC/NCEPLIBS-bufr/blob/develop/.github/workflows/main.yml

NCEPLIBS libraries are used by many UPP programs, and we need to ensure that NCEPLIBS continue to provide the services required by UPP. So UPP testing is NCEPLIBS testing too. ;-) As we test and work with NCEPLIBS, we would like to also add tests to UPP related to NCEPLIBS functionality.

If this seems a good idea, the first step would be for me to submit a build .yml file for EMC_post.

Filenames that only differ in uppercase/lowercase are incompatible with macOS

There are two pairs of files that differ only in their first character being uppercase or lowercase:

sorc/ncep_post.fd/Makefile
sorc/ncep_post.fd/makefile

and

sorc/ncep_post.fd/Makefile_lib
sorc/ncep_post.fd/makefile_lib

This is incompatible with the macOS filesystem, which is semi case-sensitive (it is case preserving, but otherwise doesn't distinguish). Are both versions required for each of those pairs? If so, can one of them be renamed?

build issues with cmake build

@aerorahul do you know what is going on with this?

Wen Meng - NOAA Affiliate
10:15 AM (1 hour ago)
to me

Hi Edward,

I have been practicing on build post executable/lib with NCEPLIBS at hpc-stack via cmake under my personal account on hera.
I did the checking up and building as:

git clone [email protected]:NOAA-EMC/EMC_post.git
cd EMC_post
mkdir build
module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack
module load hpc/1.0.0-beta1
module load hpc-intel/18.0.5.274
module load hpc-impi/2018.0.4
cmake .. -DCMAKE_INSTALL_PREFIX=./install

My building process failed at finding netcdf package as:

-- Found MPI: TRUE (found version "3.1")
CMake Error at CMakeLists.txt:30 (find_package):
By not providing "FindNetCDF.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "NetCDF", but
CMake did not find one.

Could not find a package configuration file provided by "NetCDF" with any
of the following names:

NetCDFConfig.cmake
netcdf-config.cmake

Add the installation prefix of "NetCDF" to CMAKE_PREFIX_PATH or set
"NetCDF_DIR" to a directory containing one of the above files. If "NetCDF"
provides a separate development package or SDK, be sure it has been
installed.

-- Configuring incomplete, errors occurred!
See also "/scratch2/NCEPDEV/ovp/Wen.Meng/LIBS/src/upp_v10.0.0/EMC_post/build/CMakeFiles/CMakeOutput.log".

Can you advise the fix?

Thanks,

Finalize ufs_public_release for testing within MRweather app

• New community build restructuring for DTC_post needs to be merged into ufs_public_release once testing is complete.
• cmake mods from #34 need to be merged.
• Tests need to see if new structure is compatible with umbrella nceplibs superproject.

Merge DTC_post into develop

A major sync of the DTC community branch into the develop branch was performed. This includes merging in the community build structure into the develop branch, allowing for smoother code management and development procedures between community and operations moving forward.

PR #131 details the merge.

UPP version numbers

UPP uses different version numbers for different applications. There has not been a consistent effort to maintain a proper internal UPP versions and to tie that to external applications. With UFS coming on board, and GFSv16, now is a good time to establish a set of standards for internal version numbering system.

Ability to read output from FV3 native grid for both global and SAR applications

@gsketefian @jwolff-ncar @JacobCarley-NOAA @BenjaminBlake-NOAA @JulieSchramm @mkavulich @llpcarson @panll @fentoad72

Check integrity of UPP control file for GFS

Update the GFS post control xml files to ensure enough packing precision.

Background of this issue is described in
https://docs.google.com/document/d/18GaSoq2ZSfo_WM8PPBHgQPfCJEQ9xl6Z9lTfWU-ivjg/edit?usp=sharing

A Github branch is created for this modification
https://github.com/JesseMeng-NOAA/EMC_post/tree/gfs_v16_precision

The change is summarized in
https://docs.google.com/document/d/1ulFox0KMvasvr6PyhY4Vlo4BtWZ8AAightzC1RVNF-o/edit?usp=sharing

Sample plots demonstrating the changes can be viewed in
https://www.emc.ncep.noaa.gov/mmb/gldas/gfsv16_precision

Merge release/public-v1 into develop

Changes from UFS MWR App release branch v1 need to be merged back into the develop branch. Changes should be minimal and include a couple bug fixes, clean up of makefiles, and additional files for cmake build capabilities.

Make itag fortran namelist

Request from UFS MRweather APP development. View Issue 65 on ufs-meweather-app repo.

Merge gsd UPP

Incorporate the changes of rapv5/hrrrv4 from gsd upp repository at VLAB.

Updates for Standalone release from DTC - Fall 2020

This issue contains details and tasks needed for release of standalone UPP package in conjunction with SRW release.

Details/Tasks:

• Will be same as final tag for SRW tag. Working out of the release/public-v2
• Goal is to use Cmake and retire configure/compile for community builds. (*Note: EMC/operations will continue to use gnu make and maintain their makefile)
• Uses authoritative NOAA/NCEPLIBS as dependency for build (release/public-v2 branch/tag)
• CRTM: Upgraded to v2.3.0; uses source code from NCEPLIBS
  • subset of coefficient files will be uploaded to GitHub release page
• Retire use of "unipost.exe" nomenclature from community packages and change to "ncep_post" for consistency throughout builds - COMPLETE with #197
  • Update UG to use ncep_post for executable name
  • Update run_unipost script to use ncep_post name, and rename script to run_upp
• Add fv3r model name to run script
• Version numbering for UPP will begin with v9* series in release/public-v2 - COMPLETE with #203
• Documentation: Add advanced documentation detailing how to "Add a New Variable"

Updates in branch develop

The develop is the main development branch. Since 08/17/2020, all updates in develop will be documented in this issue.

Merge cmake updates from PR #140 into release/public-v1

@aerorahul - Do the updates that went into develop on #140 need to be brought into the release/public-v1 branch in preparation for v1.1 release?

There are major differences between develop and release/public-v1 right now, so I'd like to cherry-pick the commit to bring it into release/public-v1, as opposed to merging the full develop (UFS v2.0 will branch from develop). I started this on my fork, however I'm unsure about a conflict with gitmodules settings for cmake. Should branch be modified to "develop" in release/public-v1? Can you please confirm and/or submit a PR with this cherry-pick and conflicts resolved to release/public-v1.

Modify gfs v16 products pgrb2 and pgrb2b

Fanglin sent the request for gfs v16 products:

1. move SPFH from the pgrb2b to pgrb2 group
2. In group pgrb2 , write U, V, T, RH, SPFH, W, O3, Vorticity on the same 41 isobaric layers from 1000 hPa to 0.01 hPa, Write cloud hydrometers on layers rom 1000 hPa to up to 50 hPa.
3. update analysis pgrb2 files to have the same vertical profiles as the forecast files

Reading undefined value from regional FV3 model output in netcdf

In the UPP read interface for regional FV3 in netcdf, the undefined values (missing value) is hard-wired. When the model history files have more than two undefined values, a field list has to be manually updated for the second undefined value. It is easy to make a variable's missing value mis-defined.

Need compatibility flags for gfortran-10

With gfortran 10, I still get:

/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/download/emc_post/sorc/ncep_post.fd/GFSPOST.F:221:35:

  221 |       call rsearch1(km,p,1,pvpb(k),l1)
      |                                   1
Error: Rank mismatch in argument 'l2' at (1) (rank-1 and scalar)
/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/download/emc_post/sorc/ncep_post.fd/GFSPOST.F:222:35:

  222 |       call rsearch1(km,p,1,pvpt(k),l2)
      |                                   1
Error: Rank mismatch in argument 'l2' at (1) (rank-1 and scalar)
/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/download/emc_post/sorc/ncep_post.fd/GFSPOST.F:231:33:

  231 |             call rsearch1(km,p,1,p(lu)+pd,ld)
      |                                 1
Error: Rank mismatch in argument 'z2' at (1) (rank-1 and scalar)
/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/download/emc_post/sorc/ncep_post.fd/GFSPOST.F:245:33:

  245 |             call rsearch1(km,p,1,p(lu)+pd,ld)
      |                                 1
Error: Rank mismatch in argument 'z2' at (1) (rank-1 and scalar)
/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/download/emc_post/sorc/ncep_post.fd/GFSPOST.F:436:26:

  436 |  call rsearch1(k-2,h(2),1,h(k)+hd,kd)
      |                          1
Error: Rank mismatch in argument 'z2' at (1) (rank-1 and scalar)
make[5]: *** [sorc/ncep_post.fd/CMakeFiles/nceppost.dir/GFSPOST.F.o] Error 1
make[4]: *** [sorc/ncep_post.fd/CMakeFiles/nceppost.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [nceppost/src/nceppost-stamp/nceppost-build] Error 2
make[1]: *** [CMakeFiles/nceppost.dir/all] Error 2
make: *** [all] Error 2
dom-loaner:build dom$ pwd
/usr/local/ufs-release-v2.0.0/gnu/src/NCEPLIBS/build

Seems the gfortran-10 compatibility flags are still missing. We need to fix this for the SRW App public release. I can create the PR to fix it here, then we need to create a new tag and update the NCEPLIBS umbrella builds (for both develop and release/public-v2).

Add a README in accordance with NCEPLIBS requirements

I would like to add a README to match those on the other NCEPLIBS libraries. See NOAA-EMC/NCEPLIBS#120

release/public-v1: issue with find_package(sigio REQUIRED)

I am getting this error when I run cmake:

CMake Error at CMakeLists.txt:46 (find_package):                                 
  By not providing "Findsigio.cmake" in CMAKE_MODULE_PATH this project has       
  asked CMake to find a package configuration file provided by "sigio", but      
  CMake did not find one.                                                        
                                                                                 
  Could not find a package configuration file provided by "sigio" with any of    
  the following names:                                                           
                                                                                 
    sigioConfig.cmake                                                            
    sigio-config.cmake                                                           
                                                                                 
  Add the installation prefix of "sigio" to CMAKE_PREFIX_PATH or set             
  "sigio_DIR" to a directory containing one of the above files.  If "sigio"      
  provides a separate development package or SDK, be sure it has been            
  installed.                                                                     
                                                                                 
                                                                                 
-- Configuring incomplete, errors occurred!

There is a file named cmake/Modules/FindSIGIO.cmake. Looks like cmake find_package must use correct (upper case) name of the module, here:

https://github.com/NOAA-EMC/EMC_post/blob/4511de5a221e1b9ba36f336ea60604cda4189c92/CMakeLists.txt#L46

Has this been tested on case-sensitive filesystem?

Syntax error in MDL2STD_P.f

/home/dusan/simple-ufs/src/post/sorc/ncep_post.fd/MDL2STD_P.f:254:68:                                     

  254 |             QIN(1:IM,JSTA:JEND,1:LM,nFDS)=QQW(1:IM,JSTA:JEND,1:LM)+ \
      |                                                                    1
Error: Syntax error in expression at (1)

Backslash is not standard Fortran line continuation character. Please use ampersand.

https://github.com/NOAA-EMC/EMC_post/blob/427fa51fbf42ce92d47d53b890c1865f4f296b33/sorc/ncep_post.fd/MDL2STD_P.f#L254

Update branch name to release/public-v4 for public release

It was decided to retain DTC's versions from community releases for the UFS branches and tags. DTC's UPP is working on release V4.1 at the moment. As such, need to change branch name that ufs uses for develop to release/public-v4. I will create this new branch from the existing release/public-v1 when confirmed it is ok to do so. Development will need to be done in the release/public-v4 branch after this is complete. Note that the naming convention agreed upon for tags will be v4.1.0.beta01 when we are ready. @climbfuji @mark-a-potts @kgerheiser @DusanJovic-NOAA Please let me know if you are actively working on release/public-v1 branch and want me to wait, otherwise I will plan to do this today.

Updates for public release v2 - SRW App V1.0.0

This Issue will serve as place to address all needs/updates/issues with the upcoming public-v2/SRWv1.0.0 release branch. The branch "release/public-v2" was created 17 July 2020. All development/updates for this release should be done in this branch, not develop.

Tasks & Merges to develop to be considered for merge into release/public-v2:

1. #153 (Bug/syntax fix)
2. #156 (Update to cmake pointer)
3. Updates from release/gefs_v16: #111, #136, #142, #149, and #150. -- These will no impact on fv3sar and will not change results.
4. #170 -- Instantaneous cloud fraction fix for FV3SAR needs to be added. - COMPLETE
5. Change version number/filename of modulefiles Odin and Stampede to v9.0.0 since they were updated? - dependent on implementation of upp versioning. #197
6. Will SRW use cmake or compile for UPP - Yes, eventually. (See conversation from #194 )

Modify field lists in gfs v16 PGB data sets

Per global model developer's request, modify gfs v16 PGB data sets as;

1. Make fields at isobaric levels have 41 vertical levels for all forecast hours and analysis in pgrb2 dataset.
2. Remove SPFH at isobaric levels from pgrb2b dataset.

Adding a line to fixed the bug of scattering calculation

Change line " IF (IDECID == 17 ) THEN" as
"IF (IDECID == 17 .or. IDECID == 20 .or. IDECID == 21) THEN"

FV3 LAM low/mid/high/total clouds in fv3sar.xml

The fv3sar.xml has ID's 037-039 for outputting inst low/mid/high clouds for FV3 LAM, in which post computes from the 3D cloud field. Post uses the following layers:

Low: ≥ 642 hPa
Mid: ≥ 350 hPa
High: <350 hPa

However, FV3 computes ave low/mid/high clouds using different layers. These can be output directly using ID's 300-302.

Low: ≥ 650 hPa
Mid: ≥ 400 hPa
High: <400 hPa

@WenMeng-NOAA, @JeffBeck-NOAA (or other SAR folks - feel free to tag) - I feel the preferred fields would be the direct model output, especially since the levels used are different. Same goes for total which is also directly output from model. Is there a reason the direct model output for these fields aren't being output? Also, does the 'ave' in the model output refer to time- or layer-average (more of a question for LAM folks).