noaa-emc/global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)

global-workflow

Global Workflow currently supporting the Global Forecast System (GFS) with the UFS-weather-model and GSI-based Data Assimilation System.

The global-workflow depends on the following prerequisities to be available on the system:

• Workflow Engine - Rocoto and ecFlow (for NWS Operations)
• Compiler - Intel Compiler Suite
• Software - NCEPLIBS (various), ESMF, HDF5, NetCDF, and a host of other softwares (see module files under /modulefiles for additional details)

The global-workflow current supports the following tier-1 machines:

• NOAA RDHPCS - Hera
• MSU HPC - Orion
• MSU HPC - Hercules
• NOAA's operational HPC - WCOSS2

Additionally, the following tier-2 machine is supported:

• SSEC at Univ. of Wisconsin - S4 (Note that S2S+ experiments are not fully supported)

Documentation (in progress) is available here.

Disclaimer

The United States Department of Commerce (DOC) GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use. DOC has relinquished control of the information and no longer has responsibility to protect the integrity, confidentiality, or availability of the information. Any claims against the Department of Commerce stemming from the use of its GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.

global-workflow's Issues

Restructure ROTDIR for $COMPONENT subfolders

To facilitate a structure more conducive to UFS and merging modeling systems together (e.g. GFS + GEFS) the $CYC subfolder of COMROT/ROTDIR needs to be broken into a new $COMPONENT subfolder. This will require all components of the FV3GFS to update.

From discussions with NCO (Steven Earle):

COMOUTatmos, wave, etc = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENTatmos, $COMPONENTwave, etc

obs and all the components prdgen/post will go to:
COMOUT = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENT where COMPONENT is one of obs, wave, atmos, ocean, etc.

For GFS, NET = gfs, while RUN is either gfs, gdasenkf or gdas

Will break up $CDUMP.$PDY/$CYC into atmos, wave, etc:

• $CDUMP.$PDY/$CYC/atmos
• $CDUMP.$PDY/$CYC/wave

$CDUMP.$PDY/$CYC/wave <--move gdaswave.$PDY to this

Created branch for this work: feature/gfsv16b-restructure
This branch will be merged back into feature/gfsv16b when ready. Will document that merge in issue #1

Updates from phase1/2 retirement: syndat compath and prod modules

To do:

• update compath for syndat (develop branch)
• update prod_envir module version (develop branch)
• update compath for syndat (operations branch)
• update prod_envir module version (operations branch)
• retag NCO-v15.2.9

RFC 6652 mentions update to compath. The default compath setting was changed in operations from:

arch/prod/syndat to gfs/prod/syndat

Will update syndat paths in develop and operations branches where set.

RFC 6652 – ecFlow v4.17.0 – Upgrade the ecFlow job scheduler to version 4.17.0
and also transition this over to the Dell ecFlow nodes mecflow1/2 and vecflow1/2.
In addition, numerous monitoring jobs will be modified (/prod/admin and
/prod/utility families in ecFlow), numerous jobs will be removed (sendmail,
logrotate, sev-monitor, watch jobs, satingest, transfers to p1p2, monitors, hwm
generation, cleanup, backup job, and date-setup), and remaining phase 1/2
dependencies will be removed (update to prod-envir module, update hiresw
gempak, nhc dependencies, compath fix files, version file cleanup). Was to be
implemented on March 11 at 1100Z; postponed due to CWD; new date is March
25 at 1100Z.

From email with Russ/Hang/Jiayi:

The v153rt1 (GFS v15.3) real-time parallel experienced failures in the 2020032606 and 2020032612 gdasvrfy.   Specifically, JGFS_CYCLONE_TRACKER failed with the following message

    0.172 + compath.py arch/prod/syndat

    *************************************************************
    ** FATAL ERROR: [compath] Could not find arch/prod/syndat
    ** ABNORMAL EXIT at Thu Mar 26 13:42:54 UTC 2020 on m57c34f
    *************************************************************


The above corresponds to the following line in JGFS_CYCLONE_TRACKER

    export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}


I poked around in the operational job logs.   It looks like we should replace "arch/prod/syndat" with "gfs/prod/syndat".   I did so in the working copy of the gfs.v15.3 workflow.

JGFS_CYCLONE_TRACKER (line 101) and JGFS_CYCLONE_GENESIS (line 103) in the global-workflow operations.
reference "arch/prod/syndat".   

The same "arch/prod/syndat" references are found in line 105 of JGFS_CYCLONE_GENESIS and line 101 of JGFS_CYCLONE_TRACKER in the global-workflow develop branch.     

Version of UPP inconsistent in Externals.cfg and sorc/checkout.sh

Once again, an inconsistency in versions was introduced during the latest updates to feature/gfsv16b.

GFS v16 workflow development

This issue is being opened to track workflow development for GFS v16.

Branch for work: feature/gfsv16b

Please see prior issue comments and commits:

https://vlab.ncep.noaa.gov/redmine/issues/65358

EE2 compliance, dbn_alert issues and err_chk

DBNROOT and dbn_alert are not working on EMC parallel runs even though it works fine in operations. This update removes error messages in the real time EMC parallel runs.

Add CROW to develop branch

GFS 15.2.10[11][12] updates

Process updates for GFSv15.2.10 and GFSv15.2.11. Prep for and process updates for GFSv15.2.12.

• Process v15.2.10 changes
• Process v15.2.11 changes
• Process v15.2.12 changes
• Installed obsproc_global.v3.2.6 on supported platforms

GFSv15.2.10 changes:

SVN Log:

r113 | [email protected] | 2020-04-07 12:52:04 +0000 (Tue, 07 Apr 2020) | 1 line
Changed paths:
A /trunk/gempak/fix/nagrib.tbl
M /trunk/parm/transfer_gdas_misc.list
M /trunk/parm/transfer_gfs_misc.list
M /trunk/scripts/exgoes_nawips.sh.ecf
M /trunk/scripts/exnawips.sh.ecf

RFC 6652 - WCOSS Ecflow Upgrade to v4.17.0 and end transition off tide/gyre

Update GDA in operations branch

Update the DMPDIR paths and associated scripts to use the new WCOSS P3 GDA.

GFS v15.3.2 updates

RFC 7036 – On WCOSS, implement GFS.v15.3.2 updates to the GSI fix file and global_satinfo.txt. This change is being made to address minimization issues in the GSI and tighten quality control for the seven CrIS water vapor channels. To be implemented on July 6 at 1400Z.

@RussTreadon-NOAA @MichaelLueken-NOAA Is there a new GSI tag for this version? Thanks!

Unable to generate initial conditions

I was able to successfully build after fixing #59 but now I am unable to generate initial conditions (running on Hera).

First, the UFS_UTILS in ufs_utils.fd in the sorc/ directory does not contain the util/gdas_init directory, so I cloned this repo https://github.com/NOAA-EMC/UFS_UTILS and used the develop branch because it does, and I build that.

Then, I edit config using the default date (https://github.com/NOAA-EMC/UFS_UTILS/blob/develop/util/gdas_init/config) and just changing OUTPUT_DIR and EXTRACT_DIR

Then I run

./driver.hera.sh

And that fails when I look at log.data.hires with

ERROR: [FATAL] no permission to open HPSS archive file: /NCEPPROD/hpssprod/runhistory/rh2017/201707/20170719/com2_gfs_prod_gdas.2017071918.tar

The log.data.enkf completes successfully.

Then, I change the year to 2019 with the same result.

I have HPSS access on the EMC-nems and EMC-global projects. Do I need some other permission? Is there a particular date I should try or maybe pre-generated initial conditions I could use?

Maybe @GeorgeGayno-NOAA can help?

One-step cold start implementation

Switch feature/coupled-crow to use one-step cold start.

Mid-year bufr station additions

Per users requests that accumulated throughout the year, 25 new stations are being added to operational GFS bufr station list, along with 3 existing stations modifications. This is part of a mid-year station update for all models.

EMERGENCY: ARFC turning off assimilation of OMP data

Need to use operations branch to hand off emergency change for ops.

Created operations_OMPS branch off operations branch prior to commit for updated bufr station file.

GFSv15.2.8 & GFSv15.2.9 updates

Updates:

1. new bufr station list file (already committed to operations and develop branch, documented in issue #10.
2. new obsproc_global version 3.2.4. From email from Shelley Melchior:

On January 13, 2020 both obsproc_global and obsproc_dump_post were upgraded in operations.

NCO SPA Simon needed to make changes to alert saphir data for a NESDIS customer.

I have inherited his changes into the EMC_obsproc_global and EMC_obsproc_dump_post masters.  I have also generated new OTs for those repositories to mark the state of the code in production at those versions:
OT-obsproc_global.v3.2.4-20200113
OT-obsproc_dump_post.v3.3.1-20200113

Checked output files from production and found switch to new obsproc_global version in 12z gdas (18z gfs).

3. new obsproc_global version in ops as of 2020021118 gfs & 2020021200 gdas.

tag: OT-obsproc_global.v3.2.5-20200211

From Shelley's email on February 13th:

On February 11, 2020, NCO initiated production updates to two of our obsproc software units.  These updates were required in order to install NSST (nsst.v1.0.0) on Dell ph3.

The two software units that were impacted are:
obsproc_satingest, now at v3.9.4
obsproc_global, now at v3.2.5

Remove Theia from global-workflow

Retire Theia from global-workflow scripts and files.

Original issue on VLab: https://vlab.ncep.noaa.gov/redmine/issues/70143

GFSv15.3.1 - AWIPS data card update

Effective on about June 16, 2020, beginning with the 1200 Coordinated Universal Time (UTC) run, the National Centers for Environmental Prediction (NCEP) will upgrade the Global Forecast System. The upgrade will replace the variables for precipitation and vertical velocity for all World Meteorological Organization (WMO) headed data over NOAAPort and on the Global Telecommunications System (GTS).

https://www.weather.gov/media/notification/scn20-48gfs_noaaport.pdf

Do I need to use /scratch1/NCEPDEV/global?

I generated an experiment and see that config.base sets several directories to /scratch1/NCEPDEV/global like DMPDIR, HOMEDIR, base_svn, and base_git.

I don't have write access to global. Is that a problem? Should I just set HOMEDIR and DMPDIR to somewhere else?

Change bufr station Hilo's grid point representation

Current bufr station Hilo's model grid point is on the west of the station where terrain is mountainous and it is not a good representation for the station. After thorough testings and analyses it was found a grid point on the south of the station is a better representation for the station. The original grid point (i,j=1749,600) is therefore replaced by (i,j)=(1750,601).

Create regression test for modularized exglobal fcst script solution

The modularized exglobal fcst script is targeted for multiple supporting usage. How can we be sure adding new capability or bug fix will not break the existing functionality and still be able to generate identical result? Hence, the need to create a regression test for quality control.

The regression test should do the following to certify any change before commit/merge back to main repository by code manager:

1. Run jobs using canned data on each certified target to ensure the new package reproduce the forecast result.
2. Test configuration scope should include cold/warm/rerun.
3. Test results should be uploaded to HPSS for record keeping.
4. Previously certified executable and canned data including initial condition that will be used to run the regression test should be in version control environment.
5. Each main repository merge should have documentation on how regression test was preformed.
6. Regression test should support all platform which exglobal forecast script supported.

Cleanup tasks

Take care of some small cleanup tasks:

• Update viewer based on suggests by Rahul in PR #42
• Cleanup build scripts to remove features no longer supported
• Remove duplicate Cray files (consolidate wcoss_cray and wcoss_c)
• Update prod_util version in driver scripts?
• Combine config.prepbufr with config.prep (do we need config.prepbufr?)
• Review which driver scripts to keep/remove
• Add viewer symlink to EXPDIR via setup scripts

Can't checkout gerrit repositories on Hera

I'm following the instructions in the wiki, but can't get past the checkout.sh script on Hera. All the gerrit repositories fail to clone. Is there some permission or something that I'm missing or don't have access to? I'm not familiar with gerrit.

Refactor GFS v16 workflow to reduce gdasanalcalc run time

GFS v16 workflow ecen jobs require chgres'd gdas atmospheric forecasts from the previous cycle. This chgres operation is currently performed in current cycle gdasanalcalc. The chgres executable takes about 8 minutes to run on WCOSS_D. This chgres operation could be performed in the previous cycle, thereby allowing ecen to start 8 minutes earlier in the current cycle.

Cory made the necessary changes in global-workflow branch feature/chgresfcst. These changes require changes in GFS v16 DA. Cory made the DA changes in feature/gdaschgres, a branch in his fork of the NOAA-EMC/GSI repo.

This issue is opened to document these changes, their testing, and merger into global-workflow branch feature/gfsv16b.

Updating global workflow to use hpc-stack

The develop branch of ufs-weather-model now uses hpc-stack for all the libraries.

The following changes need to be made to the global workflow

1. The codes that are in the global workflow repository should now use hpc-stack
2. The repositories that global workflow connects to also need to make updates to hpc-stack

We should track the PRs for all the component repositories through here. (This is for the top of develop)

Checklist for global-workflow external components:

• Component uses stack: FV3/ufs-weather-model
• Component uses stack: GSI (Hera only, waiting for other platform updates)
• Component uses stack: GLDAS
• Component uses stack: UPP
• Component uses stack: UFS_UTILS
• Component uses stack: EMC_gfs_wafs

Checklist for non-component/machine-specific stack updates (e.g. module_base, machine-setup.sh):

• Hera-specific changes
• Orion-specific changes
• WCOSS-Dell-specific changes

feature/coupled-crow: ww3 grib post processing producing empty grib files

The WW3 grib2 files when using my branch from feature/coupled-crow
https://github.com/JessicaMeixner-NOAA/global-workflow/tree/feature/p5ww3post
keeps having the following warning:
warning: empty layer generated
and the resulting grib2 files are all zeros. When post processing with netcdf, the fields are not empty however. This is occurring on both Hera and orion.

GFSv15.2.7 changes

The GFS was updated in operations to v15.2.7 to update a post script and pull in new CO2 fix files:

SVN Log:
------------------------------------------------------------------------
r107 | [email protected] | 2019-12-27 00:06:50 +0000 (Fri, 27 Dec 2019) | 1 line
Changed paths:
   A /trunk/fix/fix_am/co2dat_4a/global_co2historicaldata_2018.txt
   A /trunk/fix/fix_am/co2dat_4a/global_co2historicaldata_2019.txt_proj_u
   A /trunk/fix/fix_am/co2dat_4a/global_co2historicaldata_2020.txt_proj
   A /trunk/fix/fix_am/fix_co2_proj/global_co2historicaldata_2020.txt
   A /trunk/fix/fix_am/fix_co2_update/global_co2historicaldata_2019.txt
   M /trunk/scripts/exgfs_nceppost.sh.ecf

 ARFC 2020 CO2 prediction files updates and add dbn_alert GFS gfs.txxz.special.grb2fxxx for NHC

From January 10th RFC memo:

RFC 6384 – On WCOSS, added the 2020 CO2 projection file to several model fix directories (cfs.v2.1.33, gfs.v15.2.7, gsm.v12.1.6, hiresw.v7.0.15, hmon.v2.1.2, nam.v4.1.15, and sref.v7.0.23). Fixed the NAM exec/dell.exec/nam_post0 bug that caused model profiles at station locations to write out pressure values of zero for many stations. Added a dbn_alert for the GFS gfs.txxz.special.grb2fxxx product per an NHC request. Implemented on December 23 at 1300Z.

Nothing to process for new fix files, they are already in the FIX_DIRs on supported platforms and latest HPSS tarball.

Did a rsync of ../nwprod/gfsv15.2.7 into copy of operations branch and found one text change to transfer parm file that needs to get pulled in:

diff --git a/parm/transfer_gdas_misc.list b/parm/transfer_gdas_misc.list
index d65e801e..7768f72a
--- a/parm/transfer_gdas_misc.list
+++ b/parm/transfer_gdas_misc.list
@@ -23,7 +23,7 @@
 # directory are included, so if no exclude patterns match that file, it will be
 # transferred.

-com/arch/_ENVIR_/syndat/
+com/gfs/_ENVIR_/syndat/

Diffs of that transfer list file between the recent GFS versions found the change happened in v15.2.5. Diffs performed while processing changes for v15.2.5 and v15.2.6 did not show this change. Pulling it into the operations and develop branches none-the-less.

Plan of action:

1. delete NCO-v15.2.6 tag
2. pull change to parm/transfer_gdas_misc.list into the operations branch
3. remake NCO-v15.2.6 tag off operations branch
4. if post script change is in new tag will update EMC_post tag in operations branch
5. make NCO-v15.2.7 tag off operations branch

#4 will depend on if there is a new EMC_post tag. @WenMeng-NOAA is there a new EMC_post tag for the change to exgfs_nceppost.sh.ecf in production?

Will wait to perform above actions until folks are back from holiday leave next week.

Two sources of component versions

There are currently two sources of truth for the versions of components to be checked out:

• Externals.cfg
• sorc/checkout.sh

This is a bug.

This bug is causing consistency problems for downstream users of the global-workflow. All users of global-workflow need assurance that what they are checking out is correct. But that is impossible to guarantee because there are currently two sources of truth for controlling versions of global-workflow components. Consistency requires a single source of truth because reviewers of merges into feature/gfsv16b are fallible and cannot be guaranteed to 100% of the time ensure that changes to one of the above is also made in the other. Furthermore, the official instructions for checkout out the code and building state that checkout_externals is to be used for obtaining the code, not sorc/checkout.sh. Yet, sorc/checkout.sh is retained and is being updated by developers without making the same change in Externals.cfg.

One of the following paths needs to be taken:

• Delete sorc/checkout.sh
• Delete Externals.cfg and change the instructions in README.md to use sorc/checkout.sh

update feature/gfsv16b to use parallel netcdf for I/O

Created a new branch feature/gfsv16b_paranetcdf to add the options to workflow scripts to use parallel netcdf for I/O. The application varies with model resolution and computing platform. See

1. ufs-community/ufs-weather-model#43 for final model changes
2. NOAA-EMC/fv3atm#23 discussions among the developers
3. https://docs.google.com/document/d/18vqajgOv3flbS35eNPMnYpFyNiprkJpHkVZpP--dx5o/edit for Jun Wang's tests on Dell.

Adding 6 bufr stations in Thailand to the bufr sounding output

Thailand Meteorological Department (TMD) requested adding 6 bufr stations in Thailand to the bufr sounding output.

Fix Exception Handling Issue Found in build_all.sh

Arun assigned Lin to fix the exception handling issues found in build_all.sh.

Update EMC_verif-global pointer to new GitHub repo

The EMC_verif-global repo has moved from VLab to GitHub:

https://github.com/NOAA-EMC/EMC_verif-global

The v1.5.0 tag:

https://github.com/NOAA-EMC/EMC_verif-global/tree/verif_global_v1.5.0

ecflow commit for GFS.v16 implementation

This action is for ecflow commit with GFS v16 implementation. The GFS v15 style of ecflow workflow design is changed by NCO. A new coupled ecflow workflow design will replace it going forward. EMC management requested ecflow be in version control.

Port to phase 3.5

Port global-workflow to the new WCOSS phase 3.5.

GFSv15.3.0 production updates

Created release branch for hand-off to NCO: release/gfs.v15.30.0

Branch contains the following changes:

• ProdGSI tag update (from issue #37)
• bug fix to syndat_qctropcy.fd/qctropcy.f
• change to link_fv3gfs.sh to point to updated fix file set which contains updated syndat_stmnames file
• change to checkout.sh to update ProdGSI, EMC_post_gtg, and EMC_gfs_wafs tags to latest ops versions

---

The syndat_stmnames fix file under fix_am subset is being updated to fix a storm name.

It was decided to establish a frozen copy of FIX_DIR on supported platforms for v15 fix files ("fix_nco_gfsv15"). Copy established by Fanglin Yang and copied to supported platforms by Kate Friedman.

Will update link script to point to frozen "fix_nco_gfsv15" copy on supported platforms. Will freeze and version FIX_DIR like this going forward.

Remove hard-coded vertical levels from 'enkf_chgres_recenter_nc.fd' program

The enkf_chgres_recenter_nc program has logic to vertically interpolate data. However, the program is hard-wired to output the same number of vertical levels as the input file. Remove this hard-coded feature.

Supports GEFS-CHEM. Was told to start with the version under feature/gfsv16b. Don't know if the final program will reside in this repository or under GEFS. Will work under my fork's version of feature/gfsv16b.

LIngering /contrib module issues on Jet and Hera in feature/gfsv16b

The feature/gfsv16b branch currently points to the GFS.v16.0.5 tag of the GFS. However, the fixes in the fv3gfs that solve issues with /contrib modules on Jet and Hera are on the new GFS.v16.0.6 tag. The feature/gfsv16b branch needs to be updated to point to the new GFS.v16.0.6 tag.

Port to Orion

Port global-workflow and components to Mississippi machine when available (called Orion).

Branch for work: port2orion (made off of feature/gfsv16b)

Original issue on VLab: https://vlab.ncep.noaa.gov/redmine/issues/67194

Established "glopara" space: /work/noaa/global/glopara

To do list:

• Library: test version of NCEPLIBS available
• Library: NCEPLIBS official location
• Test: connection to GitHub
• Test: connection to VLab
• Test: rstprod permissions available (WON'T BE AVAILABLE)
• Install: manage_externals manic-v1.1.8 tag (personal area)
• Build: FV3GFS (w/WW3)
• Build: EMC_post
• Build: ProdGSI
• Build: GLDAS
• Build: UFS_UTILS
• Build: obsproc (modified for Orion)
• Build: chgres_recenter
• Build: chgres_recenter_ncio
• Build: fbwndgfs
• Build: fv3nc2nemsio
• Build: gaussian_sfcanl
• Build: gfs_bufr
• Build: regrid_nemsio
• Build: supvit
• Build: syndat_getjtbul
• Build: syndat_maksynrc
• Build: syndat_qctropcy
• Build: tave
• Build: tocsbufr
• Build: vint
• Establish: "glopara" space
• Establish: FIX_DIR under "glopara"
• Establish: BASE_GIT under "glopara"
• Establish: BASE_SVN under "glopara"
• Establish: GDA under "glopara"
• Establish: obsproc_prep and obsproc_global under "glopara"
• Establish: METplus
• Establish: ens_tracker under "glopara"
• Establish: copy produtil from Hera for rocoto viewer
• Create: env/ORION.env
• Add: system paths to config.base.emc.dyn
• Add: system to machine-setup.sh
• Add: system to modulefiles/module_setup.sh[csh].inc
• Add: system to setup_expt.py
• Add: system to setup_expt_fcstonly.py
• Add: system to setup_workflow.py
• Add: system to setup_workflow_fcstonly.py
• Add: system to workflow_utils.py
• Review: Orion module files for final modules and NCEPLIBS
• Test: run setup_expt.py
• Test: run setup_expt_fcstonly.py
• Test: run setup_workflow.py
• Test: run setup_workflow_fcstonly.py
• Test: crons
• Test: ROCOTO
• Test: gdas[gfs]fcst job
• Test: gdas[gfs]post job
• Test: gdasvrfy job
• Test: gfsvrfy job
• Test: METplus jobs
• Test: gdas[gfs]arch job
• Test: complete free-forecast
• Test: gdas[gfs]prep job
• Test: gdaseobs job
• Test: gdas[gfs]anal job
• Test: gdas[gfs]analcalc job
• Test: gdasanaldiag job
• Test: gdasgldas job
• Test: gdasediag job
• Test: gdaseomg job
• Test: gdaseupd job
• Test: gdasecen job
• Test: gdasesfc job
• Test: gdasefcs job
• Test: gdasepos job
• Test: gdas[gfs]waveinit job
• Test: gdas[gfs]waveprep job
• Test: gdas[gfs]wavepostsbs job
• Test: GSI monitoring jobs
• Test: ens_tracker
• Test: C192C96 cycled
• Test: C384C192 cycled
• Test: C768C384 cycled
• Pre-commit testing: cycled and ff on Hera
• Pre-commit testing: cycled and ff on WCOSS-Cray
• Pre-commit testing: cycled and ff on WCOSS-Dell
• Supported in checkout: ufs-weather-model/fv3atm
• Supported in checkout: ProdGSI
• Supported in checkout: GLDAS
• Supported in checkout: EMC_post
• Supported in checkout: UFS_UTILS
• Supported in checkout: EMC_verif-global

change modules to production paths

Per NCO ecflow requirements, all non-prod modules must be changed to prod paths.

Changes for GFSv15.3

A feature branch created from the updated operations branch. This development branch will be the basis for GFS v15.3. And the DA tag is updated to DA_GFS_v15.3.

Add documentation to repo

Add workflow documentation within repository. Add needed graphics. Consider creating Git pages using documentation.

GFSv15.2.6 production changes and bug fix

RFC 6329 – gfs v15.2.6 - On WCOSS, fix the GFS visibility decoding bug in the GFS ncep_post module, which causes a negative impact on the visibility field where it's raining. To be implemented on December 16 at 1430Z.

• new UPP tags:
  -- for UPP with gtg: EMC_post_gtg: tag ncep_post_gtg.v1.1.10 at @aebdd1be
  -- for general UPP: EMC_post: tag ncep_post.v8.1..4 at @d81523b3

• new obsproc_global tag: OT-obsproc_global.v3.2.3-20191205

Update DA tag in operations branch for v15.2.9

Updates for v15.2.9 in operations branch didn't include the satinfo file change that was made locally in ops. That update is now in a new DA tag "fv3da_gfsv15.2.9". Will update operations branch and retag NCO-v15.2.9.

Add METplus metatasks to develop

Bring the METplus work done in feature/gfsv16b branch into develop branch. Kate will begin pulling changes into develop branch and work with Mallory to validate/test.

GFSv15.3.3 - EUM bufr changes for OPS

Release branch prepared for implementation: release/gfsv15.3.3

• Updated sorc/checkout.sh to use new DA tag for EUM bufr changes: gfsda.v15.3.3
• Changed added to release branch in PR #127 .

Implementation scheduled for October 20th, 2020 12z.

Port global-workflow to S4

The global-workflow should be ported to S4.

Branch: DavidHuber-NOAA:port2s4

Below is a checklist taken from #5 for porting to Orion which I will use as a template:

• Test: connection to GitHub
• Build: hpc-stack (personal area)
• Build: hpc-stack (official area)
• Install: manage_externals manic-v1.1.8 tag (personal area)
• Build: FV3GFS (w/WW3)
• Build: ProdGSI
• Build: GLDAS
• Build: UFS_UTILS
• Build: gfs_wafs
• Build: obsproc
• Build: chgres_recenter
• Build: chgres_recenter_ncio
• Build: fbwndgfs
• Build: fv3nc2nemsio
• Build: gaussian_sfcanl
• Build: gfs_bufr
• Build: regrid_nemsio
• Build: supvit
• Build: syndat_getjtbul
• Build: syndat_maksynrc
• Build: syndat_qctropcy
• Build: tave
• Build: tocsbufr
• Build: vint
• Test: fv3gfs_chgres.sh
• Establish: "glopara" space
• Establish: FIX_DIR under "glopara"
• Establish: BASE_GIT under "glopara"
• Establish: GDA under "glopara"
• Establish: obsproc_prep and obsproc_global under "glopara"
• Establish: Fit2Obs under "glopara"
• Establish: METplus
• Establish: ens_tracker under "glopara"
• Create: standard S4 environment
• Add: system paths to config.base.emc.dyn
• Add: system to machine-setup.sh
• Add: system to modulefiles/module_setup.sh[csh].inc
• Add: system to setup_expt.py
• Add: system to setup_expt_fcstonly.py
• Add: system to setup_workflow.py
• Add: system to setup_workflow_fcstonly.py
• Add: system to workflow_utils.py
• Review: S4 module files for final modules and hpc-stack
• Test: run setup_expt.py
• Test: run setup_expt_fcstonly.py
• Test: run setup_workflow.py
• Test: run setup_workflow_fcstonly.py
• Test: crons
• Test: ROCOTO
• Test: gdas[gfs]fcst job
• Test: gdas[gfs]post job
• Test: gdasvrfy job
• Test: gfsvrfy job
• Test: METplus jobs
• Test: gdas[gfs]arch job
• Test: complete free-forecast
• Test: gdas[gfs]prep job
• Test: gdaseobs job
• Test: gdas[gfs]anal job
• Test: gdas[gfs]analcalc job
• Test: gdasanaldiag job
• Test: gdasgldas job
• Test: gdasediag job
• Test: gdaseupd job
• Test: gdasecen job
• Test: gdasesfc job
• Test: gdasefcs job
• Test: gdasepos job
• Test: gdas[gfs]waveinit job
• Test: gdas[gfs]waveprep job
• Test: gdas[gfs]wavepostsbs job
• Test: GSI monitoring jobs
• Test: Fit2Obs
• Test: ens_tracker
• Test: C96C48 cycled
• Test: C192C96 cycled
• Test: C384C192 cycled
• Test: C96 ff
• Test: C192 ff
• Test: C384 ff
• Test: beta-testers
• Pre-commit testing: cycled and ff on Hera
• Pre-commit testing: cycled and ff on WCOSS-Cray
• Pre-commit testing: cycled and ff on WCOSS-Dell
• Perform review
• Supported in checkout: ufs-weather-model/fv3atm
• Supported in checkout: ProdGSI
• Supported in checkout: GLDAS
• Supported in checkout: EMC_post
• Supported in checkout: UFS_UTILS
• Supported in checkout: EMC_verif-global

Upgrade the universal forecast script of global-workflow for CCPP namelists, PR#32

The universal forecast script located in feature/coupled-crow branch need to be able to support CCPP namelists generation.

Convert checkout.sh to manage_externals

Original issue on VLab: https://vlab.ncep.noaa.gov/redmine/issues/66742

Replace sorc/checkout.sh with manage_externals:

https://github.com/ESMCI/manage_externals

Gave demo of manage_externals and users seemed interested. Should resolve issues with lack of flexibility in checkout.sh.

• Establish initial Externals.cfg
• Install v1.1.7 manage_externals tag on WCOSS-Dell
• Install v1.1.7 manage_externals tag on WCOSS-Cray
• Install v1.1.7 manage_externals tag on Hera
• Install v1.1.8 manage_externals tag on WCOSS-Dell
• Install v1.1.8 manage_externals tag on WCOSS-Cray
• Install v1.1.8 manage_externals tag on Hera
• Create manage_externals module on WCOSS-Dell (replaces prior install)
• Create manage_externals module on WCOSS-Cray (replaces prior install)
• Create manage_externals module on Hera (replaces prior install)
• Add note about WCOSS-Cray ssh.variant change to wiki post commit
• Complete commit

FV3 empty repository on gerrit

I'm back to working with this and following along with along with the instructions. My initial gerrit issue is fixed, but the FV3 repository is empty?

Cloning into '/scratch1/NCEPDEV/nems/Kyle.Gerheiser/new_global_workflow/global-workflow/sorc/fv3gfs.fd/FV3'... warning: You appear to have cloned an empty repository.

Fetched in submodule path 'FV3', but it did not contain f4f8644fe73f10513bd318b01832c448f225c007. Direct fetching of that commit failed.

Verified by checking the repository on gerrit here:

https://vlab.ncep.noaa.gov/code-review/admin/repos/FV3,tags

With some work I found the commit on the public repo here and used that https://github.com/NOAA-EMC/fv3atm/tree/f4f8644fe73f10513bd318b01832c448f225c007

need to reduce post job # in CROW workflow for ufs-s2s-model

right now fv3 and ocean post jobs will be launched at every 6hr interval, this will bring ~300 jobs which is hard to manage. It will be good to reduce the total job numbers, for example, just split the 35 days post jobs into 5 sections.

Remove data paths from lower level scripts

Look through lower level scripts (e.g. ush) and remove path settings that should be in upper level scripts (e.g. JJOBs).

rocoto_viewer.py on WCOSS not working now without Gyre

CRITICAL ERROR: The produtil package could not be loaded from your system

Lines 194-199 of rocoto_viewer.py:

    PRODUTIL = collections.defaultdict(list)
    PRODUTIL['hera'] = '/scratch1/NCEPDEV/global/glopara/svn/nceplibs/produtil/trunk/ush'
    PRODUTIL['luna']  = '/gpfs/hps3/emc/global/noscrub/emc.glopara/svn/nceplibs/produtil/trunk/ush'
    PRODUTIL['tide']  = '/gpfs/td1/emc/global/save/emc.glopara/svn/nceplibs/produtil/trunk/ush'
    PRODUTIL['gyre']  = '/gpfs/gd1/emc/global/save/emc.glopara/svn/nceplibs/produtil/trunk/ush'
    try_clusters = ('hera','luna','tide','gyre')