Port global-workflow to S4

[noaa-emc]

Port global-workflow to S4

[DavidHuber-NOAA]

Linked issues:

Port the UFS: ufs-community/ufs-weather-model#738
Port the GSI: NOAA-EMC/GSI#177
Port GLDAS: NOAA-EMC/GLDAS#21
Port the Post: NOAA-EMC/UPP#341
Port WW3: NOAA-EMC/WW3#424
Port UFS_Utils: ufs-community/UFS_UTILS#543
Port verif-global: NOAA-EMC/EMC_verif-global#55, NOAA-EMC/EMC_verif-global#65

[DavidHuber-NOAA]

I have submitted pull requests for the following repositories
GSI (177, approved and closed, revision cca6d5562)
GLDAS (23, open)
Post (362, approved and closed, revision 27873888)
UFS_Utils (543, approved and closed, revision 6b1d6216)

The port for the UFS is complete. Pulling in the develop branch in preparation for the pull request caused the build to fail. This may be related to ufs-community/ufs-weather-model#744. I will be asking about this today.

The WW3 port is on hold and may be handled after the initial global workflow port since the atm-wav configuration does not work in the UFS at the moment. It does build on S4, though it cannot be tested.

[DavidHuber-NOAA]

The verif-global port has been completed. The tag including this port is verif_global_v2.2.1.

[JessicaMeixner-NOAA]

@DavidHuber-NOAA I want to clarify that WW3 works in UFS, it just does not work in the global-workflow at this time.

[DavidHuber-NOAA]

@JessicaMeixner-NOAA OK, thanks for the clarification. I will look into testing it outside of the global workflow and finish the port via that route.

[arunchawla-NOAA]

@DavidHuber-NOAA, @KateFriedman-NOAA, @WalterKolczynski-NOAA, @lgannoaa @aerorahul

Is this task complete? Is it waiting for some other task to be completed ? Moving forward how will S4 capability keep up to date with changes that come to the global workflow ?

[DavidHuber-NOAA]

@arunchawla-NOAA The work is nearly complete. All of the subcomponents have been ported to S4 and I am waiting for those hashes to be incorporated into the global-workflow. See this note for details on the hashes. Below are the repositories and their respective hashes/tags that I am waiting on:

UFS: f04a30ce4388eb7f003524d87745cbae0b42fd50
GSI: 99346e4c5dbb6487b7595e8b600b1e0ed5cd4194
UPP: 278738880978fac65362d2696328d563ec5b99ad
GLDAS: gldas_gfsv16_release.v1.18

When these hashes/tags are incorporated, I will finalize and test the port.

Keeping the S4 port of the global workflow and its subcomponents up to date will be my responsibility. Once the S4 port is incorporated, my plan for maintaining the build involves

1. Monitoring the current issues and preparing updates as necessary to keep the S4 port up to date
2. Performing nightly builds of the global workflow and subcomponents, updating and installing software as necessary to keep the builds maintained
3. Performing monthly 2-cycle test runs to verify that the workflow is running correctly on S4
4. Performing quarterly, 160-cycle benchmarks at 384/192 and 192/96
5. Regression testing of the UFS on a monthly basis

Additionally, if I could be notified that an upcoming change will require new installations of libraries or external software (e.g. the Tracker, MET/METplus, ObsProc) or modifications to module files, etc, that would be appreciated.

[DavidHuber-NOAA]

@KateFriedman-NOAA @WalterKolczynski-NOAA
I'm back (woohoo) and working on getting caught up on the global-workflow development that I've missed and need to apply to the S4 port. I can see a few things that need updating, with the most major being the translation of the modulefiles to Lua. There were also some updates to the machine_setup.sh and build_ww3prepost.sh scripts that require some minor changes. Are there any other significant changes over the past 8 weeks that I should be aware of?

I also read that the current develop branch of the global workflow does not run cycled tests. Is that correct?

[WalterKolczynski-NOAA]

Welcome back!

Yes, cycled mode is borked for the time being. It is unfortunately out of our hands right now.

I think the module change is the big one. Also make sure WW3 pre/post build still works after the cmake change, and that the new model version works (other than it breaking cycling). Nothing else should need special attention for S4.

Another impactful change is in the pipeline (see PR #693). I've actually made a custom hpc-stack on the three tier-1 machines as a temporary solution while we wait for hpc-stack to catch up.

[KateFriedman-NOAA]

Welcome back @DavidHuber-NOAA !

[DavidHuber-NOAA]

@WalterKolczynski-NOAA S4 must create the nsstbufr and prepbufr.acft_profiles files during the prep step (since the operational versions of these files are restricted). Currently, the nsstbufr file is created only if DO_MAKE_NSSTBUFR=YES in config.prep. The prepbufr.acft_profiles is always created during the prep step. However, the JGLOBAL_ATMOS_ANALYSIS job is currently set up to link to the GDA versions of these files, regardless of whether DO_MAKE_NSSTBUFR=YES. Since, the analysis jobs do not reference config.prep, I would like to do the following:

1. add DO_MAKE_ACFTBUFR to config.base.emc.dyn
2. move DO_MAKE_NSSTBUFR there as well
3. wrap both options with an if $machine == S4 block around it, then
4. point NSSTBF and PREPQCPF in JGLOBAL_ATMOS_ANALYSIS to COMOUT instead of COMIN_OBS if DO_MAKE*=YES.

Do you think this is a reasonable solution? And do you think I need another issue open to implement this or can I make it part of this port?

For reference, NSSTBF and PREPQCPF are defined here.