Updating global workflow to use hpc-stack

[noaa-emc]

Updating global workflow to use hpc-stack

[JessicaMeixner-NOAA]

@arunchawla-NOAA for the wave_prep for GFSv16, we need CDO, do we request this to be added to hpc-stack? For feature/coupled-crow, the ocean/ice post uses ncl. Should this also be requested to be added to hpc-stack?

[aerorahul]

@JessicaMeixner-NOAA cdo is a very useful tool and we should add it to hpc-stack. It depends on how many bells and whistles we need to add or use. For e.g. grib handling is done via grib_api from eccodes.

[JessicaMeixner-NOAA]

Made an hpc-stack issue for CDO: NOAA-EMC/hpc-stack#82

But I guess the question remains, do all modules used in global-workflow need to be added to hpc-stack?

[arunchawla-NOAA]

Yes, everything should use hpc-stack. We want this to be our way to make the environment equivalent across different platforms. CDO is available as third party libraries on WCOSS. We should add it to hpc-stack.

[JessicaMeixner-NOAA]

@HuiyaChuang-NOAA @WenMeng-NOAA is there a version of EMC_post that uses hpc-stack?

[WenMeng-NOAA]

@JessicaMeixner-NOAA Not yet. I have been working with NCEPLIBS team for testing UPP with hpc-stack. From the last update, I have been waiting for CRTM_FIX defined in the crtm module at hpc-stack.

[RussTreadon-NOAA]

The GSI is waiting for the same addition to hpc-stack. Non-stack crtm modules define CRTM_FIX. CRTM_FIX is the path to the directory containing the coefficient files required to run the CRTM.

…

On Mon, Dec 14, 2020 at 2:49 PM WenMeng-NOAA ***@***.***> wrote: @JessicaMeixner-NOAA <https://github.com/JessicaMeixner-NOAA> Not yet. I have been working with NCEPLIBS team for testing UPP with hpc-stack. From the last update, I have been waiting for CRTM_FIX defined in the crtm module at hpc-stack. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN633OVXW77KBX6OZPYE3SUZT5XANCNFSM4SZ3J73A> .

[aerorahul]

@WenMeng-NOAA @RussTreadon-NOAA
It is unlikely hpc-stack will add a CRTM_FIX in the modulefile for CRTM because the fix files are not part of the EMC_CRTM repository.

You should define this variable in your application modulefile for the appropriate machine.
E.g. for UPP in lib-orion, you should add a line

setenv CRTM_FIX /apps/contrib/NCEPLIBS/orion/fix/crtm_v2.3.0

This issue will be resolved when we move to the authoritative CRTM repository that provides the build and the CRTM coefficients and can be built and installed simultaneously in one seamless step.

[WenMeng-NOAA]

@aerorahul I can add additional line for CRTM_FIX path in the UPP standalone tests. Can NCEPLIBS teams list all CRTM_FIX location on WCOSS Dell, Cray, Hera, Jet and Orion on wiki or some places?

[RussTreadon-NOAA]

OK. Defining CRTM_FIX in DA modulefiles is easy. One detail - DA modulefiles load more modules to build than are required to run but this should not be a problem at run time apart from loading more modules than is necessary. The current EMC global workflow does not use app modulefules. Instead, it loads a single machine dependent modulefile for all apps. CRTM_FIX needs to be defined in global workflow machine specific modulefiles or the workflow needs to be updated to load app specific modulefiles. NCO ecflow scripts only load the modules required by the given job.

…

On Mon, Dec 14, 2020 at 2:58 PM Rahul Mahajan ***@***.***> wrote: @WenMeng-NOAA <https://github.com/WenMeng-NOAA> @RussTreadon-NOAA <https://github.com/RussTreadon-NOAA> It is unlikely hpc-stack will add a CRTM_FIX in the modulefile for CRTM because the fix files are not part of the EMC_CRTM repository. You should define this variable in your application modulefile for the appropriate machine. E.g. for UPP in lib-orion <https://github.com/NOAA-EMC/EMC_post/blob/develop/modulefiles/upp/lib-orion>, you should add a line setenv CRTM_FIX /apps/contrib/NCEPLIBS/orion/fix/crtm_v2.3.0 This issue will be resolved when we move to the authoritative CRTM repository that provides the build and the CRTM coefficients and can be built and installed simultaneously in one seamless step. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN633SSW5OCAJHGJGDP6LSUZU4RANCNFSM4SZ3J73A> .

[aerorahul]

Sure. The variable will/should be added to each of the machine specific modulefile under modulefiles/
The appropriate path for CRTM_FIX can be found by logging in to each of those machines, and doing the following. E.g. for emc_post:

git clone https://github.com/noaa-emc/emc_post
cd emc_post
module use $PWD/modulefiles
module load upp/lib-<machine>
echo $CRTM_FIX

[aerorahul]

@RussTreadon-NOAA
You are right; runtime modulefiles differ than build time.
This needs to be repeated (as necessary) in modulefiles in the appropriate application; gsi, upp, global-workflow, etc.

[RussTreadon-NOAA]

Machine specific modulefiles in modulefiles/upp include crtm loads. These indicated crtm modules define CRTM_FIX

…

On Mon, Dec 14, 2020 at 3:07 PM WenMeng-NOAA ***@***.***> wrote: @aerorahul <https://github.com/aerorahul> I can add additional line for CRTM_FIX path in the UPP standalone tests. Can NCEPLIBS teams list all CRTM_FIX location on WCOSS Dell, Cray, Hera, Jet and Orion on wiki or some places? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN634NS5ERIHG4Q5Z5Q43SUZWADANCNFSM4SZ3J73A> .

[WenMeng-NOAA]

@aerorahul The CRTM_FIX is used in UPP runtime configuration. I mean the path of CRTM_FIX on different platforms for the hpc-stack like:
Orion: /apps/contrib/NCEPLIBS/orion/fix/crtm_v2.3.0
Hera: /scratch1/NCEPDEV/...
Dell: /usrx//local/...

[aerorahul]

@WenMeng-NOAA I understand that CRTM_FIX is used at runtime. They are coefficients.
CRTM_FIX is not part of hpc-stack and will not be until we move to an authoritative repo, or we can host the coefficients and access them via curl, wget, git-lfs, etc.

The v2.3.0 fix files will not change and neither will their existing location.

I can find that info for you, but you can help me by executing the statements from here

[WenMeng-NOAA]

@aerorahul Thanks for clarifying. In that case, I will keep existing CRTM_FIX defined by crtm module not installed at the hpc-stack for the UPP runtime config. until transitioning to authoritative CRTM repository.

[aerorahul]

@WenMeng-NOAA Yes. That is exactly what I was suggesting so that you (and others) can continue the build and execution of your components as they transition to hpc-stack.

When CRTM_FIX is available in hpc-stack, we can go in and remove this one line from the modulefiles.

[KateFriedman-NOAA]

Created branch for global-workflow hpc-stack updates: feature/hpc-stack

[WenMeng-NOAA]

@JessicaMeixner-NOAA

is there a version of EMC_post that uses hpc-stack?

The UPP develop branch was just updated @b267dd9 with the capacity of building standalone UPP via cmake and dependency libraries from the hpc-stack. You may check out the latest UPP or the tag upp_v10.0.3. The build instruction can be found at
https://github.com/NOAA-EMC/EMC_post/wiki/Building-UPP#build-standalone-upp-via-cmake

[KateFriedman-NOAA]

@junwang-noaa @HelinWei-NOAA @RussTreadon-NOAA @MichaelLueken-NOAA @YaliMao-NOAA Hi all, could I get any updates you have on your respective repositories progress on converting to use the new hpc-stack for building? I'm ready to integrate new component develop tags that build with hpc-stack into global-workflow when you have them ready. Let me know if you need assistance starting this change. Thanks!

Thus far I know the following component develop branches use hpc-stack but I need new tags off those develop branches for some of them still:

1. UPP (have new tag, all good and build tested, thanks @WenMeng-NOAA !)
2. ufs-weather-model (see develop uses stack, need new tag off develop when available)
3. UFS_UTILS (see develop uses stack, need new tag off develop, likely after other work with @GeorgeGayno-NOAA completes)

Thanks!

[HelinWei-NOAA]

Hi kate, I checked all module files used by GLDAS. hpc is not found to be used by any of them. So I don't need to update anything. Right? Helin

…

On Mon, Jan 25, 2021 at 3:21 PM Kate Friedman ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> @HelinWei-NOAA <https://github.com/HelinWei-NOAA> @RussTreadon-NOAA <https://github.com/RussTreadon-NOAA> @MichaelLueken-NOAA <https://github.com/MichaelLueken-NOAA> @YaliMao-NOAA <https://github.com/YaliMao-NOAA> Hi all, could I get any updates you have on your respective repositories progress on converting to use the new hpc-stack for building? I'm ready to integrate new component develop tags that build with hpc-stack into global-workflow when you have them ready. Let me know if you need assistance starting this change. Thanks! Thus far I know the following component develop branches use hpc-stack but I need new tags off those develop branches for some of them still: 1. UPP (have new tag, all good and build tested, thanks @WenMeng-NOAA <https://github.com/WenMeng-NOAA> !) 2. ufs-weather-model (see develop uses stack, need new tag off develop when available) 3. UFS_UTILS (see develop uses stack, need new tag off develop, likely after other work with @GeorgeGayno-NOAA <https://github.com/GeorgeGayno-NOAA> completes) Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHKYH6FPURJNB6ICYVYLDS3XHFZANCNFSM4SZ3J73A> .

[RussTreadon-NOAA]

NOAA-EMC/GSI PR #107 <NOAA-EMC/GSI#107> contains changes required for DA apps to compile on Hera using hpc-stack.

…

On Mon, Jan 25, 2021 at 3:21 PM Kate Friedman ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> @HelinWei-NOAA <https://github.com/HelinWei-NOAA> @RussTreadon-NOAA <https://github.com/RussTreadon-NOAA> @MichaelLueken-NOAA <https://github.com/MichaelLueken-NOAA> @YaliMao-NOAA <https://github.com/YaliMao-NOAA> Hi all, could I get any updates you have on your respective repositories progress on converting to use the new hpc-stack for building? I'm ready to integrate new component develop tags that build with hpc-stack into global-workflow when you have them ready. Let me know if you need assistance starting this change. Thanks! Thus far I know the following component develop branches use hpc-stack but I need new tags off those develop branches for some of them still: 1. UPP (have new tag, all good and build tested, thanks @WenMeng-NOAA <https://github.com/WenMeng-NOAA> !) 2. ufs-weather-model (see develop uses stack, need new tag off develop when available) 3. UFS_UTILS (see develop uses stack, need new tag off develop, likely after other work with @GeorgeGayno-NOAA <https://github.com/GeorgeGayno-NOAA> completes) Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN63Y7XLLBWTVYIXKHDZTS3XHFZANCNFSM4SZ3J73A> .

[junwang-noaa]

Kate, May I ask what is this tag used for? Do we have some name for all the components? What is the current model tag used in workflow?

…

On Mon, Jan 25, 2021 at 3:21 PM Kate Friedman ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> @HelinWei-NOAA <https://github.com/HelinWei-NOAA> @RussTreadon-NOAA <https://github.com/RussTreadon-NOAA> @MichaelLueken-NOAA <https://github.com/MichaelLueken-NOAA> @YaliMao-NOAA <https://github.com/YaliMao-NOAA> Hi all, could I get any updates you have on your respective repositories progress on converting to use the new hpc-stack for building? I'm ready to integrate new component develop tags that build with hpc-stack into global-workflow when you have them ready. Let me know if you need assistance starting this change. Thanks! Thus far I know the following component develop branches use hpc-stack but I need new tags off those develop branches for some of them still: 1. UPP (have new tag, all good and build tested, thanks @WenMeng-NOAA <https://github.com/WenMeng-NOAA> !) 2. ufs-weather-model (see develop uses stack, need new tag off develop when available) 3. UFS_UTILS (see develop uses stack, need new tag off develop, likely after other work with @GeorgeGayno-NOAA <https://github.com/GeorgeGayno-NOAA> completes) Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TJAFV7CTEPTOG2USO3S3XHFZANCNFSM4SZ3J73A> .

[KateFriedman-NOAA]

Hi kate, I checked all module files used by GLDAS. hpc is not found to be used by any of them. So I don't need to update anything. Right? Helin

@HelinWei-NOAA Since the GLDAS modulefiles/build don't yet use the hpc-stack library set they need to be updated to do so. I'll send you a separate email with more details. Thanks!

NOAA-EMC/GSI PR #107 <NOAA-EMC/GSI#107> contains changes required for DA apps to compile on Hera using hpc-stack.

Thanks @RussTreadon-NOAA ! I'll take a look at that incoming version. I'm updating things for Hera first so that PR is great.

Kate, May I ask what is this tag used for? Do we have some name for all the components? What is the current model tag used in workflow?

@junwang-noaa The global-workflow develop branch is about to be using the GFS.v16.0.15 tag (once I commit the final v16 updates from NCO) but we're also supporting the ufs-weather-model develop branch at ufs-community/ufs-weather-model@b771e5b when users want to run with CCPP. Since ufs-weather-model develop now supports hpc-stack I'll move to use only that branch (move past the GFS.v16.0.15 tag) but I'll need to peg it to either a tag or a commit hash to avoid the forecast model moving from underneath folks as development continues. I prefer to stick with tags that go with certain feature releases but since we're early in GFSv17 development I can go with a hash of the develop branch for now. So let me know if you have any planned tags coming up and if not I'll go with the recent hash of develop. Thanks!

[arunchawla-NOAA]

@KateFriedman-NOAA and @WalterKolczynski-NOAA I know we still have this separation between global develop and coupled branches, but since UPP uses hpc-stack can we make sure of that in the coupled branch and test it on HERA. I ask because we need to use hpc-stack for the coupled branch for porting to different platforms

[KateFriedman-NOAA]

Helin updated GLDAS to build with hpc-stack and provided a new tag gldas_gfsv16_release.v1.14.0. Tested build on Hera (no issues). Will run in cycled test once able.

[junwang-noaa]

Kate, you can use the latest model revision 06bc77b <ufs-community/ufs-weather-model@06bc77b>. In ufs-weather-model repository, we only keep implementation/release/benchmark tags. The coupled model benchmark 6 tag will come up soon, I will let you know when it becomes available.Thanks

…

On Wed, Jan 27, 2021 at 9:27 AM Kate Friedman ***@***.***> wrote: Helin updated GLDAS to build with hpc-stack and provided a new tag gldas_gfsv16_release.v1.14.0 <https://github.com/NOAA-EMC/GLDAS/releases/tag/gldas_gfsv16_release.v1.14.0>. Tested build on Hera (no issues). Will run in cycled test once able. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TK65G24UNXNYHMKTXLS4APDPANCNFSM4SZ3J73A> .

[KateFriedman-NOAA]

Kate, you can use the latest model revision 06bc77b...coupled model benchmark 6 tag
will come up soon, I will let you know when it becomes available.Thanks

Gotcha, sounds good, thanks @junwang-noaa !

[aerorahul]

Workflow utilities under sorc/ can now built with hpc-stack and cmake.
Both those features are captured in: feature/hpc-stack-cmake
Modulefiles for orion, hera and wcoss_dell_p3 are added.
These workflow utilities were compiled on hera and orion, but not on wcoss_dell_p3.
These workflow utilities were compiled on macOS, EMC Linux box (RHEL7), Docker with GNU and IntelOneAPI containers.
All utilities built as expected. They were not tested against the hand-written makefiles on the supported HPC's for reproducibility.

[aerorahul]

The utility enkf_chgres_recenter_nc.fd depends on the netCDF IO library in the GSI specifically written to handle UFS netCDF output. This library needs to be made available outside of the GSI for other applications e.g. GEFS to be made useful.

A NCEPlibs issue is opened at NOAA-EMC/NCEPLIBS#184 for discussion.

[JessicaMeixner-NOAA]

Workflow utilities under sorc/ can now built with hpc-stack and cmake.
Both those features are captured in: feature/hpc-stack-cmake
Modulefiles for orion, hera and wcoss_dell_p3 are added.
These workflow utilities were compiled on hera and orion, but not on wcoss_dell_p3.
These workflow utilities were compiled on macOS, EMC Linux box (RHEL7), Docker with GNU and IntelOneAPI containers.
All utilities built as expected. They were not tested against the hand-written makefiles on the supported HPC's for reproducibility.

A script in sorc/ needs to be added for it's pre-and post-processors to be built: #256 and then there are two issues: 1. related to using hpc-stack: the ww3 grib which has an hpc-stack issue: NOAA-EMC/hpc-stack#137 and then 2. WW3 isn't using CMAKE for it's pre/post processor exes

[aerorahul]

@JessicaMeixner-NOAA
This feature branch does not build utilities that are checked out as part of checkout.sh. I leave that to the repository owner to make the upgrade. WW3 fits that bill.
This feature branch only builds utilities that are checked out as part of the global-workflow repository.

[KateFriedman-NOAA]

@GeorgeGayno-NOAA Any updates on updating UFS_UTILS to use hpc/1.1.0 stack? ufs-community/UFS_UTILS#248 Thanks!

[KateFriedman-NOAA]

@RussTreadon-NOAA @MichaelLueken-NOAA Looks like NOAA-EMC/GSI#107 has gone in. Is GSI ready for global-workflow to try the updated master with hpc-stack? Should we wait for any further updates or tags? Thanks!

[RussTreadon-NOAA]

NOAA-EMC/GSI#107 <NOAA-EMC/GSI#107> adds hpc-stack to Hera alone. No DA tag has been created since hpc-stack remains to be added to NOAA-EMC/GSI for other platforms. Workflow testing using DA with hpc-stack on Hera is possible.

…

On Tue, Feb 9, 2021 at 12:48 PM Kate Friedman ***@***.***> wrote: @RussTreadon-NOAA <https://github.com/RussTreadon-NOAA> @MichaelLueken-NOAA <https://github.com/MichaelLueken-NOAA> Looks like NOAA-EMC/GSI#107 <NOAA-EMC/GSI#107> has gone in. Is GSI ready for global-workflow to try the updated master with hpc-stack? Should we wait for any further updates or tags? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN63YASJECJAGH43ASJR3S6FYNXANCNFSM4SZ3J73A> .

[arunchawla-NOAA]

If it is working on HERA can we try it with another system ? Perhaps WCOSS? It should be fairly straight forward once it works on one system

[RussTreadon-NOAA]

NOAA-EMC/GSI issue #110 <NOAA-EMC/GSI#110> has been opened to document the addition and testing of hpc-stack to DA on other machines. Tests on WCOSS_D found non-reproducible results. A lengthy investigation into this is documented in hpc-stack issue #149 <NOAA-EMC/hpc-stack#149>.

…

On Tue, Feb 9, 2021 at 1:02 PM arun chawla ***@***.***> wrote: If it is working on HERA can we try it with another system ? Perhaps WCOSS? It should be fairly straight forward once it works on one system — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN635LSWXG7OUZ6TQRHFLS6F2CJANCNFSM4SZ3J73A> .

[KateFriedman-NOAA]

@RussTreadon-NOAA Thanks for the update. I will test it on Hera first when I test the full system with the stack.

[KateFriedman-NOAA]

Sync merged develop into feature/hpc-stack @ 2cce21a. Retesting on Hera and running first tests of branch on Orion/WCOSS-Dell with Russ's GSI branch.

Components as 2/22/21:

• FV3: ufs-community/ufs-weather-model@06bc77b
• GSI: Hera @ GSI master ; Orion/WCOSS-Dell @ https://github.com/RussTreadon-NOAA/GSI/tree/feature/hpc-stack
• GLDAS: https://github.com/NOAA-EMC/GLDAS/releases/tag/gldas_gfsv16_release.v1.14.0
• UFS_UTILS: https://github.com/GeorgeGayno-NOAA/UFS_UTILS/tree/feature/stack
• EMC_post/UPP: https://github.com/NOAA-EMC/EMC_post/releases/tag/upp_v10.0.3
• WAFS: no hpc-stack support yet
• EMC_verif-global: N/A (verif_global_v1.13.4)

Current testing:

Hera -

• feature/hpc-stack branch builds with above components in checkout.sh
• rerunning previously successfully completed cycled test with synced updates (no longer using hack for workflow util execs from feature/cmake branch)

Orion -

• making Orion stack updates and running cycled test
• discovered build error with GLDAS, made local fix to build component, opened NOAA-EMC/GLDAS#20

WCOSS-Dell -

• making WCOSS-Dell stack updates and prepping a cycled test
• workflow utils build failed = discovered jasper/2.0.25 isn't in hpc/1.1.0 on Venus yet after today's prod switch @kgerheiser @Hang-Lei-NOAA Is the jasper/2.0.25 library being added to hpc/1.1.0 on Venus? Thanks! (ISSUE RESOLVED - was cache issue)
• all other components built without error

ICs for all tests are C192C96L127 for 2020090118. Running with waves off.

[KateFriedman-NOAA]

Upon closer inspection of "successful" run of feature/hpc-stack on Hera I found the prep jobs utterly failed but still completed "successfully" (major silent failure) and the rest of the cycle ran "successfully" without prepbufr from prep. There are stack-related issues to resolve and silent errors to fix. Need to get obsproc codes converted to hpc-stack; have meeting with Shelley Melchior tomorrow to go over hpc-stack with her.

Also:

• Have reached out to Jiayi Peng about converting ens_tracker to hpc-stack. Will provide guidance to him.
• Yali is now working to convert WAFS to hpc-stack.

[KateFriedman-NOAA]

Met with @ShelleyMelchior-NOAA to give introduction to hpc-stack for OBSPROC. Then converted a copy of obsproc_prep_RB-5.4.0 to build with hpc-stack. Also updated a copy of obsproc_global_RB-3.4.0 with needed stack-related updates from prod_util subfolder (ush -> bin); no codes to build in obsproc_global_RB-3.4.0. Successfully built obsproc-prep codes on WCOSS-Dell:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/obsproc/obsproc_prep_RB-5.4.0
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/obsproc/obsproc_global_RB-3.4.0

Ran gdasprep and gfsprep in two tests:

1. stack test with C192C96L127 ICs from 2020090118 (technical test)
2. used NCO 30-day parallel ICs and ran just prep jobs (reproducibility test)

Both tests ran successfully and test 2 reproduced NCO parallel prepbufr files! Documented test in OBSPROC redmine task:

https://vlab.ncep.noaa.gov/redmine/issues/88618

Will now bring stack'd obsproc copies to Hera and Orion to further update and test. Will continue to document in redmine issue as well.

[arunchawla-NOAA]

@WalterKolczynski-NOAA is the coupled crow branch running with hpc-stack?

[WalterKolczynski-NOAA]

@WalterKolczynski-NOAA is the coupled crow branch running with hpc-stack?

The two programs that are currently being built for the coupled model and located in the g-w repo are (ww3prepost and reg2grb2). The model also is using stack. The post tag still needs to be updated to one using stack, and none of the programs that aren't currently being build for coupled have been updated.

[KateFriedman-NOAA]

Changes were required to two scripts that the GFS uses in EMC_post. Contacted Wen to get changes in related to the setting of POSTGRB2TBL: ${G2TMPL_SRC} -> ${g2tmpl_ROOT}/share. She folded the updates into a PR and they are now in EMC_post develop at NOAA-EMC/UPP@33eb60e. Will test with that hash now.

This hash does not build with GTG code however, have opened issue for that with EMC_post:

NOAA-EMC/UPP#272

Will run with non-GTG UPP for now. Will need GTG UPP with cmake/hpc-stack down the road for implementation.

[KateFriedman-NOAA]

OBSPROC global and prep packages that were converted to build with hpc-stack have been installed under emc.glopara[glopara] space on WCOSS-Dell, Hera, Orion, and Jet. @ShelleyMelchior-NOAA Copy for obsproc with cleaned up build.sh that supports those platforms is here on Venus:

/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/obsproc/obsproc_prep_RB-5.4.0_hpc-stack
/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/obsproc/obsproc_global_RB-3.4.0_hpc-stack

Installed in $BASE_GIT/obsproc locations under emc.glopara[glopara] on Hera, Orion, and Jet.

Have successfully run above copies on WCOSS-Dell, Hera, and Orion thus far.

[KateFriedman-NOAA]

Testing update:

1. Clone updates: refreshed copy of UFS_UTILS feature/stack in clone of global-workflow feature/hpc-stack (git pull in sorc/ufs_utils.fd) on WCOSS-Dell, Hera, and Orion. Reran build of UFS_UTILS in each (no issues). Also recloned the GSI in global-workflow clone on all three platforms (now using GSI master everywhere); reran build of GSI in each (also no issues). Running another full cycle on each platform with updated versions (including sync merge changes with global-workflow develop that were committed at bea02f8).

2. Did initial test of WAFS jobs in global-workflow feature/hpc-stack on WCOSS-Dell with WAFS branch provided by Yali. Ran into a few errors and will try again with 1) a more recent date for data availability and 2) when Yali finishes work on a cmake'd version of WAFS (that includes hpc-stack updates). Will test new version when Yali provides.

[KateFriedman-NOAA]

@JessicaMeixner-NOAA opened NOAA-EMC/hpc-stack#193 for building the WW3 execs with hpc-stack. Will need this issue resolved so we can build the WW3 execs in global-workflow with a new build script, they are now separate from the FV3 model build.

[KateFriedman-NOAA]

Email update from @WenMeng-NOAA on UPP version with hpc-stack support:

The UPP branch develop was committed at ff42e02 with PR #294 for solving UPP issue #272 (update cmake build with GTG code). You may use that harsh to update UPP version in global-workflow. Please let me know if you see issues.

Will update feature/hpc-stack branch and retest on supported platforms.

[WenMeng-NOAA]

@KateFriedman-NOAA and @junwang-noaa
I see the branch feature/hps-stack check out ufs-weather-model as:

if [[ ! -d fv3gfs.fd ]] ; then
rm -f ${topdir}/checkout-fv3gfs.log
git clone https://github.com/ufs-community/ufs-weather-model fv3gfs.fd >> ${topdir}/checkout-fv3gfs.log 2>&1
cd fv3gfs.fd
git checkout 06bc77b
git submodule update --init --recursive
cd ${topdir}
else

I would like to know if the harsh version is the commit from the develop branch or other branches at ufs-weather-model? Thanks!

[junwang-noaa]

@WenMeng-NOAA The model hash 06bc77b is from ufs-weather-model develop branch, committed on 1/25/2021.

[KateFriedman-NOAA]

From @GeorgeGayno-NOAA in ufs-community/UFS_UTILS#248 (comment)

UFS_UTILS was updated to use the official stack. Please point to this release:
https://github.com/NOAA-EMC/UFS_UTILS/releases/tag/ufs_utils_1_3_0

Updated UPP, WAFS, and UFS_UTILS versions in clone on WCOSS-Dell and committed to feature/hpc-stack @ 6ce8336 and 858f216. Testing now in a full cycle.

[KateFriedman-NOAA]

@WenMeng-NOAA @YaliMao-NOAA Please see the following post and wafs output using EMC_post @ ff42e02 and EMC_gfs_wafs at c2a29a6:

output: /gpfs/dell3/ptmp/Kate.Friedman/comrot/stacktest6/gfs.20210428/12/atmos
logs: /gpfs/dell3/ptmp/Kate.Friedman/comrot/stacktest6/logs/2021042812

Thanks!

[WenMeng-NOAA]

@KateFriedman-NOAA I don't see any issues from the UPP part and would wait for Yali's confirming on wafs dataset.

[YaliMao-NOAA]

@KateFriedman-NOAA @WenMeng-NOAA
UPP WAFS looks good now. UPP downstream WAFS products all look good except wafsgrib2, as shown in the log file gfswafsgrib2.log

wafsgrib2 still has issue when using wgrib2 to convert grib2 file with option "-set_grib_type same", as reported NOAA-EMC/hpc-stack#224. In gfswafsgrib2.log, there are multiple lines of error message 'empty layer generated'.
As the result, all fields of the WAFS file (for example gfs.t12z.wafs_grb45f09.grib2) are constant values

[KateFriedman-NOAA]

@WenMeng-NOAA @YaliMao-NOAA Thanks for checking the output and confirming what is good/bad. @YaliMao-NOAA do you know if/when a solution is coming for the wgrib2 error? I have at least another week of testing the feature/hpc-stack branch before I look to submit the PR for it into global-workflow develop. I can leave WAFS out of that update for now while the wgrib2 issue is getting resolved.

[HuiyaChuang-NOAA]

@YaliMao-NOAA I know you opened an issue for wgrib2 error. Did you report team this to the library team?

[Hang-Lei-NOAA]

We updated the jasper associated with the wgrib2. Please test it again to see if this works.

…

On Tue, May 4, 2021 at 3:09 PM HuiyaChuang-NOAA ***@***.***> wrote: @YaliMao-NOAA <https://github.com/YaliMao-NOAA> I know you opened an issue for wgrib2 error. Did you report team this to the library team? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFDPDDKXGIINYFDUHB3TMBA6FANCNFSM4SZ3J73A> .

[YaliMao-NOAA]

@KateFriedman-NOAA @HuiyaChuang-NOAA @Hang-Lei-NOAA
I did a test and the output looks good now.

[KateFriedman-NOAA]

@Hang-Lei-NOAA Thanks for the update! @YaliMao-NOAA Thanks for testing and confirming. I'll be running another cycle in my test with the newer ufs-weather-model hash (554aedc) and will have you look at the WAFS output in that test as well when ready (including the grib2 jobs).

[HuiyaChuang-NOAA]

@KateFriedman-NOAA @HuiyaChuang-NOAA @Hang-Lei-NOAA
I did a test and the output looks good now.

Good to hear. @YaliMao-NOAA please update your issue NOAA-EMC/hpc-stack#224 and mark it as closed?

[KateFriedman-NOAA]

@JessicaMeixner-NOAA I updated my clone of feature/hpc-stack to use the new ufs-weather-model hash (554aedc). I built it with build_fv3.sh:

./compile.sh "$target" "APP=ATMW 32BIT=Y SUITES=FV3_GFS_v16" 2 NO NO

Venus: /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/build_fv3.sh

I then tried the provided build_ww3prepost.sh:

Venus: /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/build_ww3prepost.sh

...and it appears to have worked. Please see build log and resulting execs:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/build_ww3.log

[Kate.Friedman@v71a1 exec]$ pwd
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/exec
[Kate.Friedman@v71a1 exec]$ ll ww3_*
-rwxr-xr-x 1 Kate.Friedman emcmodel  5453096 May  5 12:50 ww3_grib
-rwxr-xr-x 1 Kate.Friedman emcmodel 11562672 May  5 12:50 ww3_grid
-rwxr-xr-x 1 Kate.Friedman emcmodel  3835408 May  5 12:50 ww3_outp

I haven't tried running with these execs yet, that's the next step. Thanks!

[YaliMao-NOAA]

@HuiyaChuang-NOAA As your reminder, I closed the issue NOAA-EMC/hpc-stack#224

[KateFriedman-NOAA]

Sync merge with develop @ 5c042e0

[KateFriedman-NOAA]

Committed to develop @ 96347ea