Restructure ROTDIR for $COMPONENT subfolders

[noaa-emc]

Restructure ROTDIR for $COMPONENT subfolders

[KateFriedman-NOAA]

@RussTreadon-NOAA @yangfanglin @JessicaMeixner-NOAA I have committed my initial set of changes to add $COMPONENTatmos and $COMPONENTwave variables to the workflow scripts. My local clone on Mars includes modifications for the components:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

Please feel free to review my initial changes and suggest modifications. I am running a cycled experiment with these changes and will commit further changes as needed. Will merge in recent feature/gfsv16b changes this week as well.

[KateFriedman-NOAA]

First half cycle run, got prep jobs at start of first full cycle to run yesterday, and am awaiting return of Mars (parallel production test) to continue test.

[KateFriedman-NOAA]

(below is fixed based on comment from Fanglin after this was first posted)

Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT:

1. JJOB script for job with multiple components (e.g. forecast):

export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}

2. JJOB script for job with single component (e.g. non-forecast jobs):

for atmospheric components:

export COMPONENT=${COMPONENT:-atmos}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

for wave components:

export COMPONENT=${COMPONENT:-wave}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

[yangfanglin]

Kate, I thought we should not use export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave} Instead using atmos and wave directly where they are referred. export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave} Fanglin

…

On Thu, Jul 16, 2020 at 12:32 PM Kate Friedman ***@***.***> wrote: Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT: 1. JJOB script for job with multiple components (e.g. forecast): export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave} export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave} 1. JJOB script for job with single component (e.g. non-forecast jobs): for atmospheric components: export COMPONENT=${COMPONENT:-atmos} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} for wave components: export COMPONENT=${COMPONENT:-wave} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKY5N2NQ3JYXI5LLG4NFC7DR34TQXANCNFSM4OLNIMHQ> .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; [email protected] http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

[KateFriedman-NOAA]

Fanglin, you're right, let me adjust that, sorry, got my notes on that special case mixed up.

[KateFriedman-NOAA]

@yangfanglin fixed at 3d9f0dc

[KateFriedman-NOAA]

Summary of changes for FV3/ufs-weather-model component: none, all changes on global-workflow side to both JJOB and ex scripts, no changes needed in ufs-weather-model

Changes to scripts on workflow side can be seen in feature/gfsv16b-restructure branch or here:

JGLOBAL_FORECAST

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/jobs/JGLOBAL_FORECAST jobs/JGLOBAL_FORECAST
72a73,80
> # Set wave variables
> if [ ${DO_WAVE:-"NO"} = "YES" ]; then
>   # WAVE component directory
>   export CDUMPwave=${CDUMPwave:-${CDUMP}wave}
>   export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
>   export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
> fi
>

exglobal_fcst_nemsfv3gfs.sh

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh scripts/exglobal_fcst_nemsfv3gfs.sh
42a43
> CDUMPwave="${CDUMP}wave"
153,157c154
<     if [ $CDUMP = "gdas" ]; then
<         RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart
<     else
<         RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart}
<     fi
---
>     RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart
163c160
<     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/RERUN_RESTART
---
>     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos/RERUN_RESTART
205c202
< memdir=$ROTDIR/${prefix}.$PDY/$cyc/$memchar
---
> memdir=$ROTDIR/${prefix}.$PDY/$cyc/atmos/$memchar
211c208
< gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/$memchar
---
> gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/atmos/$memchar
413d409
< #-------------wavewave----------------------
415c411,412
<   for file in $(ls $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/rmp_src_to_dst_conserv_*) ; do
---
>   # Link WW3 files
>   for file in $(ls $COMINwave/rundata/rmp_src_to_dst_conserv_*) ; do
418c415
<   $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/ww3_multi.${CDUMP}wave${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
---
>   $NLN $COMINwave/rundata/ww3_multi.${CDUMPwave}${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
419a417
>   # Check for expected wave grids for this run
421a420
>
423c422,424
<     $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.mod_def.$wavGRD $DATA/mod_def.$wavGRD
---
>     # Wave IC (restart) file must exist for warm start on this cycle, if not wave model starts from flat ocean
>     # For IAU needs to use sPDY for adding IAU backup of 3h
>     $NLN $COMINwave/rundata/${CDUMPwave}.mod_def.$wavGRD $DATA/mod_def.$wavGRD
426,432c427,435
<   WAVHCYC=${WAVHCYC:-6}
<   WRDATE=`$NDATE -${WAVHCYC} $CDATE`
<   WRPDY=`echo $WRDATE | cut -c1-8`
<   WRcyc=`echo $WRDATE | cut -c9-10`
<   WRDIR=$ROTDIR/gdaswave.${WRPDY}/${WRcyc}/restart
<   datwave=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/
<   wavprfx=${CDUMP}wave${WAV_MEMBER}
---
>   # Wave IC (restart) interval assumes 4 daily cycles (restarts only written by gdas cycle)
>   # WAVHCYC needs to be consistent with restart write interval in ww3_multi.inp or will FAIL
>   export WAVHCYC=${WAVHCYC:-6}
>   export WRDATE=`$NDATE -${WAVHCYC} $CDATE`
>   export WRPDY=`echo $WRDATE | cut -c1-8`
>   export WRcyc=`echo $WRDATE | cut -c9-10`
>   export WRDIR=${ROTDIR}/${CDUMPRSTwave}.${WRPDY}/${WRcyc}/wave/restart
>   export datwave=$COMOUTwave/rundata
>   export wavprfx=${CDUMPwave}${WAV_MEMBER}
444c447
<     wavicefile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVEICE_FID}.${cycle}.ice
---
>     wavicefile=$COMINwave/rundata/${CDUMPwave}.${WAVEICE_FID}.${cycle}.ice
454c457
<     wavcurfile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVECUR_FID}.${cycle}.cur
---
>     wavcurfile=$COMINwave/rundata/${CDUMPwave}.${WAVECUR_FID}.${cycle}.cur
465a469,470
>
>   # Loop for gridded output (uses FHINC)
491a497
> fi #cplwav=true
493,495d498
< fi
< #-------------wavewave----------------------
<

[yangfanglin]

Kate,

The structure of the following block needs to be preserved for restarting wave model from a breakpoint. NCO will save breakpoint restart files for the gfs cycle in a $nwges directory , while restart files for the gdas cycle in $comrot. So in operation RSTDIR_WAVE for the gfs cycle will be defined in an ecflow suite definition file. This is also true for the atmos part.

< if [ $CDUMP = "gdas" ]; then
< RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart
< else
< RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart}
< fi

You can change this block to

if [ $CDUMP = "gdas" ]; then
RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart
else
RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart}
fi

[KateFriedman-NOAA]

Fanglin, gotcha, changed it to the block you provided at 273c535, thanks!

[KateFriedman-NOAA]

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure
Changes committed: https://github.com/KateFriedman-NOAA/GSI/commit/f6bc0f6a010b0606bf6003f14872fb3ddc80f96f

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure
Changes committed: https://github.com/KateFriedman-NOAA/GLDAS/commit/ca6188375dfb46ac8b59346437c1b96ad50ea708

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure
Changes committed: https://github.com/KateFriedman-NOAA/EMC_post/commit/9cb2723b00c276a2e004a64a4bebf1e0e3f943ca

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure
Changes committed: https://github.com/KateFriedman-NOAA/EMC_gfs_wafs/commit/52ed70d8351d186f04e25b5fe87f3bfb38b7fa5e

[KateFriedman-NOAA]

Continuing to let test run through additional cycles. Test on Mars:

EXPDIR: /gpfs/dell2/emc/modeling/save/Kate.Friedman/expdir/testrestrucb
COMROT: /gpfs/dell3/ptmp/Kate.Friedman/comrot/testrestrucb
clone: /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

[RussTreadon-NOAA]

Quick look at GFS v16 DA changes looks good. $COMPONENT has been added to DA j-jobs and two exglobal scripts. COMPONENT defaults to "atmos" if not externally set. Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

[KateFriedman-NOAA]

Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from:

$CDUMPwave.$PDY/$cyc
...to:
$CDUMP.$PDY/$CYC/wave

[RussTreadon-NOAA]

OK. I'll wait for and process the PR at the appropriate time.

…

On Thu, Jul 16, 2020 at 2:50 PM Kate Friedman ***@***.***> wrote: Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel. I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from: $CDUMPwave.$PDY/$cyc ...to: $CDUMP.$PDY/$CYC/wave — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN63ZCRRL3KGTE2FMIC3DR35DVFANCNFSM4OLNIMHQ> .

[GeorgeGayno-NOAA]

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure
Changes committed: KateFriedman-NOAA/GSI@f6bc0f6

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure
Changes committed: KateFriedman-NOAA/GLDAS@ca61883

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure
Changes committed: KateFriedman-NOAA/EMC_post@9cb2723

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure
Changes committed: KateFriedman-NOAA/EMC_gfs_wafs@52ed70d

[KateFriedman-NOAA]

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

Thanks for the confirmation that fv3gfs_chgres.sh is indeed obsolete now! I'll forgo submitting changes to you for fv3gfs_chgres.sh then, thanks @GeorgeGayno-NOAA !

[KateFriedman-NOAA]

PR for EMC_post submitted: NOAA-EMC/UPP#151

[KateFriedman-NOAA]

PR for GSI submitted: NOAA-EMC/GSI#39

PR for GLDAS submitted: NOAA-EMC/GLDAS#8

PR for WAFS submitted: NOAA-EMC/WAFS#1

[KateFriedman-NOAA]

EMC_post PR NOAA-EMC/UPP#151 merged in.

New tag created: upp_gfsv16_release.v1.0.11

Moved feature/gfsv16b-restructure checkout.sh to use new EMC_post tag. Committed at ab89088.

[KateFriedman-NOAA]

GLDAS PR NOAA-EMC/GLDAS#8 merged in.

New tag created: gldas_gfsv16_release.v1.4.0

Moved feature/gfsv16b-restructure checkout.sh to use new GLDAS tag. Committed at 95ce823.

[KateFriedman-NOAA]

Did another sync merge of global-workflow and GSI branches...no new updates to pull in.

Ran new ens_tracker.v.1.1.15.3 from Jiayi successfully. Sent a few small edits back to him and he is now pushing this version to GitHub. Will update on-disk clone of the tracker when on GitHub.

Waiting for WAFS v16 still but will ingest that into system when available and will move the real-time parallel before that.

Running final test with all pieces in place and turned on today (minus WAFS) on Mars. Sent update email to code managers. Putting together notes for Russ for moving the real-time parallel to the updated $COMPONENT pieces.

Need to install updated external pieces on other platforms to support this new structure there (obsproc, fit2obs, tracker).

[KateFriedman-NOAA]

Updated tracker in GitHub: https://github.com/NOAA-EMC/TC_tracker/tree/TC_tracker.v1.1.15.3

[KateFriedman-NOAA]

WCOSS outages this week will delay implementing these changes in the real-time parallel. Plan is to allow real-time parallel to catch back up to real-time over the weekend and early next week ahead of a prod switch next Tuesday. Will set up and test system on Venus this week and plan to move to new structure when the parallel moves to Venus post-prod switch.

Will resync global-workflow and GSI PR branches again before the end.

[KateFriedman-NOAA]

Preparing to move v16rt2 to new structure after WCOSS prod switch this Thursday. Working with Russ to set up Venus as much as we can in advance. Will submit GSI and global-workflow PRs Wednesday (if not before).

[KateFriedman-NOAA]

GSI PR completed: NOAA-EMC/GSI#39

[KateFriedman-NOAA]

Global-workflow PR completed: #109