Convert checkout.sh to manage_externals

[noaa-emc]

Convert checkout.sh to manage_externals

[KateFriedman-NOAA]

Example Externals.cfg from prior tests:

# External sub-modules of global-workflow

[NEMSfv3gfs]
tag = nemsfv3gfs_beta_v1.0.18
local_path = sorc/fv3gfs.fd
repo_url = ssh://vlab.ncep.noaa.gov:29418/NEMSfv3gfs
protocol = git
required = True

[GSI]
tag = fv3da.v1.0.43
local_path = sorc/gsi.fd
repo_url = ssh://vlab.ncep.noaa.gov:29418/ProdGSI
protocol = git
required = True

[EMC_post]
tag = ncep_post.v8.0.27e
local_path = sorc/gfs_post.fd
repo_url = ssh://vlab.ncep.noaa.gov:29418/EMC_post
protocol = git
required = True

[UFS_UTILS]
tag = v1.0.0
local_path = sorc/ufs_utils.fd
repo_url = ssh://vlab.ncep.noaa.gov:29418/UFS_UTILS
protocol = git
required = True

[EMC_gfs_wafs]
tag = gfs_wafs.v5.0.9
local_path = sorc/gfs_wafs.fd
repo_url = ssh://vlab.ncep.noaa.gov:29418/EMC_gfs_wafs
protocol = git
required = True

[externals_description]
schema_version = 1.0.0

[KateFriedman-NOAA]

Created new branch off of develop for this work: feature/manage_externals

[KateFriedman-NOAA]

Added copy of manage_externals under util folder. Created initial Externals.cfg. Test on Hera worked as intended.

Did a little house-keeping and removed scripts/files associated with prod_util and grib_util builds which are no longer supported. Also introduced .gitignore file to make things cleaner.

[KateFriedman-NOAA]

Removed copy of manage_externals from branch, will installed on supported platforms. Introduced README.md to document dependency on manage_externals for public users.

Sync merged feature/manage_externals with develop branch.

[KateFriedman-NOAA]

Installed manic-v1.1.6 tag on Hera:

/scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.6

...and on WCOSS-Dell (Venus and Mars):

/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.6

...and on WCOSS-Cray (Surge and Luna):

/gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.6

[KateFriedman-NOAA]

Tested branch checkout using new manage_externals install on Hera (worked nicely):

-bash-4.2$ /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

-bash-4.2$

[KateFriedman-NOAA]

Test on Luna received this error:

LUNA-llogin1 > /gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, ERROR:root:Command '[u'git', u'clone', u'--quiet', u'ssh://vlab.ncep.noaa.gov:29418/NEMSfv3gfs', u'fv3gfs.fd']' returned non-zero exit status 128
ERROR:root:Failed with output:
    fatal: ssh variant 'simple' does not support setting port

ERROR: In directory
    /gpfs/hps3/emc/global/noscrub/Kate.Friedman/git/feature-manage_externals/sorc
Process did not run successfully; returned status 128:
    git clone --quiet ssh://vlab.ncep.noaa.gov:29418/NEMSfv3gfs fv3gfs.fd
See above for output from failed command.

Issue resolved by adjusting git config ssh setting:

git config --global ssh.variant ssh

Will need to add a note about this to wiki section for checkout.

Test checkout worked well using newly install manage_externals tag on Luna:

LUNA-llogin1 > /gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

LUNA-llogin1 >

Same error, fix, and then successful test on Surge.

[KateFriedman-NOAA]

Successful test on Mars and Venus:

[Kate.Friedman@m71a2 feature-manage_externals]$ /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

[Kate.Friedman@m71a2 feature-manage_externals]$

Note: emc_gfs_wafs not checked out since it is set as optional.

[KateFriedman-NOAA]

Tested feature/manage_externals branch on Hera after sync merging with the develop branch. Got to see a typical Hera error in action (frequent connection issues with GitHub):

-bash-4.2$ /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ERROR:root:Command '[u'git', u'fetch', u'--quiet', u'--tags', 'origin']' returned non-zero exit status 128
ERROR:root:Failed with output:
    fatal: unable to access 'https://github.com/NOAA-EMC/EMC_post.git/': Failed connect to github.com:443; Connection timed out

ERROR: In directory
    /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-manage-externals/sorc/gfs_post.fd
Process did not run successfully; returned status 128:
    git fetch --quiet --tags origin
See above for output from failed command.

ERROR:root:Failed with output:
    fatal: unable to access 'https://github.com/NOAA-EMC/EMC_post.git/': Failed connect to github.com:443; Connection timed out

ERROR: In directory
    /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-manage-externals/sorc/gfs_post.fd
Process did not run successfully; returned status 128:
    git fetch --quiet --tags origin
See above for output from failed command.


ERROR: Failed with output:
    fatal: unable to access 'https://github.com/NOAA-EMC/EMC_post.git/': Failed connect to github.com:443; Connection timed out

ERROR: In directory
    /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-manage-externals/sorc/gfs_post.fd
Process did not run successfully; returned status 128:
    git fetch --quiet --tags origin
See above for output from failed command.

A retry a minute later was successful (no GitHub connection issues) and allowed me to see it checking the components already checked out before proceeding to the others:

-bash-4.2$ /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.6/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

-bash-4.2$

[KateFriedman-NOAA]

Tests using the manic-v1.1.6 tag failed (sub-modules not checked out for fv3gfs and gsi), that tag does not include fixes made later (which were in master copy used for initial testing). Switched to later manic-v1.1.7 and it worked. Installed manic-v1.1.7 on machines and removed manic-v1.1.6.

Will update checklist above for new tag.

[KateFriedman-NOAA]

Example of what it looks like when cloning on Hera hangs (as it often does sadly, especially with the GSI checkout which is huge):

-bash-4.2$ /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.7/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi,

Command 'git submodule update --init --recursive'
from directory /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-manage-externals/sorc/gsi.fd
has taken 300 seconds. It may be hanging.

The command will continue to run, but you may want to abort
manage_externals with ^C and investigate. A possible cause of hangs is
when svn or git require authentication to access a private
repository. On some systems, svn and git requests for authentication
information will not be displayed to the user. In this case, the program
will appear to hang. Ensure you can run svn and git manually and access
all repositories without entering your authentication information.


emc_verif-global,

-bash-4.2$

I let it keep doing its thing and manage_externals kept trying to clone GSI and then when successful it moved on to the final checkout (emc_verif-global).

What a good (hang-free) checkout looks like on Mars:

[Kate.Friedman@m72a1 feature-manage_externals]$ /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.7/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

[Kate.Friedman@m72a1 feature-manage_externals]$

[KateFriedman-NOAA]

Sync merged with develop branch and then updated EMC_verif-global tag in externals config.

New manage_externals version released on Tuesday (manic-v1.1.8). Have installed on supported platforms and will now test and update README if good.

[KateFriedman-NOAA]

Test of new manic_v1.1.8 tag was successful on Venus:

[Kate.Friedman@v72a2 feature-manage_externals]$ /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_gfs_wafs, emc_verif-global,
Checking out externals: nemsfv3gfs, emc_post, ufs_utils, gsi, emc_verif-global,

[Kate.Friedman@v72a2 feature-manage_externals]$ git status
On branch feature/manage_externals
Your branch is up-to-date with 'origin/feature/manage_externals'.

nothing to commit, working tree clean
[Kate.Friedman@v72a2 feature-manage_externals]$ cd sorc/
[Kate.Friedman@v72a2 sorc]$ ls -ltr
total 11296
...
drwxr-sr-x 17 Kate.Friedman emcmodel  2048 Mar  6 15:25 fv3gfs.fd
drwxr-sr-x 11 Kate.Friedman emcmodel  2048 Mar  6 15:26 gfs_post.fd
drwxr-sr-x 11 Kate.Friedman emcmodel   512 Mar  6 15:26 ufs_utils.fd
drwxr-sr-x 15 Kate.Friedman emcmodel  2048 Mar  6 15:27 gsi.fd
drwxr-sr-x  9 Kate.Friedman emcmodel   512 Mar  6 15:30 verif-global.fd

Test of build_all.sh after running manage_externals was successful:

[Kate.Friedman@v72a2 sorc]$ sh build_all.sh
Creating logs folder
Creating ../exec folder
 .... Building fv3 ....
 .... Building gsi ....
 .... Building ncep_post ....
 .... Building ufs_utils ....
 .... Building gaussian_sfcanl and nst_tf_chg ....
 .... Building enkf_chgres_recenter ....
 .... Building tropcy_NEMS ....
 .... Building gdas ....
 .... Building gfs_fbwndgfs ....
 .... Building gfs_overpdtg2 ....
 .... Building gfs_wintemv ....
 .... Building gfs_bufrsnd ....
 .... Building fv3nc2nemsio ....
 .... Building regrid_nemsio ....
 .... Building gfs_util ....
 .... prod_util build not currently supported ....
 .... grib_util build not currently supported ....

 .... Build system finished ....
[Kate.Friedman@v72a2 sorc]$

Will proceed with updating README to reflect new manic version.

[KateFriedman-NOAA]

Ran link script after build and noticed wafs links were broken throughout which is understandable since manage_externals didn't check it out since wafs is optional. Updated link_fv3gfs.sh to only make wafs links if sorc/gfs_wafs.fd folder exists. Removed machine dependency on wafs links.

[KateFriedman-NOAA]

Updated README with new manic version.

This is ready for a PR. Will initiate PR and ask for reviewers to test.

[KateFriedman-NOAA]

Initiated PR #29

[KateFriedman-NOAA]

Received initial buy-in from NCO (email with Steven Earle). No requirement to keep checkout.sh but will keep during transition period. Proceeding with commit (PR #29).

[KateFriedman-NOAA]

Completed PR #29. Need to work on setting up modules.

[KateFriedman-NOAA]

Will use contrib documentation for Hera to setup manage_externals module there:

https://rdhpcs-common-docs.rdhpcs.noaa.gov/wiki/index.php/Managing_Packages_in_/contrib

[KateFriedman-NOAA]

global-workflow CMs have decided not go use manage_externals and will wait to see the solution developed for the unified system. Keeping open until removed.

[WalterKolczynski-NOAA]

As soon as we remove the manage externals stuff, we can close this issue.

[aerorahul]

manage_externals needs to be tested on Orion w/ Gerrit hosted submodules.
@aerorahul to test.

[arunchawla-NOAA]

what is the status of this? It looks like we have both capabilities. Can we close this issue? If we have to remove manage externals at some point we can create a new issue then

[KateFriedman-NOAA]

We're using sorc/checkout.sh still but maintaining Externals.cfg for now. Possibly will remove Externals.cfg in the near future after we make a final decision. Will open an issue to remove it if so, as you suggest.

Closing this issue since manage_externals is supported now.