noresmhub/noresm

Norwegian Earth System Model and Documentation

noresm's Issues

CAM6-Nor documentation

Dear all,
@DirkOlivie @lisesg @oyvindseland @tto061 @Kirkevag

We will release a new version of NorESM2 the 15th of June. I hope the documentation can be (rather) done by then.

1) We are missing a model description of CAM6-Nor:
https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html

I think the best way is to make a list of the NorESM specific additions and most importantly add a description. I think the users should get a brief overview without needing to read all the referenced papers. Who is willing to do that? I don't know CAM6-Nor well enough. The easiest way is if you modify the file directly: https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html
You can also send me an email and I can copy the information into the document.

Please see the BLOM description for an example:https://noresm-docs.readthedocs.io/en/noresm2/model-description/ocn_model.html

2) We also need to add a description of the initial conditions in CAM6-Nor. Where does the model start from? Is it a cold start? What about the stratosphere? The file to modify is here: https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html

3) Code modifications. What is the best procedure to make code modifications with the new NorESMhub setup? A description can be added at the bottom of this page:
https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.htm

Let me know if you have questions!

Best regards,
Ada Gjermundsen

Coding principles: print out all file names that are about to be read or written

It's very hard to identify needed files if the code does not tell what it is doing.
In an ideal world you print out all filenames read or written with full path to make them easy to find. If you are annoyed by it, put a verbose flag somewhere.

Link to noresm.org web site

I think the README.md file should include a link to the www.noresm.org web page.

Misleading comment for supported compsets

After create_newcase and given that the set-up is supported this line is reported back

"This is a CESM scientifically supported compset at this resolution."

However it is a NorESM specific compsets.

Should we let it stand as is. Add "or NorESM" into the wording?.
Ideally we should have our own category, although likely not a priority now.

New release for NorESM with the latest CAM (cam_cesm2_1_rel_05-Nor)

I noticed that the current Externals.cfg still points to cam_cesm2_1_rel_05-Nor_v1.0.0 which do not contain changes for the trailing slash, for example (and hence does not work for external users)

More generally how is this kind of updates managed (or how will it be managed in the future)?

How are NorESM users informed of the latest version they have to use?

Should we not point in Externals.cfg to (development) branches rather than releases so we would always have the latest updates?

I do not pretend to know what is best, but I believe that now a decision has to be made

Testing of NorESM2.0.2

@DirkOlivie @monsieuralok @oyvindseland @matsbn @j34ni @adagj
When the latest issues (tagged next release) are worked up and solved, please initiate testing of the NorESM2.0.2 and report under this issue, which tests are done and what the results was.

Include emission driven and SSP extension compsets in new release

We should include these, since we have done CMIP6 simulations using these compsets. I am currently doing a test of emission driven historical and will issue pull requests to NorESM and CAM repositories when this test is successful.

No branch information in README.case

Hi,
@DirkOlivie

In README.case the name of the branch used to be included in the INFORMATION ABOUT YOUR GIT VERSION CONTROL SYSTEM section, but now only an * appear after the git branch: regardless of which branch is used. The commit nr is given, so it is possible to get the information but it is much easier to find out what is used if the branch name is listed. Is that possible to fix?

Ex:

2020-05-14 09:56:26: git branch:* (detached from cime5.6.10_cesm2_1_rel_06-Nor_v1.0.1) 3e5838e Merge pull request #5 from matsbn/feature-replace-MICOM
master 4c26f56 [origin/master] Merge pull request #3383 from cacraigucar/add_namelist_double

Best regards,
Ada

All input data needs to have the version information in the file name

We need to agree on a input data file naming scheme. The easiest is to put version information in the file name just as CESM is doing it.

additional info from Dirk:
Some NorESM data (in this case the directory) have a version date in their directory name :
Location : inputdata/noresm-only/atm/cam/camoslo/AeroTab_8jun17 :
Filenames : aerocomk0.out, aerocomk1.out, aerocomk2.out, aerocomk3.out, ...

Update ESMValTool documentation for NIRD Toolkit

I am updating the documentation about ESMValTool, mainly the part about NIRD Toolkit.

Wrong component local_path name in Externals.cfg for BLOM

In Externals.cfg change

[blom]
...
local_path = components/micom
...

to

[blom]
...
local_path = components/blom
...

Can we update Externals.cfg to required=False for pop2 and ww3?

Hi!
Most I have discussed with run into several SVN-related errors when launching the model. All need to change required=True to required=False for pop2 and ww3 in Externals.cfg.
POP2 and WW3 are not needed in NorESM2 so I'm thinking it is better if those are set to false by default.

Is it possible to update Externals.cfg so required=False for pop2 and ww3 are default?

@DirkOlivie @oyvindseland @matsbn
Ada

Draft a guide on how to contribute and update documentation

This will help potential contributors to easily update the documentation; either from the command line or online.

Students should be able to modify the documentation so the guide should be simple and understandable by anyone even without advanced git/github skills.

Expand the "Configure Externals.cfg" to list available options

The Configure Externals.cfg is basically a placeholder with a link to the CESM readme page. It would be useful to expand on this section with a list of available options.

Re-organize newbees section

• Have first how to get the code at the top
• ...
  --
  --

Explain role of branches in README.md of NorESM and CAM-Nor

The README.md file in the master branch of NorESM and CAM-Nor should shortly explain the role of the different branches.

CIME ERROR in manage_externals

Hi,
@DirkOlivie @matsbn
Kine and I get this error when trying to checkout manage_externals with the newest version. This is the case for both tag and branch.

Checking status of externals: cam, cice, cime, cism, clm, mosart, pop, blom, rtm, ww3,
Checking out externals: cam, cice, cime, ERROR:root:In repo "cime": tag "cime5.6.10_cesm2_1_rel_06-Nor_v1.0.0" is both a branch and a tag. git may checkout the branch instead of the tag depending on your version of git.

ERROR: In repo "cime": tag "cime5.6.10_cesm2_1_rel_06-Nor_v1.0.0" is both a branch and a tag. git may checkout the branch instead of the tag depending on your version of git.

Nudgning compset: Should default drydep_method in release-noresm2.0.1 be 'xactive_atm' or 'xactive_lnd'?

The default drydep_method seems to be:

 drydep_method		= 'xactive_atm'

in the new release, while in the noresm-dev version (branch featureCESM2.1.0-OsloDevelopment) on metno it has

 drydep_method		= 'xactive_lnd'

This results in substantially lower SO2 concentrations (amongst other things) and I thought that @Kirkevag and @DirkOlivie told me in a discussion on the issues there, that 'xactive_lnd' was preferrable.
So I guess my question is: is this intentional? Should I use 'xactive_lnd' or 'xactive_atm'?
Both cases are set up with compset NFHISTnorpddmsbcsdyn.

CMIP5 and CMIP6 data access

The access to the CMIP5&6, happyEVA and DCPP data should be explained in one section:

via ESGF
via nird account and read access to NS9034K
via raw output access, with knowledge of the cases used for the CMIP experiments

Label science_supported grids of NorESM2 specific compsets

Modify config_compsets.xml in CAM-Nor and in CIME : add the science_supported grids to the compsets used for CMIP6.

This avoids having to use the flag --run-unsupported when using building a case (./create_newcase) and gives an indication of which compsets have been tested and run for a longer time (and possibly contributed to CMIP6 simulations).

proposal of how to handle an input data repository

(locations are not decided of.http://ns2345k.web.sigma2.no/NorESMInputdata/ just serves as a proxy)
Organise as follows:

./NorESM2.v0.01
./NorESM2.v0.02
./NorESM2.stable
./papers
./papers/GMD.20101231
./NorESM2.current

• stable is a link to the latest stable version
• current might get opened to local project users to put their data to.
• published versions will never be altered
• all versions contain all data. Hard links will make sure disk usage is minimised
• at least GMD requires to provide a single place for input data used. Therefore also CESM data should be included there.

Add missing compsets after February

Problems when resubmitting on nebula

Hi,
I tested out the newest version of NorESM2 (branch origin/noresm2) on nebula and an old problem which was fixed reappears.

For the re-submission, the model crashes due to this error:
ERROR:
GETFIL: FAILED to get ./NF2000climo_f19_f19_mg17_20200601.mosart.rh0.0026-01-01
-00000.ncC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC ³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC³^HC`³^HC

which is a nebula specific error due to how strings are handled. That was fixed in the old noresm-dev repository by:

in components/mosart/src/riverroute/RtmRestFile.F90

changed Line 428 from
long_name=trim(lname), units=trim(uname), fill_value=spval)
to
long_name=trim(lname), units=trim(uname))

but reappeared now using the new NorESM2hub version.

Best regards,
Ada

All specific input data should be available from noresm.org

NIRD needs to be the central place for input data. ALL input data needs to be there.
Details to follow.

case folder needs to be archived

To be able to reproduce a simulation we need to archive the case folder together with the data. Aleksi suggested some tools Mats has developed. the input-group suggest to edit the env_archive.xml somehow.

Make doi for new NorESM2 release (June 2020)

Copyright of documentation

On many or all pages on https://noresm-docs.readthedocs.io/en/latest/ there is written:

© Copyright 2018, Anne Fouilloux Revision 1229b03.

and no rights information.

I suppose the copyright information is automatically generated from GitHub's upstream commits, but it is incorrect (for at least some pages).

In addition, a CC-BY-4.0 or CC-BY-SA-4.0 licence is an idea.

Multiple pull requests to replace MICOM with BLOM

The mutual dependent pull requests #63, NorESMhub/cime#5, NorESMhub/CAM#2, NorESMhub/CESM_CICE5#2, NorESMhub/BLOM#7 have been issued to complete the transition from MICOM to BLOM, except for documentation.

I have verified bit-identical results and correct restart for N1850frc2 compset with f19_tn14 resolution and NOIIAOC compset with tn14 resolution on Fram. Although output and restart files from BLOM are now are named *.blom.* compared to *.micom.* before, the updated NorESM is able to branch from old experiments with *.micom.* type restart files.

If these pull requests are accepted, new tags should be created for the updated repositories and Externals.cfg updated accordingly.

noresm2 documentation: Link in newbies guide to download info is dead readthedocs

The link in the newbies guid
https://github.com/NorESMhub/NorESM/blame/noresm2/doc/configurations/newbie-guide.rst#L13
is not working in readthedocs
https://noresm-docs.readthedocs.io/en/noresm2/configurations/newbie-guide.html

I guess this is because of the relative path, but I didn't find an easy fix yet. Does anyone have any ideas?

Github procedures

@annefou We should link to some short explanations how to fork and pull request in the contribution section. Update a fork? add a file from another branch, etc.

Update cime

OMIP case examples: do we need '--run-unsupported' flag?

Hi @matsbn @ChunchengGuo @j34ni
I see that the OMIP case examples in the documentation includes the '--run-unsupported' flag. Do we really need this? I tested these examples on FRAM without the '--run-unsupported' flag, and I don't see any problems with my runs.

Writing a release note

• Make an extended description of the tag release on github
• Should be also mentioned/copied in the documentation, Readme that there is this description
• Mentioning not resolved issues

KeyClim branches for WP6 simulations

@TomasTorsvik

For KeyClim WP6, simulations will be done with around 10-different model versions. All these model versions will be based on the release-noresm2.0.2, but include changes (often in one aspect of the model - chemistry or PBL parameterisation or ...).

For visibility and intercomparibility, it would be good that all these versions exist as branches on github.com/NorESMhub in the respective repositories (e.g, NorESM, CAM, ...) I assume. The few developers that work on one modification can than exchange via those branches their progress, and developers that work on other modifications can at least see what others do. These branches then also define the exact version used for the different simulations in WP6.

Is this a reasonable way to do this?

But then all the individual developers implied in this process should need write access in the respective NorESMhub (NorESMhub/CAM, or NorESMhub/NorESM, or ...) repositories to start a new branch, and to manage the pull requests.

Is that how it should be?

MICOM does not print out all input data files before they are tried to be read

In order to identify files needed for the input data repository a user needs to know which files are actually needed. In some cases this is not the case right now since the file name is not printed out before the code tries to open a file. The user is then just left with an error message that some file could not be opened, but he does not know what file it was.
Detailed info will follow.

Tag components for new NorESM2 release (June 2020)

cime_config directory

@annefou In the current trial setup for NorESM, cime_config and its sub-directories have been copied in manually. But shouldn't that be better a fork too? In the future, I would think that we also want to be able to update cime_config when updates in CESM appear.
This maybe implies that the NorESM repository is a fork of CESM ...

Missing NorESM contributing guidelines

It would be nice to add contributing guidelines in the NorESM repository to explain (and encourage) external contributions.

Currently this is only available for the documentation (https://github.com/NorESMhub/NorESM/blob/master/doc/README.md).

Having a CONTRIBUTING.md file is the main repository seems to be a good practice.

Highly suspected "bug" related to emission files & reproducibility issues

Adding emission files significantly affects the reproducibility of the simulations, even if they only contain zeros (whether these zeros are defined as float or as double precision).

This is very likely linked to the problem that occurred in the summer 2019 (while carrying on the CMIP6 runs) although at that time it made the model crash more/less randomly, which is not the case here.

release-noresm2.0.1

For this realease, we should take care of the following:

1. in Externals.cfg, ww3, pop, and cism should have the "required" field set to False, since we have not used and we do not explicitly support any compsets that include these components. Users can always put them back to True and re-run checkout_externals if they want, knowing that they do so on their own initiative
2. in cime_config/config_compsets.xml, components/cam/cime_config/config_compsets.xml, and components/micom/cime_config/config_compsets.xml, we need <science_support grid="xxx"/> fields, so that users know what configurations we support (and can be created without the --run-unsupported option) and what configurations we do not support (and require --run-unsupported).

NorESM2 integrity checks

Potentially linked to issue #55

This issue attempts to pull together various problems experienced so far with NorESM2 integrations:

1. occasional crashes due to NaN's generated during execution: machine dependent, difficult to reproduce, uncertain cause; possibly more frequent/likely with the NF2000climo compset; rare or absent in CMIP6 compsets; possible origin in the oslo chemistry solver; more generally uninitialised non-local arrays also possible cause
2. crashes and non-reproducibility (bfb) in CMIP6 compsets when not using "frc2" emission files; possible race condition in open/close statements using the same unit number when reading forcing files
3. uncertainty on bfb reproducibility of CAM-Nor runs under different PE counts

Proposed tests (all to be run without "airbag" mods):

• Thomas: follow up on NF2000climo problems from tetralith users, collect casistics; test bfb reproducibility of NFHISTfsstfrc2 case with different PE count on tetralith
• Øyvind: test F2000climo (CESM2) and NF2000climo (NorESM2) runs with different PE counts on vilje for crashes and for bfb reproducibility
• Alok: run long (~100 years) NF2000climo integrations on vilje and fram checking for crashes
• Ada: test NF2000climo run on Nebula
• Dirk: test two parallel, identical N1850 cases on fram to check for divergence (~200 years); using older, non-"frc2" input file set (it is assumed that "frc2" N1850 cases are reproducible bfb based on previous tests)

Dirk will also set up an explicit "zero forcing" volcanic case for Jean Iaquinta hoping that this will be useful for his current experiment.

git clone error on Vilje

Hi,

I tried to git clone from NorESMhub on Vilje, and got the following error -

git clone https://github.com/NorESMhub/NorESM.git noresm_test
Cloning into 'noresm_test'...
error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing https://github.com/NorESMhub/NorESM.git/info/refs?service=git-upload-pack
fatal: HTTP request failed

It works well on Fram and my local desktop. Does anyone have any clue on this?

Integration of BLOM in NorESM (instead of MICOM)

There is now a new version of the ocean component called BLOM instead of MICOM (see https://github.com/NorESMhub/BLOM/releases/tag/blom0_0_001).

Is there any plan to integrate these changes in the master branch of NorESM?

Is anyone already working (or willing to work) on it at the moment?

Install NorESM via conda more comfortably on any machine

Packaging the environment in a container would make installation of NorESM much easier and independent of local machine settings. It requires however root privileges on a given machine.

Initial tests are made to install NorESM via conda.

Coming soon: documentation about how to install the latest conda CESM environment and use it to run NorESM on a Virtual Machine or laptop/desktop to start with (eventually there will be a conda NorESM)​

More general: regarding containers, Sigma2 has been asked for some time to install something like Singularity or Sarus (which are perfectly secure for use on HPC, and are actually already used elsewhere) and that was never done, so Sigma2 is aware that there is a demand, but it is clearly not given any priority.

New input data structure for BLOM/iHAMOCC

@JorgSchwinger and I are working to restructure the input data for BLOM and iHAMOCC (https://github.com/NorESMhub/BLOM/tree/feature-blom_inputdata). The MICOM input data structure has essentially been fixed since 2009 and we have avoided restructuring before because it would have led to a messy and confusing structure due to the many model versions we are maintaining. It is a rare opportunity to do this restructure when transitioning from MICOM to BLOM and it should be completed before announcing a NorESM2 release to a wider user base, otherwise I fear we are stuck with the current structure for another decade. We have started this work on Fram, but Fram has experienced much unannounced downtime lately so we are delayed. We will update this issue with progress on the restructure.

Merge noresm2 into master

different BLOM inputdata structure on Vilje

The BLOM input data directory on Vilje is not the same as on Fram, therefore the case reports missing files after submitting.

The case also failed to download the input files from noresm.org, e.g. -

Checking server https://www.noresm.org/inputdata with protocol wget
ERROR: Could not connect to repo '{0}'
This is most likely either a proxy, or network issue .

One can modify the user_nl_blom file to resolve this, but is there any plan to update the inputdata structure on Vilje? or any other recommendations/tips? @matsbn @JorgSchwinger

Missing automated download of some BLOM files?

finally managed to run a NorESM test again on Fram, but got an error message about missing inputfiles:

CaseDocs/ocn_in:  INIDIC     = '/cluster/shared/noresm/inputdata/ocn/blom/inicon/glodapv2_Ct_preind_OMIPinit_20171107.nc'
CaseDocs/ocn_in:  INIALK     = '/cluster/shared/noresm/inputdata/ocn/blom/inicon/glodapv2_At_OMIPinit_20171107.nc'

Those exist on /work/shared but not when I define my own input root.
On the other hand. When I ran the model 3 weeks ago the model simulations worked without these files, they were not even listed but without extra aerosol output
When I ran the model with standard input root these files were listed also three weeks ago so it looks like building ocn_in just copies the entire directory not individual files? And now the model actually ask for these files?

Or are they only needed for increased aerosol output in the atmosphere strange as that may sound?

BLOM - list of input data

Hi all,
@matsbn @j34ni @jgriesfeller
in casefolder/Buildconf/micom.input_data_list, there are still several blom files which are not listed. E.g. ocean_regions.nc, inicon.nc. I think @j34ni may have the complete list of missing files. It is very difficult to get an overview of the missing files in BLOM if those files are not listed because you need to run the model several times, wait for it to crash and hope the error message is readable. I'm not sure if these "missing" files will be downloaded automatically if they are not listed. Then we need to add a description at least.

micom.input_data_list:
grid_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/grid.nc
ncep_landmask_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/land.sfc.gauss.nc
ncep_height_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/hgt.sfc.nc
ice_clim_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/icec_1968-1996.uf
sst_clim_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/skt_1968-1996.uf
tidal_dissipation_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/tidal_dissipation.nc
meridional_transport_index_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/mertraindex.dat
meridional_transport_basin_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/mertraoceans.dat
section_index_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/secindex.dat
dust_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/INPDUST_mhw.nc
river_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/river_nutrients_GNEWS2000.nc
n_deposition_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/ndep_tnx1v4_185001-201412.nc

Ada

Issue and pull request templates

Based on feedback from the INES annual meeting, I have been looking into the options for making templates for issues and pull requests.

It is possible to have different templates for different types of issues, e.g. bug fixes, feature requests, improve documentation, etc., and include a default issue text, preset labels and preset assigners. I have tested it out in one of my own repos. This can be nice to have if there are many issues being created by people who do not have write access to the repo, and/or is unsure about what info to include with the issue. The downside is that it creates an intermediate step whenever someone wants to create an issue, but this shouldn't really be a big obstacle. Some people may find it annoying to have these preset options, so I think it is good to also include a blank "general issue" option.

I'm not sure how useful this option really is, as we do not have much (any?) issues created from outside the development team at the moment. It's also a question if we should have preset assigners, and if so, who this should be.

Øyvind mentioned another type of "template", that sounded to me like a step-by-step instruction on how to work with the issue and pull request system in gitHub ( @oyvindseland , please correct me if this is wrong ). I created something like that for the NorESM user workshop that we had last week, the slides are in my google drive
https://docs.google.com/presentation/d/16JL15wrb56wfMETrxWPKNP8tDsK5TVsFUM-lpboKXmc/edit?usp=sharing

(a PDF-version is also available form the NorESM user wokshop repo
https://github.com/NorESMhub/INES_workshop_2020/blob/gh-pages/files/Torsvik_gitcollaborate.pdf )

If there is an interest, I could perhaps reformat this for the NorESM documentation (or maybe the link to the NorESM user workshop is enough?).