esmci/cime

Common Infrastructure for Modeling the Earth

cime

Common Infrastructure for Modeling the Earth

CIME, pronounced "SEAM", primarily consists of a Case Control System that supports the configuration, compilation, execution, system testing and unit testing of an Earth System Model. The two main components of the Case Control System are:

1. Scripts to enable simple generation of model executables and associated input files for different scientific cases, component resolutions and combinations of full, data and stub components with a handful of commands.
2. Testing utilities to run defined system tests and report results for different configurations of the coupled system.

CIME does not contain the source code for any Earth System Model drivers or components. It is typically included alongside the source code of a host model. However, CIME does include pointers to external repositories that contain drivers, data models and other test components. These external components can be easily assembled to facilitate end-to-end system tests of the CIME infrastructure, which are defined in the CIME repository.

CIME is currently used by the Community Earth System Model (CESM) and the Energy Exascale Earth System Model (E3SM).

Documentation

See esmci.github.io/cime

Developers

Lead Developers

Jim Edwards (NCAR), Jim Foucar (SNL)

Also Developed by

Alice Bertini (NCAR), Jason Boutte (LLNL), Tony Craig (NCAR), Michael Deakin (SNL), Chris Fischer (NCAR), Erich Foster (SNL), Steve Goldhaber (NCAR), Robert Jacob (ANL), Mike Levy (NCAR), Bill Sacks (NCAR), Andrew Salinger (SNL), Sean Santos (NCAR), Jason Sarich (ANL), Mariana Vertenstein (NCAR), Andreas Wilke (ANL).

Acknowledgements

CIME is jointly developed with support from the Earth System Modeling program of DOE's BER office and the CESM program of NSF's Division of Atmospheric and Geospace Sciences.

License

CIME is free software made available under the BSD License. For details see the LICENSE file.

Digital Object Identifier

DOI:10.5065/WE0D-9K91

cime's Issues

cmake build for cprnc is broken with cime0_2_1

The cmake build for cprnc is broken in cime0_2_1and probably later:

build $ make all
Scanning dependencies of target COMPARE_VARS
[ 14%] Generating compare_vars_mod.F90
Can't open perl script "/Users/xyz/cime/tools/cprnc/../../genf90/genf90.pl": No such file or directory
make[2]: *** [compare_vars_mod.F90] Error 2
make[1]: *** [CMakeFiles/COMPARE_VARS.dir/all] Error 2
make: *** [all] Error 2

The fix is:

diff --git a/cime/tools/cprnc/CMakeLists.txt b/cime/tools/cprnc/CMakeLists.txt
index 85143ba..237de4b 100644
--- a/cime/tools/cprnc/CMakeLists.txt
+++ b/cime/tools/cprnc/CMakeLists.txt
@@ -16,7 +16,7 @@ ENDIF ()

 ADD_CUSTOM_COMMAND(
   OUTPUT ${PROJECT_BINARY_DIR}/compare_vars_mod.F90
-  COMMAND perl ${PROJECT_SOURCE_DIR}/../../genf90/genf90.pl
+  COMMAND perl ${PROJECT_SOURCE_DIR}/../../externals/genf90/genf90.pl
     ${PROJECT_SOURCE_DIR}/compare_vars_mod.F90.in > ${PROJECT_BINARY_DIR}/compare_vars_mod.F90
 )

Can't submit case for X case on blues

Got a bunch of error messages.
create_newcase -case Xcase2 -compset X -res f19_g16 -model acme -mach blues -compiler intel -project ACME
./case.setup
./case.build all works.

But then ./case.submit ends with:

qsub: option requires an argument -- 'q'
Use of uninitialized value $output in scalar chomp at
/lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 220 (#1)
(W uninitialized) An undefined value was used as if it were already
defined. It was interpreted as a "" or a 0, but maybe it was a mistake.
To suppress this warning assign a defined value to your variables.

To help you figure out what was undefined, perl will try to tell you the
name of the variable (if any) that was undefined. In some cases it cannot
do this, so it also tells you what operation you used the undefined value
in.  Note, however, that perl optimizes your program and the operation
displayed in the warning may not necessarily appear literally in your
program.  For example, "that $foo" is usually optimized into "that "
. $foo, and the warning will refer to the concatenation (.) operator,
even though there is no . in your program.

Use of uninitialized value $jobstring in scalar chomp at
/lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 121 (#1)
Use of uninitialized value $jobstring in pattern match (m//) at
/lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 146 (#1)
could not ascertain dependent job id... aborting
Uncaught exception from user code:
could not ascertain dependent job id... aborting at /lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 152
at /lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 152
Batch::BatchUtils::getJobID('Batch::BatchUtils=HASH(0x1518108)', undef) called at /lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 221
Batch::BatchUtils::submitSingleJob('Batch::BatchUtils=HASH(0x1518108)', 'case.run', undef, 1) called at /lcrc/group/earthscience/jacob/cimeESMCI/utils/perl5lib/Batch/BatchUtils.pm line 177
Batch::BatchUtils::submitJobs('Batch::BatchUtils=HASH(0x1518108)') called at ./case.submit line 118
main::main() called at ./case.submit line 156
Exit 255
blogin1[79]:

Full output of case.submit at https://gist.github.com/rljacob/d9509877446b35c9850cdd4eecf685cf

Deleted and recreated tags in repo

Due to a series of operations with a detached head (have fun with that pun) I found it necessary to delete and recreate a number of tags in the repository. This may have a bad affect on forks created since 3/21/15. I apologize for the inconvenience, I think it's all cleaned up now. - Jim

Rewrite template.lt_archive in python

Avoid creating a case directory until all input options are validated.

The create_newcase command (and other scripts, e.g. create_test) should not create directories or write any files until first validating user-specified options. If someone forgets to add the -compset option, for instance, the directory is created, but the command fails, meaning that you have to go wipe it before trying again.

Python coding convention(s)

What coding convention should CIME python code use?

Setup desktop for CIME testing at NCAR

We have been utilizing desktop linux machine for ACME called "melvin" for testing smaller ACME testcases and testing scripts/infrastructure. I cannot emphasize enough how useful this machine has been for ACME by enabling quick turnaround for a significant set of our testing needs. The advantages of running on a local file system (lightning fast git operations) and not having to wait in queues have been enormously valuable.

I recommend that such a machine be set up at NCAR to enable fast, convenient testing of our shared infrastructure. Bonus points if I can get an account on this machine. I think this will be a lot more efficient than trying to do testing on some cluster that we both support.

Conflict characters left over

A few files still have conflict characters in them. May need to re-merge?

blogin3blogin3[227]: grep -lr '<<<<<<<' .
./driver_cpl/driver/seq_hist_mod.F90
./driver_cpl/shr/seq_flds_mod.F90
./scripts/Testing/Testcases/config_tests.xml
./scripts/Tools/case.build
./scripts/create_newcase
./share/csm_share/shr/shr_pio_mod.F90
./share/csm_share/test/old_unit_testers/run_file_test
./share/csm_share/test/old_unit_testers/test_shr_file.F90

blogin3[228]: grep -lr '>>>>>>' .
./externals/pio/examples/basic/perl5lib/ChangeLog

Rewrite system tests in python

From the old google doc:

1. Create a utils/python/CIME/SystemTests directory, test types will each be represented by a Class in this directory
2. write a template.case.test in python that runs the selected test (partially done)
3. case.test_build (all tests are now currently built with python, but the test scripts themselves are still csh)
4. move csh test scripts to python
5. Have tests output to teststatus.xml file in each case directory replacing the current TestStatus file
6. Have cs_status collect these teststatus.xml files into a report to cdash or the cesm testdb

New NetCDF distribution and CIME

As some of you know, NetCDF now makes the C and Fortran implementations of NetCDF available as separate packages.

For most sysadmins, this means they could be installed with completely different paths: "NetCDF C path"/lib and "NetCDF-Fortran-path"/lib. We've already seen this on Blues and its a reasonable thing to do.

The CIME build assumes the C and Fortran libs (or includes) are all in the same "lib" (or "include") directory. Should we change that assumption?

Review the usage of LockedFiles

In tag #130 the usage of LockedFiles may be compromised, we should review it and make sure files are adequately protected from compromise.

Rework logic for CPL_ALBAV xml variable

This is related to issue #119: The logic for CPL_ALBAV in driver_cpl/cime_config/config_component.xml currently looks like this:

  <entry id="CPL_ALBAV">
    <type>logical</type>
    <valid_values>true,false</valid_values>
    <default_value>false</default_value>
    <values>
      <value compset="DATM.+POP">true</value>
      <value compset="DATM.+DOCN%IAF">true</value>
    </values>
    <group>run_component_cpl</group>
    <file>env_run.xml</file>
    <desc>
      Only used for compsets with DATM and POP (currently C and G):
      If true, compute albedos to work with daily avg SW down
      If false (default), albedos are computed with the assumption that downward
      solar radiation from the atm component has a diurnal cycle and zenith-angle
      dependence. This is normally the case when using an active atm component
      If true, albedos are computed with the assumption that downward
      solar radiation from the atm component is a daily average quantity and
      does not have a zenith-angle dependence. This is often the case when
      using a data atm component. Only used for compsets with DATM and POP (currently C and G)
    </desc>
  </entry>

Really, it seems that this shouldn't depend on the compset, but rather should depend on some property of the atmospheric forcing. So this should be set by the atmosphere component, either at build-namelist time or at runtime. It seems possible to set this in datm's config_component.xml. But it seems like, to really make this robust and avoid duplicated logic (which is likely to become incorrect over time), it might need to be set at runtime, by the atmosphere component setting some flag (e.g., in infodata??).

Rewrite get_timing in python

Resolve merge issues with case.build

The merge of scripts/Tools/case.build needs to be resolved. The cime_developer test suite won't be runnable on melvin until this is done.

Failed to build A case with create_newcase on edison

Steps:

Checkout out latest master dfb98f0
setenv CIME_MODEL acme (OR cesm)
module load python
create_newcase -case Acase -res f45_g37_rx1 -compset A -mach edison -compiler intel -project acme
cd Acase
./case.setup

Everything above works. Then I tried the next step...

edison09 scripts/Acase> ./case.build
calling build.case_build with caseroot=/global/u2/r/rjacob/cimeESMCI/scripts/Acase and testmode=False
sharedlib_only is False
model_only is False
Not able to retreive value for item 'TESTCASE'
Loading input file: 'Buildconf/datm.input_data_list'
Loading input file: 'Buildconf/dlnd.input_data_list'
Loading input file: 'Buildconf/dice.input_data_list'
Loading input file: 'Buildconf/docn.input_data_list'
Loading input file: 'Buildconf/drof.input_data_list'
Loading input file: 'Buildconf/cpl.input_data_list'
Not able to retreive value for item 'CISM_USE_TRILINOS'
Not able to retreive value for item 'CLM_CONFIG_OPTS'
Not able to retreive value for item 'CAM_CONFIG_OPTS'
Not able to retreive value for item 'OCN_SUBMODEL'
ERROR: Could not find cmd_path for lang 'python'

version(s) of timing library in CIME

Based on the https://github.com/ESMCI/cime branch, it appears that the
decision has been made to use only the ACME version of the timing
library in the shared CIME version. I would rather that both versions be
available, if possible.

I need the ACME version to do my current job. As part of this I also
need that everyone else in ACME use it. Finally, I need to be able to
continue to modify the timing library as necessary for the needs of the
project, and then push this out to others in the project. However, this
is all very project specific, and seems inappropriate for the
coordinated management being discussed and impolite to impose on the
CESM and others.

For example, my latest "additions" were just conveniences, an option to
measure timing overhead directly (adding yet more overhead) and an
option to add the detail level to timer names so that I could determine
how to modify t_adj_detail calls in the code, neither of which will ever
be used during production runs. I made the mistake in the first case of
adding this option to the env_run.xml file, which introduces a conflict
with the CESM version of the timing library, but this is completely
unnecessary. Both options can simply be included in user_nl_cpl when
needed.

So, other than this one example, I have been managing to keep the two
APIs the same. And the differences in the two GetTiming scripts is very
small, and maintaining both versions should not be difficult.

To repeat, I would be much more comfortable if we could support both the
latest and greatest Jim Rosinski-based timing library and the current,
still evolving, ACME timing library. Is this possible without adding
extra work? Coordination would then simply be at the level of
maintaining compatible APIs.

This all assumes that CSEG has been happy with using the more
recent versions of GPTL, and with treating GPTL as an external library.

Fix up bad markdown in the old ChangeLog copies in README files.

Certain files render very poorly because they have copies of old change logs that were written pre-markdown. I can probably do this with a few sed commands.

Shouldn't need to list all grid combinations explicitly in config_grids.xml

My current use case is: I want to say, "For every allowed atm-lnd-ocn-rof grid combination, allow glc grids of either gl4, gl5 or gl20." But actually expressing that in config_grids.xml requires a combinatorial explosion of entries.

I am okay (I guess) needing to explicitly list out the available mapping files for pairs of grids. It's the entries like this that are causing me grief: The original entry is:

    <grid>
      <sname>1.9x2.5_gx1v6</sname>
      <alias>f19_g16</alias>
      <lname>a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null</lname>
    </grid>

and then I need to add the following:

    <grid compset="_CISM">
      <sname>1.9x2.5_gx1v6_gland4</sname>
      <alias>f19_g16_gl4</alias>
      <lname>a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%gland4_w%null</lname>
    </grid>
    <grid compset="_CISM">
      <sname>1.9x2.5_gx1v6_gl5</sname>
      <alias>f19_g16_gl5</alias>
      <lname>a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%gland5UM_w%null</lname>
    </grid>
    <grid compset="_CISM">
      <sname>1.9x2.5_gx1v6_gland20</sname>
      <alias>f19_g16_gl20</alias>
      <lname>a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%gland20_w%null</lname>
    </grid>

I need to do this for many such original entries. This incurs a painful up-front cost, as well as a long-term maintenance cost -- since the more duplication we have like this, the more likely it is that some entries are going to be incorrect from the beginning, or else are going to get out of sync (e.g., if we change the default runoff grid, as just one example): at least for me, my eyes glaze over when trying to review so many nearly-identical entries, so it's very hard to notice problems.

Keep 2 versions of pio

We're not done exploring the performance between pio1 and pio2. So we'd like to create:
cime/externals/pio1 - ACME's current version of pio1.
cime/externals/pio2 - CIME4's latest version of pio2.

BatchMaker and "odd number of tasks".

Something is wrong in BatchMaker.pm in ACME's current version. Not sure if this is a problem in latest CIME but copying this issue here just in case.

On Edison, if you
./create_newcase -case XXX -res ne30_m120 -compset A_B1850CN -mach edison -compiler intel -project acme

and then

 ./cesm_setup

the setup works fine. However, if you change 960 to 600 in the env_mach_pes.xml file, then you get the error

 odd number of tasks to handle at 

/cime/utils/perl5lib/Batch/BatchMaker.pm line 905.

This goes away if you also change

 <entry id="MAX_TASKS_PER_NODE"   value="48"  />

to

But 600 is a perfectly legal number of processes to use, and this should not break. There were issues with not filling nodes completely, but with 2 or 4 threads, 600 fills nodes exactly even with 48 threads per node.

Organization of python code within merged CIME

Suggestion:
create cime/utils/python and move the *.py files from scripts-acme to there.
rename scripts-acme to scripts-python for further development.

Get ACME internal framework tests running

Loading software environment on blues

We were having trouble running on blues. It worked fine if you did "qsub $CASE.run" but a user would get library path errors if they executed $CASE.submit.

@jayeshkrishna traced it to the way CIME loads the machine specific environment in the job submission scripts. This bug is repeated here for further discussion and to check the solution.

dabail10

Can you add me to the CSEG group? My userid is dabail10 (not dbailey).

Dave

compset matches in driver's config_component should be made more general

Moving issue CESM-Development#402 to here, in the comments that follow.

Implement new utility code and coding standards for timing.

I have proposed the changes explained here for CESM. While the driver code largely follows these standards already, some changes are needed in perf_mod to make this approach more convenient.

Rewrite xmlquery in python

Decide on documentation method for CIME

How shall we do the documentation for CIME? I'd like to start writing some.

A few requirements: Hosted at a neutral site (like this github). Writable by any CIME developer (hello github). A living document that can be updated in realtime (ditto) Can be exported to a pdf for publication as a technical note.

For that last one, there are apparently scripts that will covert a github wiki to a pdf.
http://stackoverflow.com/questions/18759738/effortless-export-from-github-wiki

Remove confopts from create_newcase

Rewrite create_newcase in python

The big one.

Problem with pio_rearr_comm_enable_hs

Cases created by the python create_test are dying in case.setup due to this error:

% /home/jgfouca/cime_unified/driver_cpl/bld/build-namelist -infile /home/jgfouca/acme/scratch/TESTBUILDFAIL.f19_g16_rx1.A.melvin_gnu.doesnt_work/Buildconf/cplconf/namelist -caseroot /home/jgfouca/acme/scratch/TESTBUILDFAIL.f19_g16_rx1.A.melvin_gnu.doesnt_wo\
rk -cimeroot /home/jgfouca/cime_unified -grid a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%rx1_m%gx1v6_g%null_w%null -rof_grid rx1 -atm_grid 1.9x2.5  -lnd_grid 1.9x2.5 -ocn_grid gx1v6 -wav_grid null

build-namelist - ERROR: No default value found for pio_rearr_comm_enable_hs
user defined attributes:
key=PIO_REARR_COMM_ENABLE_HS  val=

In ACME's CIME, I see PIO_REARR_COMM_ENABLE_HS defined in env_run.xml, but with the shared CIME, the created case doesn't have that entry anywhere.

Any ideas? I'm dead in the water until I get past this.

Basic CIME testcases need to build successfully

The tests are these:

                            "NCK_Ld3.f45_g37_rx1.A",
                             "ERI_D.f45_g37_rx1.A",
                             "SEQ_PFC_Ld3.f09_g16.X",
                             "ERS_Ld3.ne30_g16_rx1.A",
                             "ERS_N2_Ld3.f19_g16_rx1.A",
                             "ERS_Ld3.f45_g37_rx1.A"

case.setup exception statements

I've found the cause of the exception statements from Rob's demo in cime school today (3/25):
File "./xmlchange", line 125, in <module> _main_func(__doc__) File "./xmlchange", line 122, in _main_func xmlchange(caseroot, listofsettings, xmlfile, xmlid, xmlval, subgroup, append, noecho, warn, force) File "./xmlchange", line 105, in xmlchange case.set_value(xmlid, xmlval, subgroup, ignore_type=force) File "/blues/gpfs/home/sarich/cime4/scripts/Tools/../../utils/python/CIME/case.py", line 97, in set_value result = env_file.set_value(item, value, subgroup, ignore_type) File "/blues/gpfs/home/sarich/cime4/scripts/Tools/../../utils/python/CIME/XML/env_batch.py", line 31, in set_value val = EnvBase.set_value(self, vnode, value, ignore_type) File "/blues/gpfs/home/sarich/cime4/scripts/Tools/../../utils/python/CIME/XML/entry_id.py", line 75, in set_value node = self.get_optional_node("entry", {"id":vid}) File "/blues/gpfs/home/sarich/cime4/scripts/Tools/../../utils/python/CIME/XML/generic_xml.py", line 90, in get_optional_node nodes = self.get_nodes(nodename, attributes, root) File "/blues/gpfs/home/sarich/cime4/scripts/Tools/../../utils/python/CIME/XML/generic_xml.py", line 110, in get_nodes newnodes = root.findall(xpath) File "/soft/python/2.7.3/lib/python2.7/xml/etree/ElementTree.py", line 390, in findall return ElementPath.findall(self, path, namespaces) File "/soft/python/2.7.3/lib/python2.7/xml/etree/ElementPath.py", line 293, in findall return list(iterfind(elem, path, namespaces)) File "/soft/python/2.7.3/lib/python2.7/xml/etree/ElementPath.py", line 263, in iterfind selector.append(ops[token[0]](next, token)) File "/soft/python/2.7.3/lib/python2.7/xml/etree/ElementPath.py", line 224, in prepare_predicate raise SyntaxError("invalid predicate")
BatchMaker.pm::setWallTime() is calling
$self->{envBatch}->set("JOB_WALLCLOCK_TIME",$self->{job},$self->{wall_time});
where job='run'

However, the env_batch.xml file has the entry:
<entry id="JOB_WALLCLOCK_TIME" value=""> <type>char</type> <group>job_submission</group> <desc>The machine wallclock setting. Default determined in config_machines.xml can be overwritten by testing</desc> </entry>
Since the xmlchange python script is looking to match 'run' against the group 'job_submission', an exception is getting raised.

I'm not sure how to fix this, hoping somebody else has an idea

Rewrite case.setup in python

Rewrite template.case.run in python

Rewrite manage_case in python

Creating cime_config/acme

(coverted this issue from omnibus to single issue)

machines changes.
delete cime/machines
git mv cime/machines-acme to cime/cime_config/acme/machines and update acme's create_test (which still works) as needed.

Rewrite case.submit in python

cs.status is broken in cime standalone

Running the cs.status script in a cime standalone environment results in the error:
Global symbol "$ccsmroot" requires explicit package name at ./cs.status.150408-131201.mira line 37.
BEGIN not safe after errors--compilation aborted at ./cs.status.150408-131201.mira line 46.

timer changes will need to be added back into template.case.run

The changes that were made on the acme side were very minimal and can be viewed as follows:

% git log  -w -p cime/machines-acme/template.cesmrun

There were massive changes to this file on the CESG side, so I just copied their version for now.

ACME and CIME4 grid configs are not compatible

cime_config/acme/config_grids.xml
and
cime_config/cesm/config_grids.xml
... are not in the same format. The changes made on the ACME side will need to be examined and converted to the new format.

Rewrite manage_pes in python

create_newcase needs to be merged

Most of the ACME testing will fail until create_newcase is fixed.

Debug ./scripts_regression_tests on blues

With latest master fd9b1ee getting:
FAILED (failures=5, errors=1, skipped=3)

More info
https://gist.github.com/rljacob/3663fd92460d6469c044

Fix --no-batch argument to create_test

the --no-batch argument to create_test does not appear to work on a machine with batch queues.

How to handle component threads within CIME

I asked this question last week during a telecon, but wanted to ask it again here just to document the answer.

How are threads handled on a per-component basis within CIME?

Does the driver set the correct number of threads? or are component expected to set the number of threads through some pre-defined information (i.e. whatever exists in env_mach_pes.xml)?

Also, what is the preferred way to get / set the number of threads a component should be using?

Rewrite template.st_archive in python

Merge ACME env_mach_specific files for blues and titan into config_machines.xml

The files that used to define the environment for a machine, env_mach_specific.machine, have been removed on the CESG side of CIME. All the ACME env_mach_specific files will need to be removed and their contents ported into config_machines.xml.

Replace cime cs_status with cimeteststatus

Batch-related perl modules will need examination

The files:
utils/perl5lib/Batch/BatchMaker.pm
utils/perl5lib/Batch/BatchUtils.pm
will need to be closely examined and tested as they were full of conflicts. @jayeshkrishna made most of the changes on our side.