cosima / access-om3 Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 6.0 294 KB

ACCESS-OM3 global ocean-sea ice-wave coupled model

CMake 12.03% C 32.83% Fortran 54.28% Shell 0.16% Pascal 0.70%

access-om3's People

Contributors

Stargazers

Watchers

Forkers

aekiss dougiesquire anton-seaice ezhilsabareesh8 minghangli-uni

access-om3's Issues

Update with Dougie's config

I think the decision at yesterday's TWG meeting was to move @dougiesquire's CESM config here, is that right?

Then the ACCESS-NRI ocean/sea ice working group can start evaluating the output and we can get things moving forward.

automate updating dependency hashes in build.sh

build.sh currently requires dependency hashes to be manually updated if any dependencies (or their sub-dependencies) change - e.g. 0b84c2a

This is very error-prone. If the update is omitted or incorrect we can find ourselves unintentionally building with older dependencies than we intended.

Should we remove /g/data/${PROJECT}/${USER}/spack/share/spack/modules before building dependencies in build.sh?

In ACCESS-OM2 the first run has restart=false and ice_ic=‘default‘ so the sea ice and snow initial conditions are set in subroutine set_state_var in ice_init.F90 at high latitudes where SST is less than or equal to 1°C above freezing (NB: AusCOM is defined in the ACCESS-OM2 build). The initial SST is apparently read from monthly_sstsss.nc (e.g. in /g/data/ik11/inputs/access-om2/input_20201102/cice_1deg) in the AusCOM driver.

Both these steps involve some custom code in ACCESS-OM2. This raises some questions

how are initial conditions set up in CICE6?
do we want to replicate exactly the same initial condition in ACCESS-OM3, and if so, how?

I think it would be nice to replicate the ACCESS-OM2 IC so we have a closer comparison, but not if it involves a lot of custom code. Can this be achieved by a custom restart file somehow?

1deg_jra55do_ryf: Address MOM warnings

MOM throws a few warnings (see bellow) when running this configuration. We should check and eventually fix these warnings.

WARNING from PE     0: MEKE_init: Initializing MEKE with a local equilibrium balance.


WARNING from PE     0: KVML is a deprecated parameter. Use KV_ML_INVZ2 instead.


WARNING from PE     0: bkgnd_mixing_init: a nonzero constant background diffusivity (KD) is specified along with HORIZ_VARYING_BACKGROUND


WARNING from PE     0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.


WARNING from PE     0: Unused line in MOM_input : CFC_BC_FILE = cfc_atm_20230310.nc


WARNING from PE     0: Unused line in MOM_input : INT_TIDE_DECAY_SCALE = 300.3003003003003

Remove CICE mesh patch

Just noting that my PR fixing the CICE mesh mask check in the NUOPC cap has been merged into CICE-Consortium code. Once this fix is brought into the ESCOMP fork we're following, and we've updated CICE, we'll be able to remove the patch implemented in #63.

Port WOMBAT to MOM6

See discussion #6

Sea surface salinity restoring

We have sea surface salinity restoring in ACCESS-OM2 - see sec 3.5.2 of the tech report. I assume we'll need this in ACCESS-OM3, so we should work out how to do it in MOM6.

not bitwise reproducible

The current MOM6-CICE6 config (and presumably others) is not reproducible - compare these

/scratch/v45/aek156/access-om3/archive/MOM6-CICE6_ACCESS-OM3_repro_test_1
/scratch/v45/aek156/access-om3/archive/MOM6-CICE6_ACCESS-OM3_repro_test_2

Adopt CICE6 passive drifter code when available

An effort has recently begun to add passive drifter diagnostics to CICE6
CICE-Consortium/CICE#878

It would be useful to adopt this when available, e.g. for BGC tracer back-trajectories and Lagrangian data assimilation.

k-epsilon turbulence scheme in MOM6?

Alex Babanin is keen to have a k-epsilon turbulence scheme in ACCESS-OM3. This is available in MOM5 but not MOM6, so would need to be implemented. It's unclear whether this would require an unreasonably large amount of work.

It would not be possible to simply transplant the code from MOM5 to MOM6, since MOM6 internal numerics are entirely different from MOM5, and the MOM6 vertical coordinate is much more time-dependent.

If anyone has any thoughts on how large a task this might be, please add them here.

Configuration parameters set in multiple places

One of the discoveries from #81 is that some IO parameters are set in nuopc.runconfig but also could be set in ice_in and its not clear which is used.

I propose we delete the unused lines in either file. I think I have done this in ice_in through COSIMA/MOM6-CICE6#17 but there are lines to delete in nuopc.runconfig, including:

references to GLC model
MOM Parallel IO parameters (which are ignored by the MOM cap, MOM uses the netcdf library, not the Parallel IO library).
ice_nx = 360, ice_ny = 300

I suggest we take a conservative approach. Rather than audit every line, let delete the ones we know are not used and merge the change but leave this issue open. As more unused lines are discovered, we can make another change referencing this issue.

Parallel IO in CICE

In OM2 - a fair bit of work was done to add parrallel writing of netcdf output to get around delays writing daily out from CICE:

COSIMA/cice5#34

COSIMA/cice5@e9575cd

The ice_history code between CICE5 and 6 looks largely unchanged, so we will probably need to make similar changes to CICE6?

Add README

heat transfer coefficient for sea-ice basal heat flux

In OM2 - chio - the heat transfer coefficient for sea-ice basal heat flux was configurable. Although it was seeminly set to the default 0.006 in the end

To do this needed code changes, as this is hardcoded:

https://github.com/CICE-Consortium/Icepack/blob/6703bc533c96802235e2f20de5fffc0bc6cc4c97/columnphysics/icepack_therm_vertical.F90#L596C12-L596C12

And the CICE5 PR:
COSIMA/cice5#55

Do we want to be able to change this co-efficient?

Conservative vs potential temperature

My understanding is that a decision has to be made whether to use conservative or potential temperature as the prognostic temperature in MOM.

MOM6 T/S initialisation requires potential temperature, but I think ACCESS-OM2 uses conservative temperature.

Ping @aekiss to correct what I've got wrong and provide additional context/info.

Wave input parameter file in MOM6-CICE6-WWIII Config

The current MOM6-CICE6-Wave Watch III (WWIII) configuration doesn't have the crucial "ww3_multi.nml" input parameter file, which is essential for defining grid parameters (See Shou Li's configuration here). This file is needed for essential grid parameters for WWIII (see here). It is also unclear whether NUOPC sets default parameters when "ww3_multi.nml" is absent

CI reproducibility test

It would be good to have a CI reproducibility test, as this is fundamental to regression testing but isn't guaranteed #40

The test would be to cold-start two identical runs and see if the restarts are identical. The runs would need to be multi-processor but could otherwise be cheap (e.g. 1 day at 1° resolution), with restart saved at the end of each run.

Pre-compute mappings?

For performance we might want to pre-compute mappings rather than auto-generating them at runtime (see COSIMA/MOM6-CICE6#5) - but only if auto-generating consumes a lot of walltime.

Unclear whether this is currently possible with CDEPS; might require code changes.

No Output Files for WW3 in MOM6-CICE6-WW3 Configuration

The existing MOM6-CICE6-WW3 configuration is not producing any output files. Notably, the standard WW3 output is in the form of a binary file called out_grd. However, this binary file must be transformed into a NetCDF file using the ww3_ounf executable.

WW3 crash

Executables built with WW3 crash with an MPI error.

Spack libs shouldn't be in /scratch

My spack configuration is installing libraries into /scratch, which is obviously not a good idea if we want executables that link to them to work for more than 100 days.

Presumably this is due to these settings in /g/data/access/spack/modules/spack/config

  setenv SPACK_USER_CACHE_PATH /scratch/$project/$user/spack_user_cache
  setenv SPACK_INSTALL_TREE /scratch/$project/$user/opt/spack/

Exchange grids

Exchange grids (XGrids; see Balaji et al., 2006, Bauer et al., 2021 and Daniel Rosen's CW2023 talk and slides and Ufuk Turuncoglu's CW2023 talk and slides) look to be a good approach for calculating fluxes between model components on different grids, e.g. from DATM to MOM6 and CICE6, or between WW3 and MOM6/CICE6.

ESMF supports the regridding methods we'd likely want to use (patch and 1st order conservative) with XGrids http://earthsystemmodeling.org/regrid/

See #7 for a few more links and comments on exchange grids.

build with traceback

@micaeljtoliveira how do we build with traceback? it would be nice to have errors that are more helpful than this

forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
libpnetcdf.so.4.0  000014D5B5FADB2C  for__signal_handl     Unknown  Unknown
libpthread-2.28.s  000014D5B2B46CF0  Unknown               Unknown  Unknown
libpthread-2.28.s  000014D5B2B46180  nanosleep             Unknown  Unknown
libopen-rte.so.40  000014D5B1D8499A  orte_show_help_no     Unknown  Unknown
libopen-rte.so.40  000014D5B1D84CFC  orte_show_help        Unknown  Unknown
libmpi.so.40.30.4  000014D5B31A043A  PMPI_Abort            Unknown  Unknown
access-om3-20fa63  000000000082904B  Unknown               Unknown  Unknown
access-om3-20fa63  00000000010AEF93  Unknown               Unknown  Unknown
access-om3-20fa63  00000000007FA242  Unknown               Unknown  Unknown
access-om3-20fa63  00000000009A5758  Unknown               Unknown  Unknown
access-om3-20fa63  0000000000901C2A  Unknown               Unknown  Unknown
access-om3-20fa63  000000000043170C  MAIN__                    136  esmApp.F90
access-om3-20fa63  0000000000430D62  Unknown               Unknown  Unknown
libc-2.28.so       000014D5B27A8D85  __libc_start_main     Unknown  Unknown
access-om3-20fa63  0000000000430C6E  Unknown               Unknown  Unknown

Choosing model component versions

We should put some thought into how we choose component versions.

I suggest for initial testing we should match what @dougiesquire's CIME-built executables used, to see if we can resolve all issues like #21. When that is all working we should consider upgrading to newer versions.

Any comments/objections?

missing libs

I've built access-om3 here cd /g/data/v45/aek156/access-om3-build/access-om3 using 280ef02. (This required an interactive job with mem=32GB). But it the executable depends on pnetcdf and can't find it.

$ module load pnetcdf
$ /g/data/v45/aek156/access-om3-build/access-om3/build/access-om3
/g/data/v45/aek156/access-om3-build/access-om3/build/access-om3: error while loading shared libraries: libpnetcdf.so.4: cannot open shared object file: No such file or directory

$ ldd /g/data/v45/aek156/access-om3-build/access-om3/build/access-om3 | grep libpnetcdf
	libpnetcdf.so.4 => not found

Inconsistent physical constants

The default values for the latent heat of fusion and the latent heat of evaporation in MOM6 are taken from the corresponding values FMS defined here. On the other hand, when using the CESM coupler, CICE takes the default values from csm_share (see here and here).

Unfortunately these values are inconsistent. CESM solves the problem by replacing the FMS source file with a different one. This approach does not work well for us, as we are treating FMS as an external dependency. Instead, we can set the values in the input (e.g., using the LATENT_HEAT_FUSION and LATENT_HEAT_VAPORIZATION options of MOM6).

Config repos unsearchable / broken wiki links

GitHub only searches the default branch, so shifting the config content out of the main branch has made the configs unsearchable.

This has also broken a whole lot of links in the wiki (e.g. https://github.com/COSIMA/access-om3/wiki/Configurations), which were mostly GitHub search queries.

I'm not sure what to do about this. Being unable to search the branches via the UI is a real hindrance to understanding and documenting the configs.

Maybe the default branch should be something useable, e.g. 1deg_jra55do_ryf rather than just a README?

Sudden crashes with excessive SSH

MOM6-CICE6 1° configs are crashing after running for several weeks/months. Excessively large SSH appears in less than 1 day, without unusual wind stress - see
COSIMA/MOM6-CICE6#5 (comment)
COSIMA/MOM6-CICE6#5 (comment)

build and use esmf-extended-tests

esmf-extended-tests could be good to use for our configs (and maybe incorporate into CI), eg for checking new grids #36

Use grids rather than meshes?

Perhaps not a high priority, but at some point we may need to change from meshes to (auto-generated?) grids as discussed in COSIMA/MOM6-CICE6#5 and #36 (comment), because meshes are only supported in FMS1, not FMS2.

MOM6 equation of state

We should choose a suitable equation of state. ROQUET_RHO looks good - see discussion: COSIMA/mom6-panan#24

Try ADJUST_NET_SRESTORE_BY_SCALING = True

We should try out this MOM6 option, as suggested in #51 (comment)

ADJUST_NET_SRESTORE_BY_SCALING = True
				! "[Boolean] default = False
				! If true, adjustments to salt restoring to achieve zero net are
                                ! made by scaling values without moving the zero contour."

Can't patch files with the same name

I'd like to patch two files with the same name:

one in MOM6/config_src/infra/FMS1;
one in MOM6/config_src/infra/FMS2.

The current AddPatchedSource.cmake doesn't allow me to do this since it just looks for a patch named ${filename}.patch in the patches directory.

For now, I'll just patch the FMS1 directory, but this is something we might want to do in the future.

Use CICE6 C-grid

We should use the C-grid formulation in CICE6 when it is mature enough, to allow direct coupling with MOM6 on the same grid. See notes: #9

Support JRA55-do RYF

It appears CIME supports JRA RYF (not sure if this means JRA55-do - see #33) but DATM does not (yet).

ESCOMP/CDEPS#24

use ACCESS-OM2 grids, topography, etc

The ACCESS-OM3 configurations currently use an offset-pole 1° horizontal grid using these grid, mesh, topography and initial condition files.

We should change these to the tripolar grids used at each resolution in the ACCESS-OM2 suite as defined in /g/data/ik11/inputs/access-om2/input_20201102.

How to structure inputs directories

How do we want to structure the directories containing model input data on Gadi?

Different versions of the ACCESS-OM2 inputs include subdirectories for each component at each resolution - e.g. /g/data/ik11/inputs/access-om2/input_ff9890e2 has:

.
├── cice_01deg
├── cice_025deg
├── cice_1deg
├── common_01deg_jra55
├── common_025deg_jra55
├── common_1deg_core
├── common_1deg_jra55
├── minimal_mom_01deg
├── mom_01deg
├── mom_025deg
├── mom_1deg
├── oasis_core_to_1deg
├── yatm_01deg
├── yatm_025deg
└── yatm_1deg

Do we want to replicate this for ACCESS-OM3, or is something like the following sufficient:

.
├── 1deg # subdirectories for inputs specific to a resolution
└── share # inputs shared across configs

Incorrect time-stamp for MOM6 restart files

Seems like the MOM6 restart filename does not have the correct timestamp. Regardless of the actual simulation time, the file seems to always have seconds set to 0. This is probably not an issue if a calculation runs for more than one simulation day, but otherwise the code might try to write a new restart file with the same name as the previous restart file, causing an error.

tidy up drv_in

looks like the papi_inparm and prof_inparm groups in drv_in aren't read anywhere - can/should they be deleted from all the configs?

update quick start in wiki

https://github.com/COSIMA/access-om3/wiki/Quick-start
should be updated to be consistent with
https://github.com/COSIMA/MOM6-CICE6/blob/main/README.md

WW3 configuration

When we come to setting up the WW3 configuration we should see what was done with these:

@shuoli-code's WW3-MOM6-SIS2 coupled model https://github.com/shuoli-code/MOM6_WW3_SIS2_coupled
BoM's AUSWAVE-G3 http://www.bom.gov.au/research/publications/researchreports/BRR-051.pdf

update MOM_input

MOM_input needs to be updated in all configs.

I'll start with the 1deg_jra55do_ryf branch of MOM6-CICE6 and then migrate changes to other branches and repos.

The 1deg_jra55do_ryf branch of MOM6-CICE6 is currently closely based on the gmom_jra CESM branch; there have been several changes leading to the differences tabulated here.

This table compares the MOM6-CICE6 1deg_jra55do_ryf MOM_input to various other configurations (bold variables differ between the configs; this table shows only the differences).

Refine grid in Antarctic?

As mentioned in #36 (comment), we could consider refining the horizontal grid in the Antarctic and coarsening it in the Arctic.

Add simple github actions to test code compilation

This should be the starting point of our CI infrastructure.

mismatched CICE6 versions

I've tried to run with the accessom3_exe branch of @dougiesquire's MOM6-CICE6-WW3 repo
https://github.com/COSIMA/MOM6-CICE6-WW3/tree/139e1b44f8334cfa3e1c04579c9ea2e12e3d0087
which uses an executable /g/data/v45/aek156/access-om3-build/access-om3/bin/access-om3-a47478d built from this access-om3 repo rather than using CIME.

This failed with an error in CICE6

 ERROR: (abort_ice)(input_data)ERROR: tracer_nml reading
Image              PC                Routine            Line        Source
libpnetcdf.so.4.0  000014AD4B4295CB  tracebackqq_          Unknown  Unknown
access-om3-a47478  0000000002DD8E9D  shr_abort_mod_mp_          61  shr_abort_mod.F90
access-om3-a47478  000000000284D4A2  ice_exit_mp_abort          61  ice_exit.F90
access-om3-a47478  0000000002A4FF99  ice_init_mp_input         651  ice_init.F90
access-om3-a47478  000000000288EE42  cice_initmod_mp_c          53  CICE_InitMod.F90
access-om3-a47478  0000000002561D51  ice_comp_nuopc_mp         569  ice_comp_nuopc.F90
access-om3-a47478  00000000008E83F6  Unknown               Unknown  Unknown
...
access-om3-a47478  0000000000AB4F61  Unknown               Unknown  Unknown
access-om3-a47478  000000000043146F  MAIN__                    128  esmApp.F90
access-om3-a47478  0000000000430EA2  Unknown               Unknown  Unknown
libc-2.28.so       000014AD47C27D85  __libc_start_main     Unknown  Unknown
access-om3-a47478  0000000000430DAE  Unknown               Unknown  Unknown
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.
...

whereas the CIME-built exe /g/data/ik11/inputs/cime/bin/MOM6-CICE6-WW3/2023-03-10/cesm.exe works fine.

Use better regridding methods

datm.streams.xml and drof.streams.xml in the ACCESS-OM3 configurations currently use <mapalgo>bilinear</mapalgo> but there are better options - see here and here. Probably best to use conserve for fluxes and patch for everything else - this is what we did in ACCESS-OM2.

See https://github.com/COSIMA/access-om3/wiki/Configurations for details.

$ ncdump -h /g/data/ik11/inputs/cime/inputdata/2023-03-10/ocn/jra55/v1.3_noleap/JRA.v1.3.v_10.TL319.2007.171019.nc
...
// global attributes:
		:title = "v_10 -- JRA-55 based atmosphere data on TL319 grid" ;
		:Source = "Tsujino et al., 2017. JRA-55 Based Surface Dataset for Driving Ocean-Sea-Ice Models. Ocean Modell.\n",
			"Kobayashi et al., 2015. The JRA-55 Reanalysis. J. Met. Soc. Jap., 93, 5-48 " ;
		:Conventions = "CF1.0" ;
		:history = "Original file v_10.2007.18Oct2017.nc reformatted by whokim Thu Oct 19 23:59:42 MDT 2017" ;
		:notes = "JRA-55 TL319 data (v1.3, 2017 Oct 18) reformatted for use in CESM, removed all Feb 29\\\'s, calendar now noleap" ;

DATM says it supports "JRA-55 v1.3 forcing data"
https://escomp.github.io/CDEPS/versions/master/html/datm.html#supported-data-modes

CORE_IAF_JRA (datm_datamode_jra_mod.F90)
In conjunction with JRA-55 Project, provides the atmosphere forcing when coupling an active ocean model with observed atmospheric forcing. This mode and associated data sets implement the JRA-55 v1.3 forcing data.

It's unclear whether DATM would also support a newer -do version. More investigation needed.