cosima / access-om3 Goto Github PK
View Code? Open in Web Editor NEWACCESS-OM3 global ocean-sea ice-wave coupled model
ACCESS-OM3 global ocean-sea ice-wave coupled model
I think the decision at yesterday's TWG meeting was to move @dougiesquire's CESM config here, is that right?
Then the ACCESS-NRI ocean/sea ice working group can start evaluating the output and we can get things moving forward.
build.sh
currently requires dependency hashes to be manually updated if any dependencies (or their sub-dependencies) change - e.g. 0b84c2a
This is very error-prone. If the update is omitted or incorrect we can find ourselves unintentionally building with older dependencies than we intended.
Should we remove /g/data/${PROJECT}/${USER}/spack/share/spack/modules
before building dependencies in build.sh
?
In ACCESS-OM2 the first run has restart
=false and ice_ic
=‘default‘ so the sea ice and snow initial conditions are set in subroutine set_state_var in ice_init.F90
at high latitudes where SST is less than or equal to 1°C above freezing (NB: AusCOM
is defined in the ACCESS-OM2 build). The initial SST is apparently read from monthly_sstsss.nc
(e.g. in /g/data/ik11/inputs/access-om2/input_20201102/cice_1deg
) in the AusCOM driver.
Both these steps involve some custom code in ACCESS-OM2. This raises some questions
I think it would be nice to replicate the ACCESS-OM2 IC so we have a closer comparison, but not if it involves a lot of custom code. Can this be achieved by a custom restart file somehow?
MOM throws a few warnings (see bellow) when running this configuration. We should check and eventually fix these warnings.
WARNING from PE 0: MEKE_init: Initializing MEKE with a local equilibrium balance.
WARNING from PE 0: KVML is a deprecated parameter. Use KV_ML_INVZ2 instead.
WARNING from PE 0: bkgnd_mixing_init: a nonzero constant background diffusivity (KD) is specified along with HORIZ_VARYING_BACKGROUND
WARNING from PE 0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.
WARNING from PE 0: Unused line in MOM_input : CFC_BC_FILE = cfc_atm_20230310.nc
WARNING from PE 0: Unused line in MOM_input : INT_TIDE_DECAY_SCALE = 300.3003003003003
Just noting that my PR fixing the CICE mesh mask check in the NUOPC cap has been merged into CICE-Consortium code. Once this fix is brought into the ESCOMP fork we're following, and we've updated CICE, we'll be able to remove the patch implemented in #63.
See discussion #6
We have sea surface salinity restoring in ACCESS-OM2 - see sec 3.5.2 of the tech report. I assume we'll need this in ACCESS-OM3, so we should work out how to do it in MOM6.
The current MOM6-CICE6 config (and presumably others) is not reproducible - compare these
/scratch/v45/aek156/access-om3/archive/MOM6-CICE6_ACCESS-OM3_repro_test_1
/scratch/v45/aek156/access-om3/archive/MOM6-CICE6_ACCESS-OM3_repro_test_2
An effort has recently begun to add passive drifter diagnostics to CICE6
CICE-Consortium/CICE#878
It would be useful to adopt this when available, e.g. for BGC tracer back-trajectories and Lagrangian data assimilation.
Alex Babanin is keen to have a k-epsilon turbulence scheme in ACCESS-OM3. This is available in MOM5 but not MOM6, so would need to be implemented. It's unclear whether this would require an unreasonably large amount of work.
It would not be possible to simply transplant the code from MOM5 to MOM6, since MOM6 internal numerics are entirely different from MOM5, and the MOM6 vertical coordinate is much more time-dependent.
If anyone has any thoughts on how large a task this might be, please add them here.
One of the discoveries from #81 is that some IO parameters are set in nuopc.runconfig
but also could be set in ice_in
and its not clear which is used.
I propose we delete the unused lines in either file. I think I have done this in ice_in
through COSIMA/MOM6-CICE6#17 but there are lines to delete in nuopc.runconfig
, including:
ice_nx = 360
, ice_ny = 300
I suggest we take a conservative approach. Rather than audit every line, let delete the ones we know are not used and merge the change but leave this issue open. As more unused lines are discovered, we can make another change referencing this issue.
In OM2 - a fair bit of work was done to add parrallel writing of netcdf output to get around delays writing daily out from CICE:
The ice_history code between CICE5 and 6 looks largely unchanged, so we will probably need to make similar changes to CICE6?
In OM2 - chio
- the heat transfer coefficient for sea-ice basal heat flux was configurable. Although it was seeminly set to the default 0.006 in the end
To do this needed code changes, as this is hardcoded:
And the CICE5 PR:
COSIMA/cice5#55
Do we want to be able to change this co-efficient?
My understanding is that a decision has to be made whether to use conservative or potential temperature as the prognostic temperature in MOM.
MOM6 T/S initialisation requires potential temperature, but I think ACCESS-OM2 uses conservative temperature.
Ping @aekiss to correct what I've got wrong and provide additional context/info.
The current MOM6-CICE6-Wave Watch III (WWIII) configuration doesn't have the crucial "ww3_multi.nml" input parameter file, which is essential for defining grid parameters (See Shou Li's configuration here). This file is needed for essential grid parameters for WWIII (see here). It is also unclear whether NUOPC sets default parameters when "ww3_multi.nml" is absent
It would be good to have a CI reproducibility test, as this is fundamental to regression testing but isn't guaranteed #40
The test would be to cold-start two identical runs and see if the restarts are identical. The runs would need to be multi-processor but could otherwise be cheap (e.g. 1 day at 1° resolution), with restart saved at the end of each run.
For performance we might want to pre-compute mappings rather than auto-generating them at runtime (see COSIMA/MOM6-CICE6#5) - but only if auto-generating consumes a lot of walltime.
Unclear whether this is currently possible with CDEPS; might require code changes.
The existing MOM6-CICE6-WW3 configuration is not producing any output files. Notably, the standard WW3 output is in the form of a binary file called out_grd
. However, this binary file must be transformed into a NetCDF file using the ww3_ounf
executable.
Executables built with WW3 crash with an MPI error.
My spack configuration is installing libraries into /scratch
, which is obviously not a good idea if we want executables that link to them to work for more than 100 days.
Presumably this is due to these settings in /g/data/access/spack/modules/spack/config
setenv SPACK_USER_CACHE_PATH /scratch/$project/$user/spack_user_cache
setenv SPACK_INSTALL_TREE /scratch/$project/$user/opt/spack/
Exchange grids (XGrids; see Balaji et al., 2006, Bauer et al., 2021 and Daniel Rosen's CW2023 talk and slides and Ufuk Turuncoglu's CW2023 talk and slides) look to be a good approach for calculating fluxes between model components on different grids, e.g. from DATM to MOM6 and CICE6, or between WW3 and MOM6/CICE6.
ESMF supports the regridding methods we'd likely want to use (patch and 1st order conservative) with XGrids http://earthsystemmodeling.org/regrid/
See #7 for a few more links and comments on exchange grids.
@micaeljtoliveira how do we build with traceback? it would be nice to have errors that are more helpful than this
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpnetcdf.so.4.0 000014D5B5FADB2C for__signal_handl Unknown Unknown
libpthread-2.28.s 000014D5B2B46CF0 Unknown Unknown Unknown
libpthread-2.28.s 000014D5B2B46180 nanosleep Unknown Unknown
libopen-rte.so.40 000014D5B1D8499A orte_show_help_no Unknown Unknown
libopen-rte.so.40 000014D5B1D84CFC orte_show_help Unknown Unknown
libmpi.so.40.30.4 000014D5B31A043A PMPI_Abort Unknown Unknown
access-om3-20fa63 000000000082904B Unknown Unknown Unknown
access-om3-20fa63 00000000010AEF93 Unknown Unknown Unknown
access-om3-20fa63 00000000007FA242 Unknown Unknown Unknown
access-om3-20fa63 00000000009A5758 Unknown Unknown Unknown
access-om3-20fa63 0000000000901C2A Unknown Unknown Unknown
access-om3-20fa63 000000000043170C MAIN__ 136 esmApp.F90
access-om3-20fa63 0000000000430D62 Unknown Unknown Unknown
libc-2.28.so 000014D5B27A8D85 __libc_start_main Unknown Unknown
access-om3-20fa63 0000000000430C6E Unknown Unknown Unknown
We should put some thought into how we choose component versions.
I suggest for initial testing we should match what @dougiesquire's CIME-built executables used, to see if we can resolve all issues like #21. When that is all working we should consider upgrading to newer versions.
Any comments/objections?
I've built access-om3 here cd /g/data/v45/aek156/access-om3-build/access-om3
using 280ef02. (This required an interactive job with mem=32GB
). But it the executable depends on pnetcdf
and can't find it.
$ module load pnetcdf
$ /g/data/v45/aek156/access-om3-build/access-om3/build/access-om3
/g/data/v45/aek156/access-om3-build/access-om3/build/access-om3: error while loading shared libraries: libpnetcdf.so.4: cannot open shared object file: No such file or directory
$ ldd /g/data/v45/aek156/access-om3-build/access-om3/build/access-om3 | grep libpnetcdf
libpnetcdf.so.4 => not found
The default values for the latent heat of fusion and the latent heat of evaporation in MOM6 are taken from the corresponding values FMS defined here. On the other hand, when using the CESM coupler, CICE takes the default values from csm_share
(see here and here).
Unfortunately these values are inconsistent. CESM solves the problem by replacing the FMS source file with a different one. This approach does not work well for us, as we are treating FMS as an external dependency. Instead, we can set the values in the input (e.g., using the LATENT_HEAT_FUSION
and LATENT_HEAT_VAPORIZATION
options of MOM6).
GitHub only searches the default branch, so shifting the config content out of the main branch has made the configs unsearchable.
This has also broken a whole lot of links in the wiki (e.g. https://github.com/COSIMA/access-om3/wiki/Configurations), which were mostly GitHub search queries.
I'm not sure what to do about this. Being unable to search the branches via the UI is a real hindrance to understanding and documenting the configs.
Maybe the default branch should be something useable, e.g. 1deg_jra55do_ryf
rather than just a README?
MOM6-CICE6 1° configs are crashing after running for several weeks/months. Excessively large SSH appears in less than 1 day, without unusual wind stress - see
COSIMA/MOM6-CICE6#5 (comment)
COSIMA/MOM6-CICE6#5 (comment)
esmf-extended-tests could be good to use for our configs (and maybe incorporate into CI), eg for checking new grids #36
Perhaps not a high priority, but at some point we may need to change from meshes to (auto-generated?) grids as discussed in COSIMA/MOM6-CICE6#5 and #36 (comment), because meshes are only supported in FMS1, not FMS2.
We should choose a suitable equation of state. ROQUET_RHO
looks good - see discussion: COSIMA/mom6-panan#24
We should try out this MOM6 option, as suggested in #51 (comment)
ADJUST_NET_SRESTORE_BY_SCALING = True
! "[Boolean] default = False
! If true, adjustments to salt restoring to achieve zero net are
! made by scaling values without moving the zero contour."
I'd like to patch two files with the same name:
MOM6/config_src/infra/FMS1
;MOM6/config_src/infra/FMS2
.The current AddPatchedSource.cmake
doesn't allow me to do this since it just looks for a patch named ${filename}.patch
in the patches
directory.
For now, I'll just patch the FMS1 directory, but this is something we might want to do in the future.
We should use the C-grid formulation in CICE6 when it is mature enough, to allow direct coupling with MOM6 on the same grid. See notes: #9
It appears CIME supports JRA RYF (not sure if this means JRA55-do - see #33) but DATM does not (yet).
The ACCESS-OM3 configurations currently use an offset-pole 1° horizontal grid using these grid, mesh, topography and initial condition files.
We should change these to the tripolar grids used at each resolution in the ACCESS-OM2 suite as defined in /g/data/ik11/inputs/access-om2/input_20201102
.
How do we want to structure the directories containing model input data on Gadi?
Different versions of the ACCESS-OM2 inputs include subdirectories for each component at each resolution - e.g. /g/data/ik11/inputs/access-om2/input_ff9890e2
has:
.
├── cice_01deg
├── cice_025deg
├── cice_1deg
├── common_01deg_jra55
├── common_025deg_jra55
├── common_1deg_core
├── common_1deg_jra55
├── minimal_mom_01deg
├── mom_01deg
├── mom_025deg
├── mom_1deg
├── oasis_core_to_1deg
├── yatm_01deg
├── yatm_025deg
└── yatm_1deg
Do we want to replicate this for ACCESS-OM3, or is something like the following sufficient:
.
├── 1deg # subdirectories for inputs specific to a resolution
└── share # inputs shared across configs
Seems like the MOM6 restart filename does not have the correct timestamp. Regardless of the actual simulation time, the file seems to always have seconds set to 0. This is probably not an issue if a calculation runs for more than one simulation day, but otherwise the code might try to write a new restart file with the same name as the previous restart file, causing an error.
looks like the papi_inparm
and prof_inparm
groups in drv_in
aren't read anywhere - can/should they be deleted from all the configs?
https://github.com/COSIMA/access-om3/wiki/Quick-start
should be updated to be consistent with
https://github.com/COSIMA/MOM6-CICE6/blob/main/README.md
When we come to setting up the WW3 configuration we should see what was done with these:
MOM_input
needs to be updated in all configs.
I'll start with the 1deg_jra55do_ryf
branch of MOM6-CICE6 and then migrate changes to other branches and repos.
The 1deg_jra55do_ryf
branch of MOM6-CICE6 is currently closely based on the gmom_jra
CESM branch; there have been several changes leading to the differences tabulated here.
This table compares the MOM6-CICE6 1deg_jra55do_ryf
MOM_input
to various other configurations (bold variables differ between the configs; this table shows only the differences).
As mentioned in #36 (comment), we could consider refining the horizontal grid in the Antarctic and coarsening it in the Arctic.
This should be the starting point of our CI infrastructure.
I've tried to run with the accessom3_exe
branch of @dougiesquire's MOM6-CICE6-WW3 repo
https://github.com/COSIMA/MOM6-CICE6-WW3/tree/139e1b44f8334cfa3e1c04579c9ea2e12e3d0087
which uses an executable /g/data/v45/aek156/access-om3-build/access-om3/bin/access-om3-a47478d
built from this access-om3 repo rather than using CIME.
This failed with an error in CICE6
ERROR: (abort_ice)(input_data)ERROR: tracer_nml reading
Image PC Routine Line Source
libpnetcdf.so.4.0 000014AD4B4295CB tracebackqq_ Unknown Unknown
access-om3-a47478 0000000002DD8E9D shr_abort_mod_mp_ 61 shr_abort_mod.F90
access-om3-a47478 000000000284D4A2 ice_exit_mp_abort 61 ice_exit.F90
access-om3-a47478 0000000002A4FF99 ice_init_mp_input 651 ice_init.F90
access-om3-a47478 000000000288EE42 cice_initmod_mp_c 53 CICE_InitMod.F90
access-om3-a47478 0000000002561D51 ice_comp_nuopc_mp 569 ice_comp_nuopc.F90
access-om3-a47478 00000000008E83F6 Unknown Unknown Unknown
...
access-om3-a47478 0000000000AB4F61 Unknown Unknown Unknown
access-om3-a47478 000000000043146F MAIN__ 128 esmApp.F90
access-om3-a47478 0000000000430EA2 Unknown Unknown Unknown
libc-2.28.so 000014AD47C27D85 __libc_start_main Unknown Unknown
access-om3-a47478 0000000000430DAE Unknown Unknown Unknown
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.
...
whereas the CIME-built exe /g/data/ik11/inputs/cime/bin/MOM6-CICE6-WW3/2023-03-10/cesm.exe
works fine.
datm.streams.xml and drof.streams.xml in the ACCESS-OM3 configurations currently use <mapalgo>bilinear</mapalgo>
but there are better options - see here and here. Probably best to use conserve
for fluxes and patch
for everything else - this is what we did in ACCESS-OM2.
See https://github.com/COSIMA/access-om3/wiki/Configurations for details.
As mentioned in #36 (comment), at some point we should re-generate the topography and land mask to take advantage of using a C grid (narrower straits).
There seems to be a 32/64 bit mixup in some coupler diagnostics - see #40 (comment)
It would be good for reproducibility to be able to set module use
for our spack modules via config.yaml
, as this will then form part of the git history.
Might require some tweaks to payu - see payu-org/payu#347
ACCESS-OM3 should support the latest JRA55-do v1.5.0 but our test configs currently use /g/data/ik11/inputs/cime/inputdata/2023-03-10/ocn/jra55/v1.3_noleap
, which is apparently JRA-55, not JRA55-do, and an old version of JRA-55 to boot.
$ ncdump -h /g/data/ik11/inputs/cime/inputdata/2023-03-10/ocn/jra55/v1.3_noleap/JRA.v1.3.v_10.TL319.2007.171019.nc
...
// global attributes:
:title = "v_10 -- JRA-55 based atmosphere data on TL319 grid" ;
:Source = "Tsujino et al., 2017. JRA-55 Based Surface Dataset for Driving Ocean-Sea-Ice Models. Ocean Modell.\n",
"Kobayashi et al., 2015. The JRA-55 Reanalysis. J. Met. Soc. Jap., 93, 5-48 " ;
:Conventions = "CF1.0" ;
:history = "Original file v_10.2007.18Oct2017.nc reformatted by whokim Thu Oct 19 23:59:42 MDT 2017" ;
:notes = "JRA-55 TL319 data (v1.3, 2017 Oct 18) reformatted for use in CESM, removed all Feb 29\\\'s, calendar now noleap" ;
DATM says it supports "JRA-55 v1.3 forcing data"
https://escomp.github.io/CDEPS/versions/master/html/datm.html#supported-data-modes
CORE_IAF_JRA (datm_datamode_jra_mod.F90)
In conjunction with JRA-55 Project, provides the atmosphere forcing when coupling an active ocean model with observed atmospheric forcing. This mode and associated data sets implement the JRA-55 v1.3 forcing data.
It's unclear whether DATM would also support a newer -do version. More investigation needed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.