amanzi/amanzi

Amanzi primary repository

Amanzi

Master:

BACKGROUND

Amanzi provides a flexible and extensible parallel flow and reactive transport simulation capability for environmental applications. It includes general polyhedral mesh infrastructure, which leverages MSTK, advanced discretizations of process models, including traditional finite volume schemes, mimetic finite differences, and nonlinear finite volumes. In addition, it provides advanced nonlinear solvers, such as Nonlinear Krylov Acceleration (NKA) and Anderson Acceleration, and leverages Trilinos-ML and Hypre Algebraic Multigrid for scalable solvers. The reaction of contaminants and minerals carried by the flow through the surrounding rock and soil is modeled through a general software interface called Alquimia that allows Amanzi to interface with a variety of powerful geochemistry engines including PFLOTRAN and CrunchFlow. The code is parallel and leverages open-source parallel frameworks such as Trilinos, PETSc. Amanzi has been used to model contaminant migration at various DOE waste sites (e.g., Nevada National Security Site, and Savannah River), and is generally applicable to groundwater contaminant migration under partially saturated, nonisothermal conditions and its interaction with surface water.

The multiphysics framework in Amanzi is called Arcos, and it provides modelers with the flexibility they need to creatively decompose complex problems and explore a variety of mixed-dimensional model configurations to develop understanding and make predictions of environmental systems. In particular, Arcos provides flexibility for hierarchical weak and strong coupling of processes with subcycling of mixed dimensions. This capability in conjuction with Amanzi's powerful mesh infrastructure, which supports the splitting and subsetting of meshes, enables creative conceptual modeling. Applications in Amanzi include, coupling flow and transport on discrete-fracture-networks (DFNs) and the background matrix, while applications in the Advanced Terrestrial Simulator (ATS) include, integrated hydrology coupling surface and subsurface processes; and an intermediate scale thermal hydrology model of polygonal tundra based on one-dimensional columns coupled to the two-dimensional surface.

BUILDING/INSTALLING AND RUNNING AMANZI

See the INSTALL.md file in this directory.

COPYRIGHT AND LICENSE

The copyright and license are contained in the top-level file COPYRIGHT.md. The copyright is held jointly by the DOE laboratories that are contributing code to the development of Amanzi.

amanzi's Issues

GCC 6.x uses -std=gnu++14 and breaks some TPLs

Since GCC 6 switched the default language specification to g++14, Trilinos and UnitTest++ have problems. We need to patch and/or update these, ideally in a way that is backward compatible for now.

Update code for <echo_translated_input >

<echo_translated_input > should point to the native spec now. Update the code and documentation.

TPLs collection on software.lanl.gov is unavailable.

Due to cyber-security concerns, software.lanl.gov is being shutdown. This was a convenient place to host our TPLs, and so we are in search of a solution. Updates in CMake provide the option for listing multiple URLs for a given tarball, and eventually we'll want to take advantage of this feature. In the short term we are looking for a hybrid solution of a repository (amanzi-tpls), and direct downloads from mirrors.

improve error message for find region

When calling mesh.get_set_entities() with uninvalid region name the error is not helpful

libc++abi.dylib: terminating with uncaught exception of type std::out_of_range: map::at:  key not found
[pn1618593:31046] *** Process received signal ***
[pn1618593:31046] Signal: Abort trap: 6 (6)
[pn1618593:31046] Signal code:  (0)
[pn1618593:31046] [ 0] 0   libsystem_platform.dylib            0x00007fffdab18bba _sigtramp + 26
[pn1618593:31046] [ 1] 0   ???                                 0x0000000400000000 0x0 + 17179869184
[pn1618593:31046] [ 2] 0   libsystem_c.dylib                   0x00007fffda9a0440 abort + 129
[pn1618593:31046] [ 3] 0   libc++abi.dylib                     0x00007fffd950285a __cxa_bad_cast + 0
[pn1618593:31046] [ 4] 0   libc++abi.dylib                     0x00007fffd9527c37 _ZL25default_terminate_handlerv + 243
[pn1618593:31046] [ 5] 0   libobjc.A.dylib                     0x00007fffda030f33 _ZL15_objc_terminatev + 124
[pn1618593:31046] [ 6] 0   libc++abi.dylib                     0x00007fffd9524d69 _ZSt11__terminatePFvvE + 8
[pn1618593:31046] [ 7] 0   libc++abi.dylib                     0x00007fffd95247de _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
[pn1618593:31046] [ 8] 0   ats                                 0x00000001070cd925 _ZNKSt3__13mapINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN7Teuchos3RCPIKN6Amanzi14AmanziGeometry6RegionEEENS_4lessIS6_EENS4_INS_4pairIKS6_SD_EEEEE2atERSH_ + 245
[pn1618593:31046] [ 9] 0   ats                                 0x00000001070cc35a _ZNK6Amanzi14AmanziGeometry14GeometricModel10FindRegionENSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE + 42
[pn1618593:31046] [10] 0   ats                                 0x00000001070aa1e7 _ZNK6Amanzi10AmanziMesh9Mesh_MSTK25get_set_entities_and_vofsENSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEENS0_11Entity_kindENS0_13Parallel_typeEPNS2_6vectorIiNS6_IiEEEEPNSB_IdNS6_IdEEEE + 759
[pn1618593:31046] [11] 0   ats                                 0x00000001070bfe60 _ZNK6Amanzi10AmanziMesh4Mesh16get_set_entitiesENSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEENS0_11Entity_kindENS0_13Parallel_typeEPNS2_6vectorIiNS6_IiEEEE + 528```

Sources vs sinks

How to distinguish between negative total component concentration and sink? Certainly, we cannot use the concentration value. Since a well may switch from injection to pumping, we cannot use keywords like "source" and "sink".

Picard-Newton iterative method

Migrated tocket (#922):
Implement experimental Picard-Newton method. This method is an alternative to the Newton method, but unlike the latter, it allows us to control the properties of the discrete system and preconditioner on each iteration.

Essential history:

• Added a Newton-type correction to the preconditioner. Impact for DVZ 3layer problem: 304.0 sec (914 time steps) -> 243.2 sec (637 time steps). New options have to be set up manually and will be fully available in version 0.84.
• Added second convergence criterion to Newton: relative residual. Total time for DVZ, 3layer, dropped from 240 seconds to 143 seconds.

Option --with-hg looks obsolete

Bootstrap requires --with-hg which is probably a bug

Implement nonlinear finite volume discretization

Migrated ticket (#1673):
Some code was already written for the dispersion solver in transport PK but was never used. Convert this code to a new diffusion method, in addition to MFD and FV. This work is supported by a third party project.

Essential history:

• Add test of the new scheme with the nonlinear Marshak equation. Fixed failing chemistry test by adding new required element-block <amanzi_chemistry> to the spec.
• Included the NLFV scheme to the list of basic Amanzi-u schemes. Added a new diffusion solver NLFVwithGravity. Added check for flux error to the unit test.
• Added Newton correction terms for the preconditioner.
• DVZ 2D, 3 layers, default options, steady and transient runs:

MFD:   166 ( 3 FS) 79 s   680 (31 FS)  244 s
FV:    114 (17 FS) 63 s   499 ( 0 FS)  139 s
NLFV:  148 ( 2 FS) 44 s   644 (36 FS)  164 s

• Fixed a flux calculation error due internal normal on the dirichlet face. This is more typical in 3D than in 2D.
• Extended Picard initialization solver to work with NLFV.
• Fixed a bug in boundary conditions: member function MapBoundaryValue_ used incorrect face id.
  Changed solution transformation (shift by the hydrostatic solution) to avoid division by rho*g and reduce the number of flops. Added a missing parallel gather of the modified (after the shift)
  righ-hand side.
• Dropped assumption dkdp >= 0 in the upwind discretization of the Newton correction term. Added missing XML file.

Update CMake supporting files to address CMP0054 warnings

CMake 3.1 and above no longer implicitly dereference variables or interpret keywords in an if() command argument when it is a Quoted Argument or a Bracket Argument (see the CMake documentation for examples - https://cmake.org/cmake/help/v3.3/policy/CMP0054.html

As a result when we build the TPLs and Amanzi we see warnings like the following:

CMake Warning (dev) at CMakeLists.txt:249 (if):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "MPI_C_COMPILER" will no longer be dereferenced when
  the policy is set to NEW.  Since the policy is not set the OLD behavior
  will be used.

We need to address this before warnings are no longer provided and the behavior is not what we might expect.

lifting face sets onto a surface mesh fails

Lifting a surface mesh from a volumetric mesh is made to be quite general in lifting labeled sets.

1. if the label is a volumetric cell label, it becomes a surface cell label (this works)
2. if the label is a volumetric face label on the side of the volume, it becomes a surface face label (this works)
3. if the label is a volumetric face label on the surface of the volume, it becomes a surface cell label. This does not work.

Fix 3, and add this to the unit tests that does surface mesh extraction.

Boost fails in parallel

Creation of the final hard link fails in parallel runs with gcc 5 and 6 compilers.
the error message is not consistent. sometimes, we get this:
boost::filesystem::create_hard_link: File exists: "chk00313.h5", "chk_final.h5"

UnitTest++ generates linking error on OSX

There's something messed up with our configure/build/install of UnitTest++ on OSX. I'm getting the following error

[  1%] Linking CXX executable test_error_handling
ld: warning: directory not found for option '-L/ascem/amanzi/install/OSX-10.9.5-x86_64/openmpi-1.10.2-gcc-4.9.3/Release-master-alquimia-160926/tpls/lib'
Undefined symbols for architecture x86_64:
  "std::__1::__basic_string_common<true>::__throw_length_error() const", referenced from:
      __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initIPcEENS_9enable_ifIXsr21__is_forward_iteratorIT_EE5valueEvE4typeES9_S9_ in libUnitTest++.a(MemoryOutStream.cpp.o)
  "std::__1::locale::use_facet(std::__1::locale::id&) const", referenced from:
      std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) in libUnitTest++.a(Checks.cpp.o)

[snipped the remaining spew]

Check configure for new UnitTest++, since this appeared after updates to TPLs and switch to C++11 standard.

Initial support of time-space functions for BCs and sources

Migrated ticket (#1702):
Implement the multiplicative representation f(t) * g(x) of input data for BCs and sources. The final 2.x interface is under discussion right now. Temporarily, the interface looks like:
{{{
<inward_mass_flux start="0.0">


</inward_mass_flux>
}}}

Improve algebraic consistency of flow and transport

Migrated ticket (#728):
Monotone transport of concentration requires a precise balance of the saturation and Darcy flux.
Inexact solver (BDFx) breaks this balance. To restore it, the saturation should be updated from
the calculated flux and not as function S(p).

Diffusion operator does not support degenerate coefficients

Migrated ticket (#1363):
Current design of the diffusion operator cannot support any degenerate coefficient. some work has been done by Ethan and Eugene, but design is too heavy in my opinion.

Essential history:

• Added a few checks for zero local matrices.

Extend WhetStone to manifolds

Migrated ticket (#1204):
Extend package Whetstone to support manifolds.

Created templated DomainFuction

The main purpose of these modifications is to be able to define multi-component functions for
boundary conditions, source and coupling of different domains.

Each domain function is defined as a templated class DomainFunction.
Then Factory should be defined as DomainFunctionFactory<ValueType, DomainFunction>

Implement new symmetric scheme for unstructured meshes

Migrated ticket (#1590):
A fractional splitting of relative permeability between dual operators has interesting consequences. This ticket will study them. This is highly experimental and should be moved to a separate branch.

Seepage face output looks suspicious

CycleDriver | Cycle 1214: time = 900 y, dt = 100 y
TI::BDF1 | step 1214 T = 2.84018e+10 [sec] dT = 3.15576e+09
TI::BDF1 | preconditioner lag is 0 out of 0
TI::BDF1 | success: 1 nonlinear itrs error=3.24862e-14
TI::BDF1 | TS:1214 FS:454 NS:16483 PC:32966 32966 LS:160275 dt:1.0000e+00 3.1558e+09
FlowPK::Richards | reservoir water mass=8.36383e+10 ton, total influx=2.36509e+08 ton
FlowPK::Richards | seepage face: flow=0 [kg/s], total=0 [kg]
TimeStepManager | Proposed dT=3.78691e+09 [sec]. CD limits it to 1
FlowPK::Richards | Secondary fields: hydraulic head, darcy_velocity
CycleDriver | writing visualization file

Output of seepage information for the FV scheme is confusing.
The field statistics at 1000 y shows that the head is huge again which is consistent
with the average saturation = 0.999999 unless it is a bug in the FV implementation of
the seepage face.

K

CycleDriver | Field Min/Max/Avg
CycleDriver | atmospheric_pressure.......... 101325
CycleDriver | darcy_flux.................... -0.651635 / 0.587858 / 0.000106949
CycleDriver | darcy_velocity................ -0.0274792 / 0.0353596 / -1.48144e-05
CycleDriver | dt............................ 3.15576e+09
CycleDriver | fluid_density................. 998.2
CycleDriver | fluid_viscosity............... 0.001002
CycleDriver | hydraulic_head................ 8.26075 / 35959.9 / 18667.6
CycleDriver | molar_density_liquid.......... 55408.4 / 55408.4 / 55408.4
CycleDriver | permeability.................. 1.99033e-13 / 1.18472e-09 / 2.54498e-10
CycleDriver | porosity...................... 0.25 / 0.43 / 0.375751
CycleDriver | pressure...................... 87522.9 / 3.52137e+08 / 1.82593e+08
CycleDriver | prev_saturation_liquid........ 0.446343 / 1 / 0.999999
CycleDriver | prev_water_content............ 10139.8 / 23825.6 / 20819.7
CycleDriver | saturation_liquid............. 0.446343 / 1 / 0.999999
CycleDriver | viscosity_liquid.............. 0.001002 / 0.001002 / 0.001002
CycleDriver | water_content................. 10139.8 / 23825.6 / 20819.7

Upgrade upwind methods

Migrated ticket (#1678):
The existing upwind methods will be re-implemented to reduce load on class Richards PK. Logic of underlying algorithms will be simplified. One additional upwind method will be implemented and studied.

Boundary Conditions for Operator Advection

Migrated ticket (#1460):
Dirichlet boundary conditions for Advection Operator has to be treated differently than in diffusion case. Right now for the face based schemes on boundary faces we have matrices 1x1. This has to be set to zero or excluded from assembling once BC are added to rhs. Possible, but not very elegant fix is new_mpc branch.

Extracting solutes with pumping wells

Currently, the solute sources use prescribed values, so they are appropriate for sources, not sinks. New boundary condition is needed.

Update version string creation to use git.

The commands and parsing that generates version string from the latest amanzi-* tag was setup for Mercurial. With the shift to git, this code in AmanziVersion.cmake is broken and needs to be updated.

Simplex method needs stability analysis

Migrated ticket (#1162):
Simplex method can go unstable for unknown yet reasons when the number of parameters is huge. The test case is the calculation of the stiffness matrix for the dodecahedron. We do not use such meshes in practice (yet) but the issue still needs to be addressed to have a piece of mind.

Essential history:

• Changed tolerance in simplex method. Added missing converge check in the second phase of algorithm.
• Due to round-off errors, right-hand side of the constraints may be small but still negative. This breaks the internal logic. Changed pivoting rule from Dantzig's to new (searching for reference) MFD3D.

missing test file fracture.exo

commit 5ecd153 added a new test, test_nonmanifold_surfaces.cc, but is missing some supporting mesh files -- at least one called fracture.exo. Maybe more?

Rao, can you add this and any other new files needed for testing?

Thanks, glad to see this getting fleshed out more!

Bad error message

The execution control section specifies no time step decrease:

<execution_control end="1.0 d" increase_factor="1.0" init_dt="0.1 d" max_dt="0.1 d" 
                               method="bdf1" mode="transient" reduction_factor="1.0" start="0 d"/>

The simulation terminates (which makes sense) but the error message is wrong:

CycleDriver | Cycle 0: time = 0 s, dt = 2.4 h
TI::BDF1 | step 0 T = 0 [sec] dT = 8640
Solver::NKA | terminating due to preconditioner overflow: ||du_tmp||=7.9059e+40 ||du_tmp0||=3.33564e+11
TI::BDF1 | step failed with error code -2
dT change is too small, terminating...

Add location of output data to the command line

Implement new input command line option --output_prefix. The provided prefix will be added to all output files.

Physical BC for energy

develop bc for energy_pk for energy flux conditions

Adaptive time stepping for Darcy

Migrated ticket (#837):
Implement an adaptive time stepping for the Darcy flow. The selected algorithm is based on the work of Kavetski, Binning, Sloan (AWR, V.24, 2001 pp.595-605), that in turn was inspired by the work of Thomas, Gladwell (Numer.Meth.Eng. V.26, 1988 pp.39-53) on a family of stable multistage integration methods for ODEs, that in turn extended the work of Zienkiewicz el al (1984).

Update Sphinx documentation to use git for version numbers

The web front page and User Guide were setup to use data from the Mercurial repository to get the current version number. Each of these has a simple python code section in conf.py that needs to be updated to use git.

Estimate stable dT for modified Kuzmin limiter

Migrated ticket (#1456):
A modified Kuzmin limiter shows superior performance on general meshes (see the unit test) compared to the two other limiters we have, Barth-Jespersen and Tensorial. Its behavior for realistic flows requires additional study of a stable time step.

AmanziVersion.cmake does not handle detached heads

Building older versions of Amanzi for various debugging or reproducibility checks is fairly common. To restore the old version it is simply a matter of checking out the old sha1. For example,

   git checkout <sha1>

However, this puts the current working directory in a "detached head" state, and dropping its direct connection to the branch that you were working on. Moreover, the current git based branch lookup in AmanziVersion.cmake fails, causing the build to fail.

Note if the revision pre-dates our move to Git, Amanzi will build but not have accurate version information.

Applying multiple patches for HDF5 fails

For an unknown reason (maybe newer version) CMake execution of the patch script may be executed as

   sh hdf5-patch-step.sh

so this fails to pickup the #!/bin/bash at the beginning and depends on what sh executes. On some systems (e.g., Ubuntu 14.04) sh -> dash and not bash. With dash we get a syntax error

[  8%] Performing patch step for 'hdf5'
/n/swdev/ats/amanzi-tpls/build-Ubuntu-14.04-x86_64/dev/RelWithDebInfo/clang-3.6.0_gfortran-4.8.2_openmpi-1.8.4/hdf5/hdf5-patch-step.sh: 13: /n/swdev/ats/amanzi-tpls/build-Ubuntu-14.04-x86_64/dev/RelWithDebInfo/clang-3.6.0_gfortran-4.8.2_openmpi-1.8.4/hdf5/hdf5-patch-step.sh: Syntax error: "(" unexpected
CMake Error at /n/swdev/ats/amanzi-tpls/build-Ubuntu-14.04-x86_64/dev/RelWithDebInfo/clang-3.6.0_gfortran-4.8.2_openmpi-1.8.4/hdf5/hdf5-patch-step.cmake:16 (message):
  HDF5 patch command failed 'sh'
  '/n/swdev/ats/amanzi-tpls/build-Ubuntu-14.04-x86_64/dev/RelWithDebInfo/clang-3.6.0_gfortran-4.8.2_openmpi-1.8.4/hdf5/hdf5-patch-step.sh'

and then the TPL build aborts.

Alquimia not failing through Amanzi

Jobs launched through Akuna use Agni to manage the runs on NERSC. To determine if a job is "done" and report success or failure, Agni looks for keywords in the stdout. Specifically "Amanzi::SIMULATION_FAILED". If a run fails inside of Alquimia, the exception doesn't make it back up to Amanzi's messaging. I get the following without a SUCESSFUL or FAILED message:

CycleDriver | Reseting CD: TP 0 -> TP 1.
PFloTran_Alquimia_Setup() :
Reading : amanzi_0/amanzi.in
WARNING: Card (NONISOTHERMAL) not found in input file.
Aqueous primary species (Tc-99) not found in database.

ERROR: Species not found in database.
Stopping!

We need the termination to still report the expected text so Agni can kill the job.

Operators need support of high-order discretizations

Migrated ticket (#1454):
Operators need more functionality to support high-order advection operators. A special implementation of Apply() is a possible solution.

HDF5 build fails on Cori

The build environment on Cori was updated recently to craype/2.5.5:

• cray-mpich/7.4.0
• gcc/5.3.0

and HDF5 (version 1.8.8) now fails to build with the following error:

/bin/sh: line 4: 100332 Segmentation fault      LD_LIBRARY_PATH="$LD_LIBRARY_PATH`echo -L/project/projectdirs/m1012/amanzi/tpls/install/cori/mpich2-7.4.0-gnu-5.3.0/Release-0.92.24/lib |                   sed -e 's/-L/:/g' -e 's/ //g'`" ./H5make_libsettings > H5lib_settings.c
make[5]: *** [H5lib_settings.c] Error 1

This is the same problem we encountered on Hopper years ago and it neither NERSC support or HDF5 developers could fix it.

Attaching and rendering xml files

Issues need to include xml files as attachments and/or snippets need to be rendered with pygments.

Update to MSTK 2.26rc2 did not update TPLs version

The update to MSTK (2.26rc2) needs to bump the TPLs version to 0.92.25.

CMake update causes linking of extraneous HDF5 system libraries

We don't use the C++ or Fortran interfaces to HDF5, and so we don't build them. Unfortunately, the FindHDF5 module looks for them, and if it finds them on the system it does link to them.
This linking behavior results in a binary that may have dynamic dependence to a system library, and so it may not be portable.

Currently, we do use the set the HDF5_NO_SYSTEM_PATHS variable, but for some reason it is no longer enough.

Trimming unit string

" init_dt="1 s" (double spaces) breaks the unstructured converter.

Option --disable-petsc has no impact

Bootstrap ignores option --disable-petsc.

Adaptive time stepping for Richards

Migrated ticket (#921):
Implement adaptive time stepping for Richards. Minimal changes with respect to a similar strategy for Darcy are needed.

Boundary effect in Porflow 4.4

Migrated ticket (#1386):
The flux on the right side of the top (seepage) boundary is negative for the TPFA discretization. The MFD discretization has no such effect.

Essential BCs should be imposed by MPC_PK before step starts

Migrated ticket (#1534):
The very first functional calculation requires essential boundary conditions for all primary fields. These conditions should be imposed by MPC_PK before step starts.

Revise parallel communications in amanzi and related codying rules

Migrated ticket (#1603):
Currently MANAGED_COMMUNICATION is set to off by default and we do not optimize the parallel communications. The purpose of this ticket is to reactivate this flag.

Missing exoatt

exoatt is not being built (locally or nersc). Maybe related to the MSTK update.

README and INSTALL update for github

The README needs to be updated to reflect

• repo moved to github
• external collaborators can use pull requests
• if significant collaboration is planned then copyright assignment or assertion is needed

The INSTALL needs to be updated to reflect the location, and we need to get the UserGuide on line as quickly as possible.

Implement dual-porosity model

Migrated ticket (#1635):
Implement dual-porosity model. Extend it to energy equation.

Support units in Amanzi

Migrated ticket (#1703) :
Add support of units to the unstructured input converter.

History:

• Established a base line for supporting units in the input XML.
• Improved internal functions for converting units.
• Extended support of units. Boost library is really not good for managing dynamic quantaties. There is a work around based on boost::variant, but people say that it is overkill and a single class is better.
• Established a base line for supporting units in the input XML.
• Improved internal functions for converting units.
• Extended support of units. Boost library is really not good for managing
  dynamic quantities. There is a work around based on boost::variant,
  but people say that it is overkill and a single class is better.

Preconditioners for coupled systems

Migrated ticket (#1535):
Develop either an abstract framework or one specific implementation (for Energy+Flow PK) of preconditioners for coupled systems.