amrex-fhd/fhdex

AMReX-enabled code libraries for stochastic hybrid models and algorithms for fluids

fhdex's Issues

fftw3-mpi.h - fatal error

I'm having an issue with fftw3-mpi.h. When doing make I get an error:

On my local machine, I "fixed" this by installing the libfftw3-mpi-dev packages but afterwords it still wouldn't make an executable. Then I tried to uninstall and get back and then nothing worked.

To confirm, I just tried on Kong and got the same fftw3-mpi.h error. Before I "fix" this too, I thought it would be a good idea to ask.

Am I doing something weird or is this a thing? Thanks.

NUM_EDGE to common

NUM_EDGE should probably be defined in common_functions.H instead of gmres_functions.H. One could image a code that cared about the number of edges in 2d vs 3d that doesn't use GMRES.

Need horizontal average utility

We need a routine that takes 2D or 3D dataset and horizontally averages to a 1D text file output.

Unit tests

Tests can be found under /exec/tests subdir.
RFD (/tests/RFDanalysis):
Run with rfd_tog on and off, analyse particle distribution using Mathematica file. PASSED.

periodic electrostatic (/tests/electroPeriodic):
Run using input file and results for various particle positions will print out

• force between two particles across x boundary between two boxes, within p3m range: PASSED
• force between two particles across x boundary in same box, within p3m range: PASSED
• force between two particles in opposite periodic corners, within p3m range: MAYBE, ~6% error?
• force between two particles across x boundary between two boxes, outside p3m range: PASSED, ~2.3% error

Derive Runiv from Avogradro and k_B

Instead of deriving avogadro from k_B and Runiv we want to read in avogadro and k_B and set Runiv

compressible and compressible_mui driver merge

compressible_mui can share the same main_driver as compressible/ with just one ifdef flag; this will save time by not having to manually merge changes from the compressible main_driver

Robustness of the structure factor routine

The structure factor (SF) routine currently seems to work well for 3D MultiFabs (MF) with more than 1 cell in every direction. However, the following issues are unresolved:

1. SF routine does not work for a flattened MF obtained either from averaging or slicing a 3D MF in any direction (x, y, or z).
2. Compute SF for a 3D MF with only 1 cell in any direction (x, y or z), and compare with above.
3. The issues 1 and 2 need to be resolved on both GPU and CPU.
4. Check for consistencies in SF computation in an actual FHDeX run. Set up regression tests.
5. Check for any memory leaks in SF computations.

Compressible and Particle namespace

-Create a compressible namespace so the common namespace isn't so bloated - CREATED, but many variables are still in the common namespace.
-Create a particle namespace so the common namespace isn't so bloated.

FHDeX Bug list

• For BOBA codes, particle gen function generates wrong number of particles if number of mpi ranks is not equal to number of boxes. For now run with one box per mpi.

nvars and nprimvars

nvars and nprimvars could be moved out of the common namespace and into the compressible namespace.

Add Restart Capability to immersedIons

There are 3 aspects here; the hydro, particle, and random number states. Katie and I are going to start tackling particle and hydro. Hydro first in the hydro/ example that can be ported to immersedIons.

Immersed Ions Current Variance with Tiles

For immersedIons, the current variance is not being computed correctly when using tiles. Multiple grids works fine.

molmass and molweight

split molmass into molmass and molweight and make sure they are being used correctly, and only one of them is being read in and filling the other.

[compressible_mui] struct fact routine hanging

it is not observed in the compressible code but observed in the compressible_mui code.

Replace SWFFT

The current SWFFT implementation of the structure factor is a bit brittle in parallel (only works for sure if the # of processors = the # of grids; the actual requirement may be slightly looser but this may not be coded up correctly on our end)

We may want to consider what WarpX does using standard FFTW libraries.

Also, the shifting algorithm (to put k=0 in the center) is performed on one grid and eventually may not fit in memory for large problems.

Finding weights when doing midpoint correction in immersedIOns

Currently when doing midpoint stepping it is possible for the code to read out of bounds when calculating weights to do an interpolation. This will have no effect, because the interpolation will never take place, but it may cause an out of bounds violation when running in debug. This issue is a reminder for me to fix this when I refactor the particle move function.

Code Consistency

These codes:

compressible
compressible_stag
compressible_mui
hydro
multispecies
immersedIons
immsersed_boundary_flagellum (?)

Should support some common features including the following. Almost all of these are implemented somewhere but these can all be easily implemented in all of the frameworks.

RNG
-Proper initialization of the C++ engine on multiple MPI ranks with seed=0: random and seed>0: fixed

Checkpointing
-RNG (seed=-1 means use the seed from restart, 0=random, >0=fixed)
-Ability to restart into a new BoxArray
-Currently structure factor is not saved in Checkpoints. Need to add this capability.

Structure Factor
-Full dimensional
-Flattened (there are plans to switch the flattened support to work on 1 grid / 1 MPI using our own wrapper around fftw since SWFFT is buggy)

Statistics
-Mean and variance, including checkpointing statsCount so it restarts
-Smartly set up stats calculations, particularly for the multi-species case. There are a lot of variables in this case, and user should be able to choose what stats are required. Also, adding new stats to the existing is cumbersome -- need a better data structure than a fixed vector initialized at the start-up. For the SF this is already possible with hard-coding, but if we could control this in the inputs file somehow it would be better.

n_steps_skip
-a negative value should reset both the mean/variance and structure factor statistics. We should create a standalone routine to reset the statistics so we can do this AFTER writing the plotfiles (both mean/variance and SF) so we don't have use tricks like writing out SF plotfile 9999 before step 10000 since we are resetting the sums)

Add restart capability to `immersed_boundary` codes

• figure out how to get the IBMarkerContainer class happy with restarts.

Boundary Issues in the Cell-Centered Compressible FHD

The cell-centered code for compressible FHD has several outstanding issues listed below. A major re-factor of the code was performed in Feb 2022 for a better prescription of the Dirichlet boundary conditions in the PR #98. In this PR, Dirichlet boundary conditions are assigned at the boundaries rather than a linear extrapolation to ghost cells for increased numerical stability. However several issues have emerged, both after the PR (post-Feb version) and before the PR (pre-Feb version)

1. Pre-Feb version: A binary mixture of gases at equilibrium confined between isothermal, no-slip walls exhibit incorrect density and energy variance in the boundary cells. When slip walls are used, all variances appear to be correct (see attached pdf file, and attached inputs file for testing). 0328MON_sim_res_pre-Feb_version.pdf and inputs_cell_cent_test.txt
2. Post-Feb version: For the similar setup as above, incorrect variances are observed for both slip and no-slip boundaries, with the results being much worse than the Pre-Feb version.
3. Revisit the treatment of energy fluxes at the walls. A previous PR (#37) addressed this issue by separately treating the various contributions (heat, Dufour, and viscous heating) at the walls.

hydro code for Turblence; GPU vs CPU

E(k) for GPU vs CPU appears to be different for hydro turbulence runs.

hydro restart into finer

For hydro, Add a new 'init type' that reads in a plotfile or checkpoint and injects the data into a finer-by-X refinement ration and then does a projection

Structure Factor Restart

Implement restart ability for structure factors.

Add support for non-cubic `dx`

... or add abort/warning when dx[0] != dx[1] != dx[2]

Remove Fortran RNGs

Completely eliminate the fortran random number code. The main downfall is that we can no longer have separate engines. There is simply one master engine for all processes and RNG types. The hydro example shows how to initialize and restart the C++ engine properly.

Memory optimization in main_drivers

Many main_drivers have multifabs that are built but never used. One example is the structure factor multifabs. They should be declare but only defined if the structure factor is needed.

Add support for restart from latest checkpoint folder in any path

Add support to provide a restart option such as "restart PATH/*", where the program will look for the checkpoint folder in PATH, and the wildcard * would search and use the latest checkpoint folder. This would enable easy daisy-chaining long jobs with dependency restarts without baby-sitting them, especially on systems with short wall times. A similar capability exists in LAMMPS, so that could be adopted here.

Flattened SF Scaling

On branch SF_flat_test
Directory FHDeX/exec/structFactTest_flat
The test inputs_3d shows the flattened structure factor is scaled by 32 compared to the full. They should be the same.

evaluateStats with stats_int

The statistics in the compressible code take 4x longer to run than the actual time step (and some problems dont' need them). I want to implement a stats_int (or similar; or use something we already have) that, when set to -1, doesn't do the evaluateStats call. The default value will be 1 so nothing else is affected.

Standalone E(k) program

We need a program that takes in a plotfile and compute E(k)