[FLASH-USERS] Issues with MPI and meshreplication and rad-hydro problems

[Klaus Weide]

On Mon, 9 Mar 2020, Nelson, Peter wrote:

> 2) meshCopyCount seems to multiply the internal energy initialized in the
> Simulation_initBlock routine.  if I print the solnData at the end of
> simulationInitBlock, everything looks fine, but when the simulation runs it
> somehow ends up with the internal energy getting multiplied by
> meshCopyCount.    Same issue on all three systems (with FLASH4.6.2
> installs).  I checked multiple supplied test problems and they all present
> the same problem.

Peter,

I wonder whether it makes sense for you to used mesh replication (i.e., 
meshCopyCount > 1) in the first place. If you just started using it in an 
attempt to work around a problem in your MPI installation, then it may not 
make sense anyway.

Mesh replication was introduced in FLASH to deal with problems in large,
highly parallel simulations with large memory and CPU time requirements.
It is likely not so useful for a 1D configuration.

Anyway, when you use mesh replications, most of the solution is 
represented redundantly multiple times (except for the radiation group 
energies), just as most of the computation is performed, redundantly, 
multiple times in parallel (except for the radiation transport).
It seems likely that the version of IO_writeIntegralQuantities.F90
that you are using, possibly the default implementation from the IO unit, 
simply is unaware that mesh replication is in effect, and is summing
contributions to energy (and density, and other variables) over *all*
replicated instances of the mesh, instead of just *one*.

The solution may be as simple as replacing io_globalComm, io_globalMe
with io_meshComm, io_meshMe, respectively, in 
IO_writeIntegralQuantities.F90, *and* making sure that only *one* instance 
of the mesh [which one?] actually performs the summation.

I have not tried this, and do not know how this needs to be modified to 
actually have all the radiation group energies appear in the flash.dat 
file (if one is really using MGD for RadTrans with mesh replication). It 
is possible that people have written customized versions of 
IO_writeIntegralQuantities.F90 for specific simulations that deal with 
this properly.

Klaus