[FLASH-USERS] Issues with MPI and meshreplication and rad-hydro problems

[Nelson, Peter]

Thanks Klaus,

   I do intend to extend this into 2d, and possibly 3d, on large parallel
machines, so the mesh replication is certainly of interest.    The reason I
was concerned was that the mesh replication actually altered the simulation
variables in my setup.  The deposition energy in the simulation is doubled
when I set meshCopyCount =2.    The setup was modified from the LaserSlab
example initially.  But when I run some of the example problems (LaserSlab,
Sedov, Rad Blast Wave and few others) the .dat file is inaccurate, but the
results of the simulation do not change in the way my setup does.   I'm not
sure what I did in my setup, but I'm testing a few changes to see if I can
resolve the issue.

   I see some examples use solnData() to set the initial simulation
variables.  Others use Grid_putPointData().  Is there a preferred method or
issues with using one or the other than might be the issue?

Regards,
Pete



On Mon, Mar 9, 2020 at 1:08 PM Klaus Weide <klaus at flash.uchicago.edu> wrote:

>
>
> On Mon, 9 Mar 2020, Nelson, Peter wrote:
>
> > 2) meshCopyCount seems to multiply the internal energy initialized in the
> > Simulation_initBlock routine.  if I print the solnData at the end of
> > simulationInitBlock, everything looks fine, but when the simulation runs
> it
> > somehow ends up with the internal energy getting multiplied by
> > meshCopyCount.    Same issue on all three systems (with FLASH4.6.2
> > installs).  I checked multiple supplied test problems and they all
> present
> > the same problem.
>
> Peter,
>
> I wonder whether it makes sense for you to used mesh replication (i.e.,
> meshCopyCount > 1) in the first place. If you just started using it in an
> attempt to work around a problem in your MPI installation, then it may not
> make sense anyway.
>
> Mesh replication was introduced in FLASH to deal with problems in large,
> highly parallel simulations with large memory and CPU time requirements.
> It is likely not so useful for a 1D configuration.
>
> Anyway, when you use mesh replications, most of the solution is
> represented redundantly multiple times (except for the radiation group
> energies), just as most of the computation is performed, redundantly,
> multiple times in parallel (except for the radiation transport).
> It seems likely that the version of IO_writeIntegralQuantities.F90
> that you are using, possibly the default implementation from the IO unit,
> simply is unaware that mesh replication is in effect, and is summing
> contributions to energy (and density, and other variables) over *all*
> replicated instances of the mesh, instead of just *one*.
>
> The solution may be as simple as replacing io_globalComm, io_globalMe
> with io_meshComm, io_meshMe, respectively, in
> IO_writeIntegralQuantities.F90, *and* making sure that only *one* instance
> of the mesh [which one?] actually performs the summation.
>
> I have not tried this, and do not know how this needs to be modified to
> actually have all the radiation group energies appear in the flash.dat
> file (if one is really using MGD for RadTrans with mesh replication). It
> is possible that people have written customized versions of
> IO_writeIntegralQuantities.F90 for specific simulations that deal with
> this properly.
>
> Klaus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20200311/e9c2cba3/attachment.htm>