[FLASH-USERS] Bug in 3T hy_uhd_DataReconstructNormalDir_MH.F90

[Klaus Weide]

On Tue, 10 May 2016, Mark Richardson wrote:

>   Currently, I'm getting a weird issue that in the step after a refinement
> occurs the value of grid_changed is 0 again in gr_hypreGridStatus, which
> means the hypre array gr_hypreLower is improperly allocated by the time I
> get to gr_hypreSetIniGuess for some processors, and I crash.
> 
>  Looking in Driver_evolveFlash, I see that in the step after a refinement
> occurs, grid_changed gets changed from 1 back to 0 in Hydro, but this is
> before the source terms are called, so they don't actually know that the
> grid has changed.
> 
> Is this the expected behaviour of grid_changed? Did it used to happen at
> another time so that the source terms would also see it?

The exact behavior of the 'grid_changed' flag - in particular, who is 
allowed (or expected) to reset it in a FLASH simulation - has never been 
precisely specified, unfortunately.

To avoid problems, the flag should ONLY be reset when 
Grid_updateRefinement / gr_updateRefinement are called at the end of a 
time step.  The one place is find where this expectation is violated in 
general code (i.e., in released code, outside of specific Simulations)
is amr_mg_morton_process.F90, which is called by Multigrid solvers.
So if, in a time step, you first perform a Multigrid solve and later
a HYPRE solve, the HYPRE solve may see grid_changed==0 even though the
grid DID change.

It should be safe to disable the 'grid_changed = 0' statement in
amr_mg_morton_process.F90. This could result in some inefficiencies
because some setup code may be unnecessarily executed more than once,
but I don't think that will have a significant impact. (It should
only happen in cases where amr_mg_init would be called more than once 
during a timestep.)

It is tempting to think that the multigrid code, in particular 
amr_mg_morton_process.F90 from the PARAMESH code, is doing something wrong 
here. However, historically the 'grid_changed' flag was just a flag
used privately within PARAMESH, so PARAMESH routines were not required to 
consider the impact of (re)setting this flag on other pieces of code that
just use PARAMESH. FLASH then started using this flag, assuming a meaning 
that appeared obvious but was not guaranteed by PARAMESH.

A cleaner way to handle this might involve introducing a mechanism by 
which various code units could get notified when the Grid gets changed by 
Grid_updateRefinement / gr_updateRefinement at the end of a step.
So far the need for this had not really come up. Your pioneering
use of both GridSolvers/Multigrid and GridSolvers/HYPRE in the 
same simulation has made the latent problem visible.

Klaus