[FLASH-USERS] Error in AMR Prolong Routine

Fri May 1 15:14:41 EDT 2009

How embarrassing...yes, that routine does not change anything (I  
judged the function by its cover...). However, that change does show  
that there are guard cells that have 0 values after the amr_prolong  
routine runs, the WARNINGs below are absent prior to the change:

[ 04-30-2009  12:12:55.164 ] [GRID amr_refine_derefine]: refinement  
complete
[ 04-30-2009  12:12:55.626 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=329 type=1
[ 04-30-2009  12:12:55.628 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=329 type=1
[ 04-30-2009  12:12:55.631 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=329 type=1
[ 04-30-2009  12:12:56.151 ] step: n=5555 t=5.608191E+04 dt=1.325975E+01
[ 04-30-2009  12:12:56.281 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1547129, buffer_dim_recv=1504989
[ 04-30-2009  12:13:52.188 ] [IO writePlotfile] open: type=plotfile  
name=multitidal_hdf5_plt_cnt_0561
[ 04-30-2009  12:14:02.278 ] [IO_writePlotfile] close: type=plotfile  
name=multitidal_hdf5_plt_cnt_0561
[ 04-30-2009  12:14:02.304 ] step: n=5556 t=5.610843E+04 dt=1.334945E+01
[ 04-30-2009  12:14:55.353 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1113465, buffer_dim_recv=1083101
[ 04-30-2009  12:14:57.828 ] [GRID amr_refine_derefine]: initiating  
refinement
[GRID amr_refine_derefine] min blks 330    max blks 334    tot blks  
21217
[GRID amr_refine_derefine] min leaf blks 288    max leaf blks 292     
tot leaf b
lks 18565
[ 04-30-2009  12:14:58.453 ] [GRID amr_refine_derefine]: refinement  
complete
[ 04-30-2009  12:14:59.018 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=330 type=1
[ 04-30-2009  12:14:59.021 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=330 type=1
[ 04-30-2009  12:14:59.024 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=330 type=1
[ 04-30-2009  12:14:59.025 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=331 type=1
[ 04-30-2009  12:14:59.026 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=331 type=1
[ 04-30-2009  12:14:59.057 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=331 type=1
[ 04-30-2009  12:14:59.059 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=332 type=2
[ 04-30-2009  12:14:59.061 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=332 type=2
[ 04-30-2009  12:14:59.063 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=332 type=2
[ 04-30-2009  12:14:59.064 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=333 type=1
[ 04-30-2009  12:14:59.066 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=333 type=1
[ 04-30-2009  12:14:59.068 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=333 type=1
[ 04-30-2009  12:15:00.796 ] step: n=5557 t=5.613513E+04 dt=1.327865E+01
[ 04-30-2009  12:15:00.924 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1570821, buffer_dim_recv=1555957
[ 04-30-2009  12:15:53.087 ] step: n=5558 t=5.616168E+04 dt=1.337244E+01
[ 04-30-2009  12:16:45.713 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1130501, buffer_dim_recv=1119733
[ 04-30-2009  12:16:48.160 ] [GRID amr_refine_derefine]: initiating  
refinement
[GRID amr_refine_derefine] min blks 331    max blks 335    tot blks  
21297
[GRID amr_refine_derefine] min leaf blks 288    max leaf blks 293     
tot leaf b
lks 18635
[ 04-30-2009  12:16:48.823 ] [GRID amr_refine_derefine]: refinement  
complete
[ 04-30-2009  12:16:49.260 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=334 type=1
[ 04-30-2009  12:16:49.263 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=334 type=1
[ 04-30-2009  12:16:49.265 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=334 type=1
[ 04-30-2009  12:16:50.520 ] step: n=5559 t=5.618843E+04 dt=1.350858E+01
[ 04-30-2009  12:16:50.650 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1581705, buffer_dim_recv=1566841
[ 04-30-2009  12:17:47.727 ] [IO writePlotfile] open: type=plotfile  
name=multitidal_hdf5_plt_cnt_0562
[ 04-30-2009  12:18:01.029 ] [IO_writePlotfile] close: type=plotfile  
name=multitidal_hdf5_plt_cnt_0562
[ 04-30-2009  12:18:01.102 ] step: n=5560 t=5.621545E+04 dt=1.351893E+01
[ 04-30-2009  12:18:53.623 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1138313, buffer_dim_recv=1127545
[ 04-30-2009  12:18:56.063 ] [GRID amr_refine_derefine]: initiating  
refinement
[GRID amr_refine_derefine] min blks 332    max blks 335    tot blks  
21321
[GRID amr_refine_derefine] min leaf blks 289    max leaf blks 293     
tot leaf b
lks 18656
[ 04-30-2009  12:18:56.893 ] [GRID amr_refine_derefine]: refinement  
complete
[ 04-30-2009  12:18:57.740 ] step: n=5561 t=5.624248E+04 dt=1.347266E+01
[ 04-30-2009  12:18:57.869 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1581705, buffer_dim_recv=1566841
[ 04-30-2009  12:19:49.553 ] step: n=5562 t=5.626943E+04 dt=1.344924E+01
[ 04-30-2009  12:20:42.142 ] [mpi_amr_comm_setup]:  
buffer_dim_send=1138313, buffer_dim_recv=1127545
[ 04-30-2009  12:20:44.669 ] [GRID amr_refine_derefine]: initiating  
refinement
[GRID amr_refine_derefine] min blks 332    max blks 336    tot blks  
21369
[GRID amr_refine_derefine] min leaf blks 290    max leaf blks 293     
tot leaf b
lks 18698
[ 04-30-2009  12:20:45.336 ] [GRID amr_refine_derefine]: refinement  
complete
[ 04-30-2009  12:20:45.742 ] WARNING after amr_prolong: min.  
unk(DENS_VAR)=0.000000000000000000000 PE=0 block=335 type=1
[ 04-30-2009  12:20:45.744 ] WARNING after amr_prolong: min.  
unk(ENER_VAR)=0.000000000000000 PE=0 block=335 type=1
[ 04-30-2009  12:20:45.747 ] WARNING after amr_prolong: min.  
unk(EINT_VAR)=0.000000000000000 PE=0 block=335 type=1

In regards to this comment:

> By the way, gr_updateRefinement (or its caller,  
> Grid_updateRefinement) is
> not required to leave *guard cells* in a "sane" state; it is the
> responsibility of any routine that might be called next - in  
> particular,
> any physics unit routine that applies some operator to advance the
> solution - to fill guard cells as needed, usually by calling
> Grid_fillGuardCells.  Interior cells must of course have valid cell
> data after Grid_updateRefinement returns.   (But even that can only
> be expected to be true for leaf blocks, as discussed recently in some
> other flash-users messages.)

So in my calculation the only physics is the Hydro routine, and  
checking the hy_updateSoln routine, the sanitizing calls are in the  
same loops as the hydro updates which only run over the interior  
cells. So since Grid_updateRefinement doesn't sanitize the guard  
cells...and neither does Hydro...nothing prevents the prolong routine  
from using zero-value cells in its calculations. I think the guard  
cells *should* be sanitized prior to running prolong to avoid ending  
up with grid cells where rho < smlrho, the question is where?

I wish I could explain why the code works *now* and didn't work  
before...there were literally no other changes to the code (I haven't  
touched it in 2 weeks) other than the harmless change to the  
gr_updateRefinement routine. It is possible the system I was running  
on was having memory issues that have since been resolved; beyond that  
I am at a loss for explanation.

-- 
James Guillochon
Department of Astronomy & Astrophysics
University of California, Santa Cruz
jfg at ucolick.org

On May 1, 2009, at 8:34 AM, Klaus Weide wrote:

> On Thu, 30 Apr 2009, James Guillochon wrote:
>
>> Problem solved! So it seems the problem was that the  
>> gr_updateRefinement
>> routine does not sanitize the guard cells. So if you happen to  
>> derefine a
>> block that has density values close to smlrho, it is possible that  
>> guard cells
>> will end up with 0 density values and will never be sanitized. This  
>> is
>> resolved by changing a line in gr_updateRefinement from this:
>>
>> call gr_sanitizeDataAfterInterp(gr_blkList, numLeafBlocks, 'after
>> amr_prolong', 0, 0, 0)
>>
>> to this:
>> call gr_sanitizeDataAfterInterp(gr_blkList, numLeafBlocks, 'after
>> amr_prolong', NGUARD, NGUARD, NGUARD)
>
> James,
>
> It is very surprising that you should get such an effect from  
> changing the
> last three arguments to gr_sanitizeDataAfterInterp!  That subroutine  
> (in
> spite of its name) does not modify any data at all, it only does some
> simple checks of variable values after (in this case) prolongation,
> and produces warning messages if it finds suspicious values.
>
> Your change would simply increase the number of cells to which the  
> checks
> apply, from all interior cells to all cells (including guard cells).
>
> Have you perhaps made some other changes at the same time that could
> explain why your problem disappeared?
>
>
> By the way, gr_updateRefinement (or its caller,  
> Grid_updateRefinement) is
> not required to leave *guard cells* in a "sane" state; it is the
> responsibility of any routine that might be called next - in  
> particular,
> any physics unit routine that applies some operator to advance the
> solution - to fill guard cells as needed, usually by calling
> Grid_fillGuardCells.  Interior cells must of course have valid cell
> data after Grid_updateRefinement returns.   (But even that can only
> be expected to be true for leaf blocks, as discussed recently in some
> other flash-users messages.)
>
>
> Finally a caveat, the calling convention for  
> gr_sanitizeDataAfterInterp
> has changed slightly in FLASH3.1 and later; the last three integer
> arguments are now replaced by an integer array.  This shouldn't matter
> as long as you don't mix source files from different releases in the
> same Grid source subtree (definitely not recommended!), but it  
> explains
> why the calls quoted above (from FLASH3.0) look different in the most
> recent code.
>
> Klaus
>
> !DSPAM:10135,49fb1670159511625623900!
>