[FLASH-USERS] Filling Guard Cells in Burn

[Klaus Weide]

On Tue, 28 Oct 2008, James Guillochon wrote:
> So I'm using the nuclearBurn unit (FLASH 3.0), and the Grid_fillGuardCells 
> routine at the top of Burn.F90 seems to be taking an inordinate amount of 
> time to run. My simulation has sporadic burning and thus does not need the 
> network to run at every time step, but the fillGuardCells routine is taking 
> most of the processing time even with T << nuclearTempMin!

James,

First of all, I am surprised this takes an "inordinate amount of time".
I would expect that the Grid_fillGuardCells calls in other places, 
predominantly in Hydro, use more time, just because they would be
called more often (at least when NDIM > 1).  I am thinking of a "normal"
FLASH simulation that uses Hydro, and Burn, and probably some other 
physics units.

If you just mean that the Grid_fillGuardCells in Burn dominates the total 
time spent in Burn - that may well be, but see below.

> The question is: Do I have to fill the guard cells if the temperature 
> criteria for burning isn't met? Is there some reason the fillGuardCells 
> routine is taking so long in this context?

I am assuming you are referring to this source file:
    source/physics/sourceTerms/Burn/BurnMain/nuclearBurn/Burn.F90 ,
let me know if I'm wrong!

There has been a code change in this file from FLASH 3.0 to 3.1,
associated with this comment:

=========================================================================================================
Filename: trunk/source/physics/sourceTerms/Burn/BurnMain/nuclearBurn/Burn.F90
Revision 8763
Modified Fri Mar 21 13:28:56 2008 CDT (7 months, 1 week ago) by townsley
File length: 9383 byte(s)
Diff to previous 7876

[...]
Also limited nuclear Burn calculations to only interior cells,
no need to do the guard cells.
=========================================================================================================

Here is the diff:

=========================================================================================================
*** trunk/source/physics/sourceTerms/Burn/BurnMain/nuclearBurn/Burn.F90	2008/01/07 16:23:52	7876
--- trunk/source/physics/sourceTerms/Burn/BurnMain/nuclearBurn/Burn.F90	2008/03/21 18:28:56	8763
***************
*** 150,158 ****


        ! now guaranteed that tmp, rho, etc. exist
!      do k = blkLimitsGC(LOW,KAXIS), blkLimitsGC(HIGH,KAXIS)
!         do j = blkLimitsGC(LOW,JAXIS), blkLimitsGC(HIGH,JAXIS)
!            do i = blkLimitsGC(LOW,IAXIS), blkLimitsGC(HIGH,IAXIS)
                 okBurnTemp = .FALSE.
                 okBurnDens = .FALSE.
                 okBurnShock = .FALSE.
--- 150,158 ----


        ! now guaranteed that tmp, rho, etc. exist
!      do k = blkLimits(LOW,KAXIS), blkLimits(HIGH,KAXIS)
!         do j = blkLimits(LOW,JAXIS), blkLimits(HIGH,JAXIS)
!            do i = blkLimits(LOW,IAXIS), blkLimits(HIGH,IAXIS)
                 okBurnTemp = .FALSE.
                 okBurnDens = .FALSE.
                 okBurnShock = .FALSE.
=========================================================================================================

(There were also changes in that update related to enabling Burn_computeDt. If you use
Burn_computeDt, it probably should have the same kind of blkLimitsGC -> blkLimits change.)

I believe that with the FLASH 3.1 version, the only reason for having a Grid_fillGuardCells call
at the top of Burn.F90 is to provide valid data for Hydro_detectShock.  (That's why Grid_fillGuardCells
is only called 'if (.NOT. bn_useShockBurn)').

If you look at the code of Hydro_detectShock.F90, you will find that it requires only 
a limited number of guard cell layers (1 or 2?) for its stencil, and it refers only to
a small number of UNK variables (VEL{X,Y,Z}_VAR and PRES_VAR).  You could therefore
try to speed up the Grid_fillGuardCells calls somewhat by limiting the number of
cell layers and/or variables that are exchanged - see Grid_fillGuardCells.F90 for
the various OPTIONAL arguments that may apply.  To restrict the set of variables
that are exchanged, also make sure the runtime parameter
   enableMaskedGcFill
is TRUE.


There is currently no way to limit guard cell exchanges only to some 
blocks or processors that may need them - it's an all-or-nothing thing in 
that way.  But if you know of some criteria to determine globally that in 
a given invocation of Burn, there is no need to call Hydro_detectShock on 
ANY block in the simulation (for example, because you can determine 
positively that the Burn variables okBurnTemp .AND. okBurnDens will be 
FALSE in all cells in all leaf blocks) - then you can of course skip the 
call in that invocation of Burn.



Klaus