[FLASH-USERS] Load Balancing with the Burn Unit
James Guillochon
jguillochon at cfa.harvard.edu
Fri Apr 25 17:08:13 EDT 2014
Hi Sean, that's fantastic! I'll give it a try and let you know how it works
for me.
On Fri, Apr 25, 2014 at 3:57 PM, Sean Couch <smc at flash.uchicago.edu> wrote:
> Hi James,
>
> I’ve done this before with what I think was good success. It’s a pretty
> straight-forward, if hack-tastic, modification. What I did was to modify
> bn_burner.F90 such that the variable xoktot became a counter of the max
> number of burn sub-steps of any zone in a given block. I use this as an
> estimate of the “work” required to advance the block and this is what will
> go into the Morton curve weighting. So around line 198 in bn_burner.F90 I
> had:
>
> *Line 198* *Line 198* end if end if xoktot = xoktot +
> real(nok) !!xoktot = xoktot + real(nok) !! Custom usage by SMC
> xoktot = max(xoktot, real(nok)) xbadtot = xbadtot + real(nbad)
> xbadtot = xbadtot + real(nbad)
>
>
> I then changed Burn.F90 to simply reset this counter:
>
> *Line 54* *Line 54* use Burn_data, ONLY: bn_nuclearTempMin,
> bn_nuclearTempMax, bn_nuclearDensMin, & use Burn_data, ONLY:
> bn_nuclearTempMin, bn_nuclearTempMax, bn_nuclearDensMin, & &
> bn_nuclearDensMax, bn_nuclearNI56Max, bn_useShockBurn, & &
> bn_nuclearDensMax, bn_nuclearNI56Max, bn_useShockBurn, & &
> bn_smallx, bn_useBurn, bn_meshMe & bn_smallx, bn_useBurn,
> bn_meshMe, xoktot use bn_interface, ONLY : bn_mapNetworkToSpecies,
> bn_burner use bn_interface, ONLY : bn_mapNetworkToSpecies, bn_burner
> *Line 114* *Line 114* ! start the timer ticking ! start the
> timer ticking call Timers_start("burn") call Timers_start("burn")
> ! Restart counter xoktot = 0.0 ! make sure that guardcells are
> up to date ! make sure that guardcells are up to date if (.NOT.
> bn_useShockBurn) then if (.NOT. bn_useShockBurn) then call
> Grid_fillGuardCells(CENTER, ALLDIR) call Grid_fillGuardCells(CENTER,
> ALLDIR) endif endif
>
>
> Now in Burn_computeDt.F90 I accessed some private Grid data (shame on
> me, but hey, I said it was hack-tastic):
>
> *Line 79* *Line 79* solnData, &
> solnData, & dt_burn,
> dt_minloc) dt_burn, dt_minloc) use
> Burn_data, ONLY: bn_enucDtFactor, bn_useBurn, bn_meshMe use Burn_data,
> ONLY: bn_enucDtFactor, bn_useBurn, bn_meshMe, xoktot use
> Driver_interface, ONLY : Driver_abortFlash use Driver_interface, ONLY :
> Driver_abortFlash use tree, ONLY : bflags implicit none implicit
> none #include "constants.h" #include "constants.h" *Line 130* *Line
> 131* ! the inverse of what we want, and then only (un)invert
> that inverse ! the inverse of what we want, and then only
> (un)invert that inverse ! if it is a reasonable number.
> ! if it is a reasonable number. energyRatioInv =
> abs(solnData(ENUC_VAR,i,j,k)) / eint_zone energyRatioInv =
> abs(solnData(ENUC_VAR,i,j,k)) / eint_zone !if
> (energyRatioInv > 1.0e-1) bflags(1,blockid) = 4.0
> bflags(1,blockid) = xoktot #ifdef DTBN_VAR
> solnData(DTBN_VAR,i,j,k) = xoktot #endif if (energyRatioInv
> > dt_tempInv) then if (energyRatioInv > dt_tempInv) then
> dt_tempInv = energyRatioInv dt_tempInv =
> energyRatioInv dt_temp = 1.0 / energyRatioInv
> dt_temp = 1.0 / energyRatioInv
>
> You can ignore DTBN_VAR. That was just a variable I used for diagnostics.
> The key is the bflags array. That comes from the PARAMESH tree data
> module and basically does nothing. It’s just a handy array for this,
> PARAMESH does nothing else with it.
>
> Then, finally, in
> source/Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source/mpi_amr_refine_derefine.F90
> I added the following:
>
> *Line 309* *Line 309* work_block(:) = 0. work_block(:) = 0.
> Do i = 1,lnblocks Do i = 1,lnblocks if
> (nodetype(i).eq.1) then if (nodetype(i).eq.1) then
> work_block(i) = 2. !<<< USER EDIT ! work_block(i) = 2. !<<<
> USER EDIT work_block(i) = max(2.,float(bflags(1,i))) !<<<
> by SMC #ifdef FLASH_DEBUG_AMR #ifdef FLASH_DEBUG_AMR
> lnblocks_leaf = lnblocks_leaf + 1 lnblocks_leaf =
> lnblocks_leaf + 1 #endif #endif
>
>
> This sets the Morton curve weighting parameter for the block to the
> maximum number of burn sub-steps that were required for advancement. In my
> limited experimentation, this straightened out the Morton curve in regions
> of rapid burning nicely. Fewer ‘burning’ blocks per MPI rank. Increased
> the efficiency of the simulations I was running quite a bit. Caveat
> emptor: this all could use some tweaking for your particular application
> and YMMV.
>
> Best regards,
> Sean
>
>
> --------------------------------------------------------
> Sean M. Couch
> Hubble Fellow
> Flash Center for Computational Science
> Department of Astronomy & Astrophysics
> The University of Chicago
> 5747 S Ellis Ave, Jo 315
> Chicago, IL 60637
> (773) 702-3899
> www.flash.uchicago.edu/~smc
>
>
>
>
> On Apr 22, 2014, at 3:00 PM, James Guillochon <jguillochon at cfa.harvard.edu>
> wrote:
>
> Hi all, I'm running a simulation using one of the burning networks, which
> unfortunately is leading to a runtime efficiency of 35% as only ~10% of
> blocks are above the burning network thresholds, but those blocks take
> several times longer than non-burning regions.
>
> I noticed that the only weighting done currently is to give leaf blocks a
> factor of "2", and everything else "1". I think it would be relatively easy
> to count cells that have burned in a block, and then add an additional work
> factor to account for this overhead (say number of cells burned times a
> constant, with a block in which all cells are burning being a factor of
> 5-10 times more expensive). Ideally I'd want FLASH's efficiency to be 80%+
> no matter what the Burn unit is doing.
>
> My question is in implementation: Would it make sense to add to the
> "work_block" (which is in the "tree" module) scaling factor directly in the
> Burn unit? Or is this the wrong place in the code to make this change?
>
> Thanks!
> - James
>
> --
> James Guillochon
> Einstein Fellow at the Harvard-Smithsonian CfA
> jguillochon at cfa.harvard.edu
>
>
>
--
James Guillochon
Einstein Fellow at the Harvard-Smithsonian CfA
jguillochon at cfa.harvard.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20140425/38498c2d/attachment.htm>
More information about the flash-users
mailing list