<div dir="ltr">Hi Sean, that's fantastic! I'll give it a try and let you know how it works for me.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 25, 2014 at 3:57 PM, Sean Couch <span dir="ltr"><<a href="mailto:smc@flash.uchicago.edu" target="_blank">smc@flash.uchicago.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi James,<div><br></div><div>I’ve done this before with what I think was good success. It’s a pretty straight-forward, if hack-tastic, modification. What I did was to modify bn_burner.F90 such that the variable xoktot became a counter of the max number of burn sub-steps of any zone in a given block. I use this as an estimate of the “work” required to advance the block and this is what will go into the Morton curve weighting. So around line 198 in bn_burner.F90 I had:</div>
<div><br></div><div><div><table border="0" cellspacing="0" cellpadding="0" width="100%" style="font-family:Times"><tbody><tr style="background-color:rgb(153,204,204)"><td width="50%"><table width="100%" border="1" cellpadding="5">
<tbody><tr><td><b>Line 198</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 198</b> <span style="font-size:smaller"></span></td>
</tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> end if</td><td style="font-family:sans-serif;font-size:smaller"> end if</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
</td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
xoktot = xoktot + real(nok)</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> !!xoktot = xoktot + real(nok)</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> !! Custom usage by SMC</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller"> </td>
<td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> xoktot = max(xoktot, real(nok))</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> xbadtot = xbadtot + real(nbad)</td>
<td style="font-family:sans-serif;font-size:smaller"> xbadtot = xbadtot + real(nbad)</td></tr></tbody></table><div><br></div></div><div><br></div><div>I then changed Burn.F90 to simply reset this counter:</div><div><br>
</div><div><table border="0" cellspacing="0" cellpadding="0" width="100%" style="font-family:Times"><tbody><tr style="background-color:rgb(153,204,204)"><td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr>
<td><b>Line 54</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 54</b> <span style="font-size:smaller"></span></td>
</tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> use Burn_data, ONLY: bn_nuclearTempMin, bn_nuclearTempMax, bn_nuclearDensMin, &</td>
<td style="font-family:sans-serif;font-size:smaller"> use Burn_data, ONLY: bn_nuclearTempMin, bn_nuclearTempMax, bn_nuclearDensMin, &</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> & bn_nuclearDensMax, bn_nuclearNI56Max, bn_useShockBurn, &</td>
<td style="font-family:sans-serif;font-size:smaller"> & bn_nuclearDensMax, bn_nuclearNI56Max, bn_useShockBurn, &</td></tr><tr><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
& bn_smallx, bn_useBurn, bn_meshMe</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> & bn_smallx, bn_useBurn, bn_meshMe, xoktot</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
use bn_interface, ONLY : bn_mapNetworkToSpecies, bn_burner</td><td style="font-family:sans-serif;font-size:smaller"> use bn_interface, ONLY : bn_mapNetworkToSpecies, bn_burner</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
</td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr style="background-color:rgb(153,204,204)">
<td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 114</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%"><table width="100%" border="1" cellpadding="5">
<tbody><tr><td><b>Line 114</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> ! start the timer ticking</td><td style="font-family:sans-serif;font-size:smaller">
! start the timer ticking</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> call Timers_start("burn")</td><td style="font-family:sans-serif;font-size:smaller"> call Timers_start("burn")</td>
</tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="background-color:rgb(204,204,204);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(170,255,170);font-family:sans-serif;font-size:smaller"> ! Restart counter</td></tr><tr><td style="background-color:rgb(204,204,204);font-family:sans-serif;font-size:smaller"> </td>
<td style="background-color:rgb(170,255,170);font-family:sans-serif;font-size:smaller">
xoktot = 0.0</td></tr><tr><td style="background-color:rgb(204,204,204);font-family:sans-serif;font-size:smaller"> </td><td style="background-color:rgb(170,255,170);font-family:sans-serif;font-size:smaller"> </td></tr>
<tr>
<td style="font-family:sans-serif;font-size:smaller"> ! make sure that guardcells are up to date</td><td style="font-family:sans-serif;font-size:smaller"> ! make sure that guardcells are up to date</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
if (.NOT. bn_useShockBurn) then</td><td style="font-family:sans-serif;font-size:smaller"> if (.NOT. bn_useShockBurn) then</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller">
</td></tr><tr><td style="background-color:rgb(255,170,170);font-family:sans-serif;font-size:smaller"> </td><td style="background-color:rgb(204,204,204);font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
call Grid_fillGuardCells(CENTER, ALLDIR)</td><td style="font-family:sans-serif;font-size:smaller"> call Grid_fillGuardCells(CENTER, ALLDIR)</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td>
<td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> endif</td><td style="font-family:sans-serif;font-size:smaller"> endif<br><br></td></tr></tbody></table>
</div><div><br></div><div>
<div style="text-align:-webkit-auto;text-indent:0px;word-wrap:break-word"><span style="border-collapse:separate;border-spacing:0px"><span style="border-collapse:separate;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word">
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Now in Burn_computeDt.F90 I accessed some private Grid data (shame on me, but hey, I said it was hack-tastic):</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><br></div><div><table border="0" cellspacing="0" cellpadding="0" width="100%" style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Times;font-weight:normal;word-spacing:0px">
<tbody><tr style="background-color:rgb(153,204,204)"><td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 79</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%">
<table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 79</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> solnData, &</td>
<td style="font-family:sans-serif;font-size:smaller"> solnData, &</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> dt_burn, dt_minloc)</td><td style="font-family:sans-serif;font-size:smaller">
dt_burn, dt_minloc)</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td><td style="font-family:sans-serif;font-size:smaller"> </td></tr><tr><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
use Burn_data, ONLY: bn_enucDtFactor, bn_useBurn, bn_meshMe</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> use Burn_data, ONLY: bn_enucDtFactor, bn_useBurn, bn_meshMe, xoktot</td>
</tr><tr><td style="font-family:sans-serif;font-size:smaller"> use Driver_interface, ONLY : Driver_abortFlash</td><td style="font-family:sans-serif;font-size:smaller"> use Driver_interface, ONLY : Driver_abortFlash</td>
</tr><tr><td style="background-color:rgb(204,204,204);font-family:sans-serif;font-size:smaller"> </td><td style="background-color:rgb(170,255,170);font-family:sans-serif;font-size:smaller"> use tree, ONLY : bflags</td>
</tr>
<tr><td style="font-family:sans-serif;font-size:smaller"> implicit none</td><td style="font-family:sans-serif;font-size:smaller"> implicit none</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> </td>
<td style="font-family:sans-serif;font-size:smaller">
</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> #include "constants.h"</td><td style="font-family:sans-serif;font-size:smaller"> #include "constants.h"</td></tr><tr style="background-color:rgb(153,204,204)">
<td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 130</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%"><table width="100%" border="1" cellpadding="5">
<tbody><tr><td><b>Line 131</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> ! the inverse of what we want, and then only (un)invert that inverse</td>
<td style="font-family:sans-serif;font-size:smaller"> ! the inverse of what we want, and then only (un)invert that inverse</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> ! if it is a reasonable number.</td>
<td style="font-family:sans-serif;font-size:smaller"> ! if it is a reasonable number.</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> energyRatioInv = abs(solnData(ENUC_VAR,i,j,k)) / eint_zone</td>
<td style="font-family:sans-serif;font-size:smaller"> energyRatioInv = abs(solnData(ENUC_VAR,i,j,k)) / eint_zone</td></tr><tr><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> !if (energyRatioInv > 1.0e-1) bflags(1,blockid) = 4.0</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> bflags(1,blockid) = xoktot</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> #ifdef DTBN_VAR</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller"> </td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
solnData(DTBN_VAR,i,j,k) = xoktot</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller"> </td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
#endif</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> if (energyRatioInv > dt_tempInv) then</td><td style="font-family:sans-serif;font-size:smaller"> if (energyRatioInv > dt_tempInv) then</td>
</tr><tr><td style="font-family:sans-serif;font-size:smaller"> dt_tempInv = energyRatioInv</td><td style="font-family:sans-serif;font-size:smaller"> dt_tempInv = energyRatioInv</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
dt_temp = 1.0 / energyRatioInv</td><td style="font-family:sans-serif;font-size:smaller"> dt_temp = 1.0 / energyRatioInv</td></tr></tbody></table><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<br></div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">You can ignore DTBN_VAR. That was just a variable I used for diagnostics. The key is the bflags array. That comes from the PARAMESH tree data module and basically does nothing. It’s just a handy array for this, PARAMESH does nothing else with it.</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><br></div><div>Then, finally, in <span style="text-align:-webkit-auto">source/Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source/m</span><span style="text-align:-webkit-auto">pi_amr_refine_derefine.F90 I added the following:</span></div>
<div><span style="text-align:-webkit-auto"><br></span></div><div><table border="0" cellspacing="0" cellpadding="0" width="100%" style="font-family:Times"><tbody><tr style="background-color:rgb(153,204,204)"><td width="50%">
<table width="100%" border="1" cellpadding="5"><tbody><tr><td><b>Line 309</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td><td width="50%"><table width="100%" border="1" cellpadding="5"><tbody><tr>
<td><b>Line 309</b> <span style="font-size:smaller"></span></td></tr></tbody></table></td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> work_block(:) = 0.</td><td style="font-family:sans-serif;font-size:smaller">
work_block(:) = 0.</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> Do i = 1,lnblocks</td><td style="font-family:sans-serif;font-size:smaller"> Do i = 1,lnblocks</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
if (nodetype(i).eq.1) then</td><td style="font-family:sans-serif;font-size:smaller"> if (nodetype(i).eq.1) then</td></tr><tr><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller">
work_block(i) = 2. !<<< USER EDIT</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> ! work_block(i) = 2. !<<< USER EDIT</td></tr><tr><td style="background-color:rgb(238,238,119);font-family:sans-serif;font-size:smaller">
</td><td style="background-color:rgb(255,255,119);font-family:sans-serif;font-size:smaller"> work_block(i) = max(2.,float(bflags(1,i))) !<<< by SMC</td></tr><tr><td style="font-family:sans-serif;font-size:smaller">
#ifdef FLASH_DEBUG_AMR</td><td style="font-family:sans-serif;font-size:smaller"> #ifdef FLASH_DEBUG_AMR</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> lnblocks_leaf = lnblocks_leaf + 1</td>
<td style="font-family:sans-serif;font-size:smaller"> lnblocks_leaf = lnblocks_leaf + 1</td></tr><tr><td style="font-family:sans-serif;font-size:smaller"> #endif</td><td style="font-family:sans-serif;font-size:smaller">
#endif<br><br></td></tr></tbody></table></div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<br></div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">This sets the Morton curve weighting parameter for the block to the maximum number of burn sub-steps that were required for advancement. In my limited experimentation, this straightened out the Morton curve in regions of rapid burning nicely. Fewer ‘burning’ blocks per MPI rank. Increased the efficiency of the simulations I was running quite a bit. Caveat emptor: this all could use some tweaking for your particular application and YMMV.</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><br></div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
Best regards,</div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Sean</div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<br></div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><br></div>--------------------------------------------------------</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Sean M. Couch</div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
Hubble Fellow</div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Flash Center for Computational Science</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Department of Astronomy & Astrophysics</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">The University of Chicago</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
5747 S Ellis Ave, Jo 315</div><div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">Chicago, IL 60637</div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><a href="tel:%28773%29%20702-3899" value="+17737023899" target="_blank">(773) 702-3899</a></div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><a href="http://www.flash.uchicago.edu/~smc" target="_blank">www.flash.uchicago.edu/~smc</a></div>
<div style="line-height:normal;letter-spacing:normal;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px"><br></div></div></span></span></div>
<br><br>
</div><div><div class="h5">
<br><div><div>On Apr 22, 2014, at 3:00 PM, James Guillochon <<a href="mailto:jguillochon@cfa.harvard.edu" target="_blank">jguillochon@cfa.harvard.edu</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">Hi all, I'm running a simulation using one of the burning networks, which unfortunately is leading to a runtime efficiency of 35% as only ~10% of blocks are above the burning network thresholds, but those blocks take several times longer than non-burning regions.<div>
<br></div><div>I noticed that the only weighting done currently is to give leaf blocks a factor of "2", and everything else "1". I think it would be relatively easy to count cells that have burned in a block, and then add an additional work factor to account for this overhead (say number of cells burned times a constant, with a block in which all cells are burning being a factor of 5-10 times more expensive). Ideally I'd want FLASH's efficiency to be 80%+ no matter what the Burn unit is doing.</div>
<div><br></div><div>My question is in implementation: Would it make sense to add to the "work_block" (which is in the "tree" module) scaling factor directly in the Burn unit? Or is this the wrong place in the code to make this change?</div>
<div><br></div><div>Thanks!</div><div>- James<br clear="all"><div><br></div>-- <br><div dir="ltr">James Guillochon<br><span style="font-family:arial;font-size:small">Einstein Fellow at the Harvard-Smithsonian CfA</span><br>
<a href="mailto:jguillochon@cfa.harvard.edu" target="_blank">jguillochon@cfa.harvard.edu</a><br></div>
</div></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">James Guillochon<br><span style="font-family:arial;font-size:small">Einstein Fellow at the Harvard-Smithsonian CfA</span><br>
<a href="mailto:jguillochon@cfa.harvard.edu" target="_blank">jguillochon@cfa.harvard.edu</a><br></div>
</div>