[FLASH-USERS] Improving the load balancing in FLASH

[Mike Zingale]

Brian is right -- the situation is entirely problem dependent.  For cases 
where you are locally refining only a small region of the domain, 
subcycling in time can give you a big boost.  Jonathan and I considered 
cases that were typical to the applications that the FLASH center was 
working on at the time -- large regions were typically refined.

Mike


On Tue, 
8 Sep 2009, Brian O'Shea wrote:

> In all fairness, the statement that Anshu made depends a lot on the type of 
> problem you are using.  For example, the Enzo AMR code is mainly used for 
> astrophysical applications that are gravity-dominated, so most of the action 
> takes place in a very small fraction of the simulation volume, and can use 
> many levels of refinement.  The only way that this sort of problem is 
> tractable is by using adaptive time-stepping.  One of the main complications 
> of such an adaptive time-stepping scheme is keeping all of the mesh levels 
> synched up.  It also makes load-balancing a significant challenge - one is 
> forced to distribute grids on a level-by-level basis, and if only a small 
> fraction of the volume is refined (as in our cosmology simulations...) you 
> destroy grid locality, which increases communication and decreases 
> scalability.
>
> Brian
>
>> This was considered several years ago. Then Mike Zingale and Jonathan
>> Dursi proved that the computational savings from coarse blocks having
>> a coarse time step are too insignificant to be worth pursuing. This is
>> because the computational time in the finer blocks completely
>> dominates that in coarser blocks for the same area of computational
>> domain. I believe that study appeared in some proceedings, if you are
>> interested I can dig up the reference.
>> 
>> Anshu
>> 
>> On Tue, Sep 8, 2009 at 4:17 AM, Ross Parkin<phy1erp at leeds.ac.uk> wrote:
>>> Hello,
>>> 
>>> I've been thinking about the load balancing method used in FLASH. The
>>> timestep used by the hydro units is the same on all refinement levels and 
>>> so
>>> blocks are essentially distributed by paramesh so that all processors have
>>> the roughly the smae number of blocks. Has anyone tried modifying FLASH so
>>> that each refinement level has its own timestep? To do this properly you
>>> would need to modify how PARAMESH distributes blocks amongst processors,
>>> taking account of the relative work done by each block on each refinement
>>> level, i.e. a block on a finer refinement level may need to do twice as 
>>> many
>>> hydro loops before its simulation time equals that of a coarser block.
>>> 
>>> Any ideas guys?
>>> 
>>> Cheers,
>>> 
>>> Ross Parkin
>>> 
>


-----------------------------------------------------------------------------
Michael Zingale (mzingale at mail.astro.sunysb.edu)
Assistant Professor

Dept. of Physics and Astronomy     office: ESS 440
Stony Brook University             phone:  631-632-8225
Stony Brook, NY 11794-3800         web: http://www.astro.sunysb.edu/mzingale
-----------------------------------------------------------------------------