[FLASH-USERS] Improving the load balancing in FLASH

[Ross Parkin]

I can see how the choice is problem specific. Sadly, the problem I'm
working on at the minute requires a few levels more refinement in a
very small fraction of the grid and the use of a global timestep is
making the simulation run quite slowly.

I had a quick look in the paramesh manual and they do discuss using
paramesh with different timesteps on different refinement levels. As
far as I can see this would involve altering how FLASH (or the
paramesh bits of FLASH to be more exact) does the loadbalancing, then
set the hydro loop to run with a level dependent timestep. I was
wondering if anyone had attempted to re-jig FLASH to do this, but from
the sounds of things it's a bigger task than I first anticipated.

Thanks for the input,

Ross

Quoting Anshu Dubey <dubey at flash.uchicago.edu>:

> Brian,
>
> The statement pertains to Paramesh as much as the problem itself, I
> should have stated that in my earlier email.  Paramesh is Oct-tree
> based and has the requirement that adjacent blocks cannot be more than
> a factor of two different in their resolution. Enzo I believe is patch
> based where these restriction don't apply, and therefore my earlier
> statement may not be valid for many problems.
>
> Anshu
>
> On Tue, Sep 8, 2009 at 8:35 AM, Brian O'Shea<oshea at msu.edu> wrote:
>> In all fairness, the statement that Anshu made depends a lot on the type of
>> problem you are using.  For example, the Enzo AMR code is mainly used for
>> astrophysical applications that are gravity-dominated, so most of the action
>> takes place in a very small fraction of the simulation volume, and can use
>> many levels of refinement.  The only way that this sort of problem is
>> tractable is by using adaptive time-stepping.  One of the main complications
>> of such an adaptive time-stepping scheme is keeping all of the mesh levels
>> synched up.  It also makes load-balancing a significant challenge - one is
>> forced to distribute grids on a level-by-level basis, and if only a small
>> fraction of the volume is refined (as in our cosmology simulations...) you
>> destroy grid locality, which increases communication and decreases
>> scalability.
>>
>> Brian
>>
>>> This was considered several years ago. Then Mike Zingale and Jonathan
>>> Dursi proved that the computational savings from coarse blocks having
>>> a coarse time step are too insignificant to be worth pursuing. This is
>>> because the computational time in the finer blocks completely
>>> dominates that in coarser blocks for the same area of computational
>>> domain. I believe that study appeared in some proceedings, if you are
>>> interested I can dig up the reference.
>>>
>>> Anshu
>>>
>>> On Tue, Sep 8, 2009 at 4:17 AM, Ross Parkin<phy1erp at leeds.ac.uk> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I've been thinking about the load balancing method used in FLASH. The
>>>> timestep used by the hydro units is the same on all refinement levels and
>>>> so
>>>> blocks are essentially distributed by paramesh so that all processors
>>>> have
>>>> the roughly the smae number of blocks. Has anyone tried modifying FLASH
>>>> so
>>>> that each refinement level has its own timestep? To do this properly you
>>>> would need to modify how PARAMESH distributes blocks amongst processors,
>>>> taking account of the relative work done by each block on each refinement
>>>> level, i.e. a block on a finer refinement level may need to do twice as
>>>> many
>>>> hydro loops before its simulation time equals that of a coarser block.
>>>>
>>>> Any ideas guys?
>>>>
>>>> Cheers,
>>>>
>>>> Ross Parkin
>>>>
>>>
>>
>>
>




----- End forwarded message -----