[FLASH-USERS] Improving the load balancing in FLASH

[Anshu Dubey]

I take that back. Here is a simple example to counter my argument. n
blocks at level m, 1 block refined to level m+1. If n >> 1, subcycling
would gain almost a factor of 2.

That said, Ross you are right that there is no simple way to do
subcycling in FLASH right now. If you can quantify the loss in
computational efficiency you are seeing, and if it is likely to be
worth your while to spend some time in coding, we can spend some time
thinking about how to do it.

Anshu

On Tue, Sep 8, 2009 at 8:56 AM, Anshu Dubey<dubey at flash.uchicago.edu> wrote:
> I suppose if you have an extremely unbalanced oct-tree that could work out.
>
> But I'd have to work through an example to be fully convinced of it, I
> have a feeling we will find that the factor of two constraint will
> limit much of gain to be had by subcycling.
>
> Anshu
>
> On Tue, Sep 8, 2009 at 8:43 AM, Mike
> Zingale<mzingale at scotty.ess.sunysb.edu> wrote:
>> Brian is right -- the situation is entirely problem dependent.  For cases
>> where you are locally refining only a small region of the domain, subcycling
>> in time can give you a big boost.  Jonathan and I considered cases that were
>> typical to the applications that the FLASH center was working on at the time
>> -- large regions were typically refined.
>>
>> Mike
>>
>>
>> On Tue, 8 Sep 2009, Brian O'Shea wrote:
>>
>>> In all fairness, the statement that Anshu made depends a lot on the type
>>> of problem you are using.  For example, the Enzo AMR code is mainly used for
>>> astrophysical applications that are gravity-dominated, so most of the action
>>> takes place in a very small fraction of the simulation volume, and can use
>>> many levels of refinement.  The only way that this sort of problem is
>>> tractable is by using adaptive time-stepping.  One of the main complications
>>> of such an adaptive time-stepping scheme is keeping all of the mesh levels
>>> synched up.  It also makes load-balancing a significant challenge - one is
>>> forced to distribute grids on a level-by-level basis, and if only a small
>>> fraction of the volume is refined (as in our cosmology simulations...) you
>>> destroy grid locality, which increases communication and decreases
>>> scalability.
>>>
>>> Brian
>>>
>>>> This was considered several years ago. Then Mike Zingale and Jonathan
>>>> Dursi proved that the computational savings from coarse blocks having
>>>> a coarse time step are too insignificant to be worth pursuing. This is
>>>> because the computational time in the finer blocks completely
>>>> dominates that in coarser blocks for the same area of computational
>>>> domain. I believe that study appeared in some proceedings, if you are
>>>> interested I can dig up the reference.
>>>>
>>>> Anshu
>>>>
>>>> On Tue, Sep 8, 2009 at 4:17 AM, Ross Parkin<phy1erp at leeds.ac.uk> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I've been thinking about the load balancing method used in FLASH. The
>>>>> timestep used by the hydro units is the same on all refinement levels
>>>>> and so
>>>>> blocks are essentially distributed by paramesh so that all processors
>>>>> have
>>>>> the roughly the smae number of blocks. Has anyone tried modifying FLASH
>>>>> so
>>>>> that each refinement level has its own timestep? To do this properly you
>>>>> would need to modify how PARAMESH distributes blocks amongst processors,
>>>>> taking account of the relative work done by each block on each
>>>>> refinement
>>>>> level, i.e. a block on a finer refinement level may need to do twice as
>>>>> many
>>>>> hydro loops before its simulation time equals that of a coarser block.
>>>>>
>>>>> Any ideas guys?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Ross Parkin
>>>>>
>>>
>>
>>
>> -----------------------------------------------------------------------------
>> Michael Zingale (mzingale at mail.astro.sunysb.edu)
>> Assistant Professor
>>
>> Dept. of Physics and Astronomy     office: ESS 440
>> Stony Brook University             phone:  631-632-8225
>> Stony Brook, NY 11794-3800         web: http://www.astro.sunysb.edu/mzingale
>> -----------------------------------------------------------------------------
>>
>>
>