[FLASH-USERS] FLASH memory usage: Static, Dynamic and Machine Specific

[Carlo Graziani]

I agree that multipole self-gravity is not a good choice for "general"
mass distributions.  But for distributions with large central mass concentrations --
basically, stars -- multipole self-gravity is clearly an optimal choice that should
lead to a better balance of accuracy and computational efficiency than multigrid.

Carlo

On 08/11/2016 12:10 PM, Tomasz Plewa wrote:
> Accuracy and functionality does not necessarily come together. If one
> really needs accurate solutions for general mass distributions, there is
> no more accurate method than the multigrid.
>
> Tomek
> --
> On 08/11/16 13:00, Carlo Graziani wrote:
>> Actually, the old solver is considerably less accurate than the new one,
>> particularly for large multipole cutoff values.  In fact, the old solver -- the
>> old algorithm, for that matter -- fails altogether to converge in l. The new
>> solver does in fact converge.
>>
>> Furthermore, the optimization of the center of the expansion implemented in the new
>> solver is designed to accelerate the convergence with l, so that it should not
>> be necessary to go to lmax=60 in most cases.  All this is described in
>> https://arxiv.org/abs/1307.3135.
>>
>> Note also that if you're running a 3D problem, the array sizes scale as (lmax+1)^2,
>> so part of your problem may be due to choosing an excessive lmax.  You could
>> try cutting it down a bit, to see if the problem fits.
>>
>> Carlo
>>
>> On 08/11/2016 11:47 AM, Tomasz Plewa wrote:
>>> Why not to use the original multipole solver? It is not really any less
>>> accurate.
>>>
>>> Tomek
>>> --
>>> On 08/11/16 12:41, Klaus Weide wrote:
>>>> On Wed, 10 Aug 2016, Rahul Kashyap wrote:
>>>>
>>>>> Yes, I forgot to mention that I'm using new multipole implementation with
>>>>> 60 poles.
>>>>>
>>>>> I have attached a small txt files with short summary on three runs which
>>>>> very well describes my problem. 1024 proc have been used for all runs with
>>>>> fixed lrefinemax and base blocks. I get three differenet error for three
>>>>> different maxblocks value.
>>>>>
>>>>> My understanding was that reasonable use of maxblocks avoids any such
>>>>> memory failures.
>>>> Rahul,
>>>>
>>>> It appears that the total memory required by
>>>>
>>>>     PARAMESH Grid + multipole solver ( + Particles + other units )
>>>>
>>>> is just too much; I suspect that this is PRIMARILY due to the memory
>>>> requirements of the Multipole(_new) solver.
>>>>
>>>> There are several large arrays allocated, see in particular statements
>>>> like
>>>>
>>>>     allocate (gr_mpoleScratch (1:gr_mpoleMaxLM,1:gr_mpoleMaxQ  ),...)
>>>>
>>>> in gr_mpoleAllocateRadialArrays.F90, where gr_mpoleMaxQ may be very large
>>>> and gr_mpoleMaxLM ~ gr_mpoleMaxL ** 2 ( == 60**2 in your case?).
>>>>
>>>> Unfortunately this memory is required for each process, you cannot reduce
>>>> this by running on more procs.
>>>>
>>>> It makes sense in general to try to reduce the memory required by the
>>>> PARAMESH Grid by lowering maxblocks, but this can go only so far;
>>>> maxblocks has to leave room for "a few" more blocks than the number
>>>> actually required by the distributed grid. These additional slots
>>>> are needed for temporary storage during processing by some internal
>>>> PARAMESH routines for things like block redistribution. I don't
>>>> know of a reliable way to predict how low "a few" can go in a given
>>>> case, so this has to be determined empirically. Apparently,
>>>> this was too low in your maxblocks=5 case.
>>>>
>>>> It may be possible to tweak the maxblocks value further, possibly in
>>>> connection with also modifying the values of maxblocks_alloc and
>>>> maxblocks_tr (see amr_initialize.F90 and paramesh_dimensions.F90),
>>>> in order to allow the Grid domain initialization to proceed with
>>>> maxblocks < 10; but this may then still not give you enough free memory
>>>> for the multipole solver (and particles, etc).
>>>>
>>>> So you should investigate ways to lower the memory requirements of the
>>>> Poisson solver; you may have to lower the resolution (not sure which
>>>> runtime parameters to change), or perhaps use a different implementation.
>>>>
>>>> Klaus
>>
>


-- 
Carlo Graziani                                 (773) 702-7973 (Voice)
University of Chicago Flash Center             (773) 702-6645 (FAX)
5747 South Ellis Avenue     -------------------------------------
Jones 314                  | The less a statesman amounts to, the
Chicago, IL 60637          | more he loves the flag.
carlo at oddjob.uchicago.edu  | 		   --- Kin Hubbard