[FLASH-USERS] FLASH memory usage: Static, Dynamic and Machine Specific

[Carlo Graziani]

Actually, the old solver is considerably less accurate than the new one,
particularly for large multipole cutoff values.  In fact, the old solver -- the
old algorithm, for that matter -- fails altogether to converge in l. The new
solver does in fact converge.

Furthermore, the optimization of the center of the expansion implemented in the new
solver is designed to accelerate the convergence with l, so that it should not
be necessary to go to lmax=60 in most cases.  All this is described in
https://arxiv.org/abs/1307.3135.

Note also that if you're running a 3D problem, the array sizes scale as (lmax+1)^2,
so part of your problem may be due to choosing an excessive lmax.  You could
try cutting it down a bit, to see if the problem fits.

Carlo

On 08/11/2016 11:47 AM, Tomasz Plewa wrote:
> Why not to use the original multipole solver? It is not really any less
> accurate.
>
> Tomek
> --
> On 08/11/16 12:41, Klaus Weide wrote:
>> On Wed, 10 Aug 2016, Rahul Kashyap wrote:
>>
>>> Yes, I forgot to mention that I'm using new multipole implementation with
>>> 60 poles.
>>>
>>> I have attached a small txt files with short summary on three runs which
>>> very well describes my problem. 1024 proc have been used for all runs with
>>> fixed lrefinemax and base blocks. I get three differenet error for three
>>> different maxblocks value.
>>>
>>> My understanding was that reasonable use of maxblocks avoids any such
>>> memory failures.
>> Rahul,
>>
>> It appears that the total memory required by
>>
>>    PARAMESH Grid + multipole solver ( + Particles + other units )
>>
>> is just too much; I suspect that this is PRIMARILY due to the memory
>> requirements of the Multipole(_new) solver.
>>
>> There are several large arrays allocated, see in particular statements
>> like
>>
>>    allocate (gr_mpoleScratch (1:gr_mpoleMaxLM,1:gr_mpoleMaxQ  ),...)
>>
>> in gr_mpoleAllocateRadialArrays.F90, where gr_mpoleMaxQ may be very large
>> and gr_mpoleMaxLM ~ gr_mpoleMaxL ** 2 ( == 60**2 in your case?).
>>
>> Unfortunately this memory is required for each process, you cannot reduce
>> this by running on more procs.
>>
>> It makes sense in general to try to reduce the memory required by the
>> PARAMESH Grid by lowering maxblocks, but this can go only so far;
>> maxblocks has to leave room for "a few" more blocks than the number
>> actually required by the distributed grid. These additional slots
>> are needed for temporary storage during processing by some internal
>> PARAMESH routines for things like block redistribution. I don't
>> know of a reliable way to predict how low "a few" can go in a given
>> case, so this has to be determined empirically. Apparently,
>> this was too low in your maxblocks=5 case.
>>
>> It may be possible to tweak the maxblocks value further, possibly in
>> connection with also modifying the values of maxblocks_alloc and
>> maxblocks_tr (see amr_initialize.F90 and paramesh_dimensions.F90),
>> in order to allow the Grid domain initialization to proceed with
>> maxblocks < 10; but this may then still not give you enough free memory
>> for the multipole solver (and particles, etc).
>>
>> So you should investigate ways to lower the memory requirements of the
>> Poisson solver; you may have to lower the resolution (not sure which
>> runtime parameters to change), or perhaps use a different implementation.
>>
>> Klaus
>


-- 
Carlo Graziani                                 (773) 702-7973 (Voice)
University of Chicago Flash Center             (773) 702-6645 (FAX)
5747 South Ellis Avenue     -------------------------------------
Jones 314                  | The less a statesman amounts to, the
Chicago, IL 60637          | more he loves the flag.
carlo at oddjob.uchicago.edu  | 		   --- Kin Hubbard