[FLASH-USERS] unexpected memory increase

[Yingchao Lu]

Instead of using impi, I think it would be better to use the default
compiler wrappers for compiling FLASH and HYPRE. The environment variables
for compiling HYPRE is
export FC=ftn
export CC=cc
export CXX=CC

I tested in Prg-Env-intel environment and there is be no memory leakage.

Since the hdf5 1.8.x were removed on NERSC, I have to use 1.10.x. I applied
the the following patch provided by Klaus and the IO works well.
http://flash.uchicago.edu/pipermail/flash-users/2017-December/002531.html

Best,
Yingchao

On Thu, Oct 25, 2018 at 10:22 AM Yingchao Lu <yingchao.lu at gmail.com> wrote:

> I looked into the memory leakage by using valgrind and figured out that
> all the leakages happens in HYPRE library. The HYPRE library I used is in
> the cray-tpsl module on CORI. Then I compiled HYPRE instead of using the
> module on the machine. I used the impi(see the link) instead of the default
> compiler and wrapper.
>
> The modules I load to compile HYPRE and FLASH:
> module remove PrgEnv-intel
> module load impi
> module load intel
> module load cray-hdf5-parallel
>
> The link for impi:
>
> http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/#toc-anchor-9
>
> The environment variables for compilers:
> export FC=mpiifort
> export CC= mpiicc
> export CXX=mpiicpc
>
> The memory behavior seems normal after I link the code against the
> self-compiled HYPRE.
>
> --Yingchao
>
> On Thu, Oct 25, 2018 at 9:57 AM Klaus Weide <klaus at flash.uchicago.edu>
> wrote:
>
>> Hi Yingchao.
>>
>> On Tue, 23 Oct 2018, Yingchao Lu wrote:
>>
>> > memory information is printed every 30 steps. At the beginning, it is
>> >
>> >  [ 10-22-2018  14:02:36.279 ] memory: /proc vsize    (MiB):      926.67
>> > (min)       1383.60 (max)       1150.19 (avg)
>> >  [ 10-22-2018  14:02:36.287 ] memory: /proc rss      (MiB):      310.10
>> > (min)        397.40 (max)        367.95 (avg)
>>
>> > It increases a lot by 30 steps
>> >
>> >  [10-22-2018  14:16:48.437 ] memory: /proc vsize    (MiB):     2714.04
>> > (min)      31565.56 (max)      16816.94 (avg)
>> >  [ 10-22-2018  14:16:48.449 ] memory: /proc rss      (MiB):      456.39
>> > (min)        716.37 (max)        585.65 (avg)
>>
>> A jump like that, from before the first time step to after, does not see
>> unusual.
>>
>> > And the last memory info print before the code aborts
>> >  [ 10-22-2018  21:48:59.453 ] memory: /proc vsize    (MiB):     5074.70
>> > (min)      44690.05 (max)      25471.68 (avg)
>> >  [ 10-22-2018  21:48:59.472 ] memory: /proc rss      (MiB):     1316.33
>> > (min)       4799.35 (max)       3162.94 (avg)
>>
>> An increase like that, especially if it happens gradually but at a more
>> or
>> less constant rate, looks like an obvious memory leak.
>>
>> Another thing I notice is that memory use appears very unbalanced -
>> there are large difference between the (min) and (max) numbers.
>> I wonder why that is.
>>
>> > I use the compiler flag -DPOINTER_INTENT_OUT=pointer on nersc. I don't
>> know
>> > if it will be a problem. How should I fix the problem?
>>
>> I don't think that that compiler flag matters.
>>
>> I would try to narrow down the possible source of the memory leak.
>> Maybe eliminate some code units and see whether memory use still
>> increases.
>>
>> Do you have any modified code? I would look there first.
>>
>> Is there a (much) smaller version of the setup and conditions that
>> still shows this behavior? That would make it much easier to narrow
>> down the culprit.
>>
>> Klaus
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20181026/b369fa3a/attachment.htm>