[FLASH-USERS] unexpected memory increase

[Yingchao Lu]

I looked into the memory leakage by using valgrind and figured out that all
the leakages happens in HYPRE library. The HYPRE library I used is in the
cray-tpsl module on CORI. Then I compiled HYPRE instead of using the module
on the machine. I used the impi(see the link) instead of the default
compiler and wrapper.

The modules I load to compile HYPRE and FLASH:
module remove PrgEnv-intel
module load impi
module load intel
module load cray-hdf5-parallel

The link for impi:
http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/#toc-anchor-9

The environment variables for compilers:
export FC=mpiifort
export CC= mpiicc
export CXX=mpiicpc

The memory behavior seems normal after I link the code against the
self-compiled HYPRE.

--Yingchao

On Thu, Oct 25, 2018 at 9:57 AM Klaus Weide <klaus at flash.uchicago.edu>
wrote:

> Hi Yingchao.
>
> On Tue, 23 Oct 2018, Yingchao Lu wrote:
>
> > memory information is printed every 30 steps. At the beginning, it is
> >
> >  [ 10-22-2018  14:02:36.279 ] memory: /proc vsize    (MiB):      926.67
> > (min)       1383.60 (max)       1150.19 (avg)
> >  [ 10-22-2018  14:02:36.287 ] memory: /proc rss      (MiB):      310.10
> > (min)        397.40 (max)        367.95 (avg)
>
> > It increases a lot by 30 steps
> >
> >  [10-22-2018  14:16:48.437 ] memory: /proc vsize    (MiB):     2714.04
> > (min)      31565.56 (max)      16816.94 (avg)
> >  [ 10-22-2018  14:16:48.449 ] memory: /proc rss      (MiB):      456.39
> > (min)        716.37 (max)        585.65 (avg)
>
> A jump like that, from before the first time step to after, does not see
> unusual.
>
> > And the last memory info print before the code aborts
> >  [ 10-22-2018  21:48:59.453 ] memory: /proc vsize    (MiB):     5074.70
> > (min)      44690.05 (max)      25471.68 (avg)
> >  [ 10-22-2018  21:48:59.472 ] memory: /proc rss      (MiB):     1316.33
> > (min)       4799.35 (max)       3162.94 (avg)
>
> An increase like that, especially if it happens gradually but at a more or
> less constant rate, looks like an obvious memory leak.
>
> Another thing I notice is that memory use appears very unbalanced -
> there are large difference between the (min) and (max) numbers.
> I wonder why that is.
>
> > I use the compiler flag -DPOINTER_INTENT_OUT=pointer on nersc. I don't
> know
> > if it will be a problem. How should I fix the problem?
>
> I don't think that that compiler flag matters.
>
> I would try to narrow down the possible source of the memory leak.
> Maybe eliminate some code units and see whether memory use still
> increases.
>
> Do you have any modified code? I would look there first.
>
> Is there a (much) smaller version of the setup and conditions that
> still shows this behavior? That would make it much easier to narrow
> down the culprit.
>
> Klaus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20181025/0d3ec0d6/attachment.htm>