[FLASH-USERS] unexpected memory increase

[Klaus Weide]

Hi Yingchao.

On Tue, 23 Oct 2018, Yingchao Lu wrote:

> memory information is printed every 30 steps. At the beginning, it is
> 
>  [ 10-22-2018  14:02:36.279 ] memory: /proc vsize    (MiB):      926.67
> (min)       1383.60 (max)       1150.19 (avg)
>  [ 10-22-2018  14:02:36.287 ] memory: /proc rss      (MiB):      310.10
> (min)        397.40 (max)        367.95 (avg)

> It increases a lot by 30 steps
> 
>  [10-22-2018  14:16:48.437 ] memory: /proc vsize    (MiB):     2714.04
> (min)      31565.56 (max)      16816.94 (avg)
>  [ 10-22-2018  14:16:48.449 ] memory: /proc rss      (MiB):      456.39
> (min)        716.37 (max)        585.65 (avg)

A jump like that, from before the first time step to after, does not see 
unusual.

> And the last memory info print before the code aborts
>  [ 10-22-2018  21:48:59.453 ] memory: /proc vsize    (MiB):     5074.70
> (min)      44690.05 (max)      25471.68 (avg)
>  [ 10-22-2018  21:48:59.472 ] memory: /proc rss      (MiB):     1316.33
> (min)       4799.35 (max)       3162.94 (avg)

An increase like that, especially if it happens gradually but at a more or 
less constant rate, looks like an obvious memory leak.

Another thing I notice is that memory use appears very unbalanced -
there are large difference between the (min) and (max) numbers.
I wonder why that is.

> I use the compiler flag -DPOINTER_INTENT_OUT=pointer on nersc. I don't know
> if it will be a problem. How should I fix the problem?

I don't think that that compiler flag matters.

I would try to narrow down the possible source of the memory leak.
Maybe eliminate some code units and see whether memory use still 
increases.

Do you have any modified code? I would look there first.

Is there a (much) smaller version of the setup and conditions that
still shows this behavior? That would make it much easier to narrow
down the culprit.

Klaus