<div dir="ltr"><div dir="ltr">Instead of using impi, I think it would be better to use the default compiler wrappers for compiling FLASH and HYPRE. The environment variables for compiling HYPRE is<div><div>export FC=ftn</div><div>export CC=cc</div><div>export CXX=CC</div></div><div><br></div><div>I tested in Prg-Env-intel environment and there is be no memory leakage. </div><div><br></div><div>Since the hdf5 1.8.x were removed on NERSC, I have to use 1.10.x. I applied the the following patch provided by Klaus and the IO works well.</div><div><a href="http://flash.uchicago.edu/pipermail/flash-users/2017-December/002531.html">http://flash.uchicago.edu/pipermail/flash-users/2017-December/002531.html</a></div><div><br></div><div>Best,</div><div>Yingchao</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 25, 2018 at 10:22 AM Yingchao Lu <<a href="mailto:yingchao.lu@gmail.com">yingchao.lu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">I looked into the memory leakage by using valgrind and figured out that all the leakages happens in HYPRE library. The HYPRE library I used is in the cray-tpsl module on CORI. Then I compiled HYPRE instead of using the module on the machine. I used the impi(see the link) instead of the default compiler and wrapper. <div><br></div><div>The modules I load to compile HYPRE and FLASH:</div><div><div>module remove PrgEnv-intel</div><div>module load impi</div><div>module load intel</div><div>module load cray-hdf5-parallel</div></div><div><br></div><div>The link for impi:</div><div><a href="http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/#toc-anchor-9" target="_blank">http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/#toc-anchor-9</a></div><div><br></div><div>The environment variables for compilers:</div><div><div>export FC=mpiifort</div><div>export CC= mpiicc</div><div>export CXX=mpiicpc</div></div><div><br></div><div>The memory behavior seems normal after I link the code against the self-compiled HYPRE.<br></div><div><br></div><div>--Yingchao</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 25, 2018 at 9:57 AM Klaus Weide <<a href="mailto:klaus@flash.uchicago.edu" target="_blank">klaus@flash.uchicago.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Yingchao.<br>
<br>
On Tue, 23 Oct 2018, Yingchao Lu wrote:<br>
<br>
> memory information is printed every 30 steps. At the beginning, it is<br>
> <br>
> [ 10-22-2018 14:02:36.279 ] memory: /proc vsize (MiB): 926.67<br>
> (min) 1383.60 (max) 1150.19 (avg)<br>
> [ 10-22-2018 14:02:36.287 ] memory: /proc rss (MiB): 310.10<br>
> (min) 397.40 (max) 367.95 (avg)<br>
<br>
> It increases a lot by 30 steps<br>
> <br>
> [10-22-2018 14:16:48.437 ] memory: /proc vsize (MiB): 2714.04<br>
> (min) 31565.56 (max) 16816.94 (avg)<br>
> [ 10-22-2018 14:16:48.449 ] memory: /proc rss (MiB): 456.39<br>
> (min) 716.37 (max) 585.65 (avg)<br>
<br>
A jump like that, from before the first time step to after, does not see <br>
unusual.<br>
<br>
> And the last memory info print before the code aborts<br>
> [ 10-22-2018 21:48:59.453 ] memory: /proc vsize (MiB): 5074.70<br>
> (min) 44690.05 (max) 25471.68 (avg)<br>
> [ 10-22-2018 21:48:59.472 ] memory: /proc rss (MiB): 1316.33<br>
> (min) 4799.35 (max) 3162.94 (avg)<br>
<br>
An increase like that, especially if it happens gradually but at a more or <br>
less constant rate, looks like an obvious memory leak.<br>
<br>
Another thing I notice is that memory use appears very unbalanced -<br>
there are large difference between the (min) and (max) numbers.<br>
I wonder why that is.<br>
<br>
> I use the compiler flag -DPOINTER_INTENT_OUT=pointer on nersc. I don't know<br>
> if it will be a problem. How should I fix the problem?<br>
<br>
I don't think that that compiler flag matters.<br>
<br>
I would try to narrow down the possible source of the memory leak.<br>
Maybe eliminate some code units and see whether memory use still <br>
increases.<br>
<br>
Do you have any modified code? I would look there first.<br>
<br>
Is there a (much) smaller version of the setup and conditions that<br>
still shows this behavior? That would make it much easier to narrow<br>
down the culprit.<br>
<br>
Klaus<br>
</blockquote></div>
</blockquote></div>