[FLASH-USERS] Memory leak when using cray compilers

[Haakon Andresen]

Hi,


Thank you for the reply. I will test your suggestions and I will see if I can get the sysadmins to enable me to use a different MPI implementation. At this point, its a bit obscure what they actually use, since its all handled by wrappers.


Is there an easy way of setting up the simulation setup you suggest or some simulation model that has something like it already? I can write it myself, but I would be happy to speed up the process.


Thanks again.


Best,
Haakon

________________________________
From: Reyes, Adam <adam.reyes at rochester.edu>
Sent: 07 June 2024 13:28:15
To: Haakon Andresen
Cc: flash-users at flash.rochester.edu
Subject: Re: [FLASH-USERS] Memory leak when using cray compilers

Hi Haakon,

I’ve not observed this behavior before, but I also don’t have much experience compiling FLASH with the cray compilers. That it’s only happening with cray and for internode communication it sounds like it might be a compiler/mpi bug. Are you able to test with another MPI library like MPICH or OpenMPI?

If you want to narrow it down more in the code the simplest would be uniform grid with periodic boundary conditions on all sides. In this path Grid_fillGuardCells should only be doing communications with MPI_SendRecv that are pretty straightforward to follow and you don’t have to worry about gr_bcApplyToAllBlks. I don’t think FLASH would be allocating any memory either so any memory leak would have to come from MPI.
*********************************************
Adam Reyes

[FLASH.jpg]
Code Group Leader, Flash Center for Computational Science
Research Scientist, Dept. of Physics and Astronomy
University of Rochester
River Campus: Bausch and Lomb Hall, 369
500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
Email adam.reyes at rochester.edu
Web https://flash.rochester.edu
 (he / him / his)
[FLASH-pride-sml.png]

*********************************************



On Jun 7, 2024, at 11:47 AM, Haakon Andresen <haakon.andresen at astro.su.se> wrote:

Dear Flash users,

I am experiencing a memory leak when trying to use the cray-compiler environment. The memory monitor
reports an increase in the max  rss and vsize memory. The increasing trend is more or less linear, but shows some variation
with problem and processor.  The issue does not persist when using gnu compilers. I think the issue is related to internode communication,
but I have not been able to track it down.

What I have found so far:
- Occurs in both 2D and 3D
- Occurs in the Sedov and Sod test problems.
- Not every process leaks, but there is no clear pattern either.
- Number of leaking processes is often small compared to the total number of processes
- Occurs for both UG and Paramesh and Uniform Grid.
- Does not occur when running on a single node.
- Seems to happen during guard-cell fills

I found a report in the mailing list archive of a similar issue, but it was related to HYPRE, which
I do not use.

Any input/ideas would be appreciated.

Best,
Haakon



_______________________________________________
flash-users mailing list
flash-users at flash.rochester.edu<mailto:flash-users at flash.rochester.edu>

For list info, including unsubscribe:
https://flash.rochester.edu/mailman/listinfo/flash-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/24e4f4c2/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FLASH.jpg
Type: image/jpeg
Size: 23876 bytes
Desc: FLASH.jpg
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/24e4f4c2/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FLASH-pride-sml.png
Type: image/png
Size: 12732 bytes
Desc: FLASH-pride-sml.png
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/24e4f4c2/attachment-0001.png>