[FLASH-USERS] Memory leak when using cray compilers

[Reyes, Adam]

Hi Haakon,

I’ve not observed this behavior before, but I also don’t have much experience compiling FLASH with the cray compilers. That it’s only happening with cray and for internode communication it sounds like it might be a compiler/mpi bug. Are you able to test with another MPI library like MPICH or OpenMPI?

If you want to narrow it down more in the code the simplest would be uniform grid with periodic boundary conditions on all sides. In this path Grid_fillGuardCells should only be doing communications with MPI_SendRecv that are pretty straightforward to follow and you don’t have to worry about gr_bcApplyToAllBlks. I don’t think FLASH would be allocating any memory either so any memory leak would have to come from MPI.
*********************************************
Adam Reyes


Code Group Leader, Flash Center for Computational Science  
Research Scientist, Dept. of Physics and Astronomy
University of Rochester
River Campus: Bausch and Lomb Hall, 369  
500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
Email adam.reyes at rochester.edu
Web https://flash.rochester.edu
 (he / him / his)


*********************************************



> On Jun 7, 2024, at 11:47 AM, Haakon Andresen <haakon.andresen at astro.su.se> wrote:
> 
> Dear Flash users,
> 
> I am experiencing a memory leak when trying to use the cray-compiler environment. The memory monitor
> reports an increase in the max  rss and vsize memory. The increasing trend is more or less linear, but shows some variation
> with problem and processor.  The issue does not persist when using gnu compilers. I think the issue is related to internode communication,
> but I have not been able to track it down.
> 
> What I have found so far:
> - Occurs in both 2D and 3D
> - Occurs in the Sedov and Sod test problems.
> - Not every process leaks, but there is no clear pattern either.
> - Number of leaking processes is often small compared to the total number of processes
> - Occurs for both UG and Paramesh and Uniform Grid.
> - Does not occur when running on a single node.
> - Seems to happen during guard-cell fills
> 
> I found a report in the mailing list archive of a similar issue, but it was related to HYPRE, which
> I do not use.
> 
> Any input/ideas would be appreciated. 
> 
> Best,
> Haakon
> 
> 
> 
> _______________________________________________
> flash-users mailing list
> flash-users at flash.rochester.edu <mailto:flash-users at flash.rochester.edu>
> 
> For list info, including unsubscribe:
> https://flash.rochester.edu/mailman/listinfo/flash-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/2634c664/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FLASH.jpg
Type: image/jpeg
Size: 23876 bytes
Desc: not available
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/2634c664/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FLASH-pride-sml.png
Type: image/png
Size: 12732 bytes
Desc: not available
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/2634c664/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1391 bytes
Desc: not available
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240607/2634c664/attachment-0001.p7s>