[FLASH-USERS] Memory leak when using cray compilers

[Tomasz Plewa]

Hi Haakon -

I have access to

    $ ftn -V
    Cray Fortran : Version 17.0.0  Fri Jun 07, 2024  05:44:48
    $ mpif90 -show
    crayftn -I/opt/cray/pe/mpich/8.1.28/ofi/cray/17.0/include
    -I/opt/cray/pe/mpich/8.1.28/ofi/cray/17.0/include
    -L/opt/cray/pe/mpich/8.1.28/ofi/cray/17.0/lib -lmpifort_cray -lmpi_cray

Which FLASH version and Cray compiler version are you using?

I would not be able to debug the code, but can test the code in our 
environment and have access to other compilers (GNU, NVIDIA, etc.).

It would also be helpful if you share either a log file from one of your 
Sod or Sedov 2D runs.

Best -

Tomek
-- 
On 6/7/24 07:28, Reyes, Adam wrote:
> Hi Haakon,
>
> I’ve not observed this behavior before, but I also don’t have much 
> experience compiling FLASH with the cray compilers. That it’s only 
> happening with cray and for internode communication it sounds like it 
> might be a compiler/mpi bug. Are you able to test with another MPI 
> library like MPICH or OpenMPI?
>
> If you want to narrow it down more in the code the simplest would be 
> uniform grid with periodic boundary conditions on all sides. In this 
> path Grid_fillGuardCells should only be doing communications with 
> MPI_SendRecv that are pretty straightforward to follow and you don’t 
> have to worry about gr_bcApplyToAllBlks. I don’t think FLASH would be 
> allocating any memory either so any memory leak would have to come 
> from MPI.
> *********************************************
> Adam Reyes
>
> FLASH.jpg
> Code Group Leader, Flash Center for Computational Science
> Research Scientist, Dept. of Physics and Astronomy
> University of Rochester
> River Campus: Bausch and Lomb Hall, 369
> 500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
> Emailadam.reyes at rochester.edu
> Webhttps://flash.rochester.edu
> (he / him / his)
> FLASH-pride-sml.png
>
> *********************************************
>
>
>
>> On Jun 7, 2024, at 11:47 AM, Haakon Andresen 
>> <haakon.andresen at astro.su.se> wrote:
>>
>> Dear Flash users,
>>
>> I am experiencing a memory leak when trying to use the cray-compiler 
>> environment. The memory monitor
>> reports an increase in the max  rss and vsize memory. The increasing 
>> trend is more or less linear, but shows some variation
>> with problem and processor.  The issue does not persist when using 
>> gnu compilers. I think the issue is related to internode communication,
>> but I have not been able to track it down.
>>
>> What I have found so far:
>> - Occurs in both 2D and 3D
>> - Occurs in the Sedov and Sod test problems.
>> - Not every process leaks, but there is no clear pattern either.
>> - Number of leaking processes is often small compared to the total 
>> number of processes
>> - Occurs for both UG and Paramesh and Uniform Grid.
>> - Does not occur when running on a single node.
>> - Seems to happen during guard-cell fills
>>
>> I found a report in the mailing list archive of a similar issue, but 
>> it was related to HYPRE, which
>> I do not use.
>>
>> Any input/ideas would be appreciated.
>>
>> Best,
>> Haakon
>>
>>
>>
>> _______________________________________________
>> flash-users mailing list
>> flash-users at flash.rochester.edu
>>
>> For list info, including unsubscribe:
>> https://flash.rochester.edu/mailman/listinfo/flash-users
>
>
> _______________________________________________
> flash-users mailing list
> flash-users at flash.rochester.edu
>
> For list info, including unsubscribe:
> https://flash.rochester.edu/mailman/listinfo/flash-users