<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Klaus,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We tried to run our code with FLASH4.5, but we were still getting a deadlock in the refinement part of the code.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Later we found out, that on using the default mpi library(Intel mpi/18.0.2) on stampede2 was leading to the deadlock, but when we tried with openmpi-3.1.2, we were able to get past the deadlock, which is very strange.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Vishal<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Robert Fisher <rfisher1@umassd.edu><br>
<b>Sent:</b> Monday, February 25, 2019 9:17 PM<br>
<b>To:</b> Vishal Tiwari<br>
<b>Cc:</b> Klaus Weide<br>
<b>Subject:</b> Re: [FLASH-USERS] MPI deadlock in amr_refine_derefine</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Klaus :<br>
<div><br>
</div>
<div> That's a great suggestion. Vishal and I were discussing migrating to 4.5 just recently, since there are a number of new features which are relevant to us -- including the improved treatment of cylindrical geometry and improvements to Tree gravity. We
tend to be slow to upgrade, since the science we are doing is outgrowths of the work we started with Suoqing Ji around 2012, and is still using the same base version of FLASH to guarantee backwards compatibility.</div>
<div><br>
</div>
<div> Best wishes,</div>
<div><br>
</div>
<div> Bob</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Feb 25, 2019 at 8:25 PM Vishal Tiwari <<a href="mailto:vtiwari@umassd.edu">vtiwari@umassd.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Klaus,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank you for your email.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am using FLASH 4.0.1.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I will try the recent FLASH 4.5 version if I get the issue.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank you!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Vishal</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Graduate Student, Physics</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
UMass, Dartmouth<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_gmail-m_7839998702000405304appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_7839998702000405304divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Klaus Weide <<a href="mailto:klaus@flash.uchicago.edu" target="_blank">klaus@flash.uchicago.edu</a>><br>
<b>Sent:</b> Monday, February 25, 2019 6:08 PM<br>
<b>To:</b> Vishal Tiwari<br>
<b>Cc:</b> <a href="mailto:flash-users@flash.uchicago.edu" target="_blank">flash-users@flash.uchicago.edu</a>; Robert Fisher<br>
<b>Subject:</b> Re: [FLASH-USERS] MPI deadlock in amr_refine_derefine</font>
<div> </div>
</div>
<div class="x_gmail-m_7839998702000405304BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_gmail-m_7839998702000405304PlainText">On Sun, 24 Feb 2019, Vishal Tiwari wrote:<br>
<br>
> Hello,<br>
> <br>
> I am facing issues with my simulations when running on stampede2, which gets stuck in the refinement part of the code. The code keeps refining until the number of blocks requested is smaller than the number of tasks, but hangs when no. of blocks > ntasks.
Looking at the trace of the code using ddt suggests that there is a MPI deadlock. (see the figure attached).<br>
> <br>
> This issue occurs only on the stampede2 because it was refining fine on stampede1 and works fine on a local cluster on my campus.<br>
> <br>
> Further, I found that people were facing the exact same issue in this thread [1]<<a href="http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html" target="_blank">http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html</a>>,
but the thread wasn't concluded with a solution.<br>
> <br>
> I would be grateful for any pointers with regards to this issue.<br>
<br>
Vishal,<br>
<br>
You did not say which version of FLASH you are using. I does not seem the <br>
be the latest, since according to your tack trace, there should be a <br>
WAITALL call on line 720 of mpi_amr_redist_blk.F90. This the case in<br>
<br>
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source/mpi_amr_redist_blk.F90<br>
<br>
of the FLASH 4.4 release code, but not in the same file from the FLASH <br>
4.5 release. So there have been code changes in a file that plays an <br>
important role in your stack trace. You should try whether you get the <br>
same problem with the most recent release, FLASH 4.5.<br>
<br>
Klaus<br>
<br>
</div>
</span></font></div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">Dr. Robert Fisher<br>
Associate Professor / Graduate Program Director<br>
University of Massachusetts/Dartmouth<br>
Department of Physics<br>
285 Old Westport Road<br>
North Dartmouth, Massachusetts 02747<br>
<a href="mailto:robert.fisher@umassd.edu" target="_blank">robert.fisher@umassd.edu</a><br>
<a href="http://www.novastella.org" target="_blank">http://www.novastella.org</a></div>
</div>
</div>
</body>
</html>