<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Rayan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for your email.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I tried with increased the number of nodes, but the issue remained. We have also tried to reduce the maxblocks per processor(currently using 30), but it didn't work. We also looked at the memory usage per node by logging into the nodes where the flash code
was being executed, but the memory is not being exceeded.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Please find attached the log file for the run. We are using KNL compute nodes for the simulation.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am running this code using 256 processes over 16 nodes.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for your help.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Vishal Tiwari</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Graduate Student, Physics</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
UMass, Dartmouth<br>
</div>
<br>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ryan Farber <rjfarber@umich.edu><br>
<b>Sent:</b> Monday, February 25, 2019 2:57 PM<br>
<b>To:</b> Vishal Tiwari<br>
<b>Cc:</b> flash-users@flash.uchicago.edu; Robert Fisher<br>
<b>Subject:</b> Re: [FLASH-USERS] MPI deadlock in amr_refine_derefine</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Vishal,
<div><br>
</div>
<div>I've had a similar issue when I'm not allocating enough memory to a job on Stampede2. You can try requesting one additional node as a test.</div>
<div>Note that reducing maxblocks (in the thread you linked) has the effect of requiring less memory.</div>
<div><br>
</div>
<div>If you're still having trouble, could you attach your logfile and mention how many nodes (and how many processors if you're not using 256 as suggested by your DDT attachment) and what node type (SKX or KNL) you're using?</div>
<div><br>
</div>
<div>Best,<br clear="all">
<div>
<div dir="ltr" class="x_m_5422507138861831754gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">--------
<div>Ryan</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Feb 24, 2019 at 12:27 PM Vishal Tiwari <<a href="mailto:vtiwari@umassd.edu" target="_blank">vtiwari@umassd.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am facing issues with my simulations when running on stampede2, which gets stuck in the refinement part of the code. The code keeps refining until the number of blocks requested is smaller than the number of tasks, but hangs when no. of blocks > ntasks.
Looking at the trace of the code using ddt suggests that there is a MPI deadlock. (see the figure attached).<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This issue occurs only on the stampede2 because it was refining fine on stampede1 and works fine on a local cluster on my campus.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Further, I found that people were facing the exact same issue in this thread <a href="http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html" title="http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html" target="_blank">
[1]</a>, but the thread wasn't concluded with a solution.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I would be grateful for any pointers with regards to this issue.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank you!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[1] <a href="http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html" target="_blank">
http://flash.uchicago.edu/pipermail/flash-users/2017-September/002402.html</a><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Vishal Tiwari<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Graduate Student, Physics<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
University of Massachusetts, Dartmouth<br>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>