<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:713312845;
mso-list-type:hybrid;
mso-list-template-ids:1856001006 -818541744 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:30;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">An issue/bug has been discovered up in the behavior of Paramesh in FLASH 4.5 and 4.6.1. Specifically, a run will hang and produce no error or abort while in the subroutine mpi_xchange_blocks(…) located in
/source/Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source/mpi_lib.F90. The hang is semi-random with bitwise identical runs stalling at nearly, but not exactly, the same location. The stall occurs when one processor becomes stuck in one of
the MPI_Ssend calls seemingly unable to complete the data transfer while the receiving processor waits indefinitely. I found that this occurs only in cross-node communications and so appears to be due to a bug within OpenMPI (I am using version 3.1.1) and
its use of Infiniband communication methods. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ultimately, the issue appears to be due to the get/put protocols used by OpenMPI for large messages and can be resolved by forcing OpenMPI to communicate via Infiniband using the send/recv protocols for ALL
messages (normally used for sub 1Mb messages only). To do this, set the following mca parameter in your mpirun/mpiexec call:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> --mca btl_openib_flags 305<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have not experienced the stalling/hang issue since setting this parameter.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Conversations of a similar nature have cropped up in the past, with suggested fixes such as downgrading gcc versions (<a href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fflash.uchicago.edu%2Fpipermail%2Fflash-users%2F2015-February%2F001637.html&data=02%7C01%7Cscl63%40drexel.edu%7Ca52406dc63d9492e3bc708d82522a503%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637300178016653772&sdata=AgrqMgUdEmc7%2FbPGFWInZSz%2Fkbf%2FOh0VReKO0C08n6w%3D&reserved=0" title="Original URL:
http://flash.uchicago.edu/pipermail/flash-users/2015-February/001637.html
Click to follow link."><span style="color:#00006A">http://flash.uchicago.edu/pipermail/flash-users/2015-February/001637.html</span></a>)
and increasing maxblocks (<u><span style="color:#00006A"><a href="http://flash.uchicago.edu/pipermail/flash-users/2018-June/002652.html">http://flash.uchicago.edu/pipermail/flash-users/2018-June/002652.html</a></span></u>). But I have found this fix to be
the most successful. The main drawback of this fix is that the send/recv protocols are not as efficient as the get/put protocols so a performance hit is inevitable, but I have found this to be on the order of 10% slower in my personal runs. To be more clear,
the stalling/hang issue has not been seen by myself or colleagues while using vanilla FLASH or in standard test problems. However, in my application, no modifications were made to the offending subroutine. Regardless, the issue seems to be a step removed from
FLASH in OpenMPI communication protocols. I wanted to share my findings in case anyone else ever runs into similar issues in the future.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hope this is helpful!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-Sean<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Times New Roman",serif;color:black">Sean C. Lewis<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">Doctoral Candidate<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">Department of Physics<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">Drexel University</span><o:p></o:p></p>
</div>
</body>
</html>