<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
Hi Mark,
<div><br>
</div>
<div>You might also try adding `useFortran2003=True` to your setup line. On Intel compilers, at least, this will spit out some useful memory usage statistics toward the beginning of the .log file.</div>
<div><br>
</div>
<div>Best of luck,</div>
<div>Sean</div>
</div>
<div name="messageSignatureSection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<br>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
---------------------------------------------------------------------------------------------</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
Sean M. Couch</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
Assistant Professor</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
Department of Physics and Astronomy</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
Department of Computational Mathematics, Science, and Engineering</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
National Superconducting Cyclotron Laboratory/Facility for Rare Isotope Beams</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
Michigan State University</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
567 Wilson Rd, 3250 BPS</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
East Lansing, MI 48824</p>
<p style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(51, 51, 51);">
(517) 884-5035 —— <a href="mailto:couch@pa.msu.edu">couch@pa.msu.edu</a> —— <a href="http://www.pa.msu.edu/~couch">
www.pa.msu.edu/~couch</a></p>
</div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
On Jun 21, 2018, 10:37 AM -0400, Mark Richardson <mark.richardson.work@gmail.com>, wrote:<br>
<blockquote type="cite" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">
<div dir="ltr">
<div>
<div>Hi Josh,<br>
</div>
Thanks for your suggestions. I have turned on debug with -O0 and -g, and it didn't affect the outcome. All gcc compilers on stampede2 are version > 5, so I might try and install an earlier version. Further, their mpif90 etc is built with intel, while my builds
that work elsewhere are all gnu built.<br>
<br>
</div>
<div>I have maxblocks set to 200, but I am only trying to allocate an average of 63 blocks per processor. I have played with the number of processors per node and number of node, effectively changing both the average blocks per processor being allocated, and
the memory available to each processor. Neither averted the hang up.<br>
<br>
</div>
<div>Thanks again,<br>
</div>
-Mark<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 18 June 2018 at 07:59, Joshua Wall <span dir="ltr"><<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div dir="ltr">Hello Mark,
<div><br>
</div>
<div> I seem to remember some issue with the code hanging that was due to optimization with newer versions of the GCC compiler suite. Indeed, I think I also implemented this to get past a hang:</div>
<div><br>
</div>
<div><a href="http://flash.uchicago.edu/pipermail/flash-users/2015-February/001637.html" target="_blank">http://flash.uchicago.edu/<wbr>pipermail/flash-users/2015-<wbr>February/001637.html</a> <br>
</div>
<div><br>
</div>
<div>Does attempting this fix (or running with -O0 ) help at all?</div>
<div><br>
</div>
<div>As a final note, I have also seen the code hang silently when in fact I had either 1) exceeded the maximum number of blocks per processor or 2) run out of RAM on a node. So those are things to check as well.</div>
<div><br>
</div>
<div>Hope that helps!</div>
<div><br>
</div>
<div>Josh</div>
</div>
<br>
<div class="gmail_quote">
<div>
<div class="h5">
<div dir="ltr">On Mon, Jun 18, 2018 at 1:47 AM Mark Richardson <<a href="mailto:mark.richardson.work@gmail.com" target="_blank">mark.richardson.work@gmail.<wbr>com</a>> wrote:<br>
</div>
</div>
</div>
<blockquote class="gmail_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;">
<div>
<div class="h5">
<div style="word-wrap:break-word">
<div>Hello,</div>
<div><br>
</div>
<div> My current FLASH build worked fine on the original stampede, and on small local clusters. But on both KNL and SKX nodes on Stampede2, I get a hang during refinement in mpi_amr_redist_blk. If I build the initial simulation on a different cluster, then
the hang happens on Stampede2 the first time the grid structure changes. If I build the initial simulation on Stampede2, it hangs after triggering level 6 in that initial Simulation_initBlk loop, but still in mpi_amr_redist_blk. </div>
<div><br>
</div>
<div>Setup call: </div>
<div> ./setup -auto -3d -nxb=32 -nyb=16 -nzb=8 -maxblocks=200 species=rock,watr +uhd3tr mgd_meshgroups=1 Simulation_Buiild</div>
<div><br>
</div>
<div> Using: ifort (IFORT) 17.0.4 20170411 </div>
<div><br>
</div>
<div>Log file tail in file Logfile.pdf</div>
<div><br>
</div>
<div>I’ve change maxblocks, and number of nodes, without getting out of this issue. </div>
<div><br>
</div>
<div>I’ve changed the “iteration, no. not moved” output to occur for each processor, and they all print out the identical correct info. I’ve added per processor print statements before the nrecv>0 waitall and nsend>0 waitall in mpi_amr_redist_blk.F90 and see
that about 25% of processors are waiting indefinitely in the nrecv>0 waitall, while the other 75% complete the resist_blk subroutine and are waiting later for the remaining processors to finish. </div>
<div><br>
</div>
<div>I’ve tried adding sleep(1) inside the niter loop, as suggested in the past for someone who found niter going to 100 (note, I’m getting niter = 2 with no. not move=0, so all processors successfully exit that loop but hang later). This didn’t change the
result.</div>
<div><br>
</div>
<div>Has anyone else seen similar hanging occurring, on any cluster? Any suggestions for overcoming this hang event? </div>
<div><br>
</div>
<div>Thank you for your help,</div>
<div> -Mark</div>
<div><br>
</div>
<div> </div>
</div>
</div>
</div>
<div style="word-wrap:break-word">
<div></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<span class="HOEnZb"><font color="#888888"><br>
</font></span>
<div><span class="HOEnZb"><font color="#888888"></font></span>
<div dir="auto" style="word-wrap:break-word"><span class="HOEnZb"><font color="#888888"></font></span>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span class="HOEnZb"><font color="#888888">-- <br>
<br>
Mark Richardson<br>
MAT Postdoctoral Fellow<br>
Department of Astrophysics<br>
American Museum of Natural History<br>
<a href="mailto:MRichardson@amnh.org" target="_blank">MRichardson@amnh.org</a><br>
<a href="https://sites.google.com/site/marklarichardson/" target="_blank">My Website</a><br>
<a href="tel:(212)%20496-3432" value="+12124963432" target="_blank">212 496 3432</a></font></span></div>
</div>
</div>
<span class="HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
</div>
<span class="HOEnZb"><font color="#888888">--<br>
</font></span>
<div dir="ltr" class="m_5046780262454028167gmail_signature" data-smartmail="gmail_signature">
<span class="HOEnZb"><font color="#888888"></font></span>
<div dir="ltr"><span class="HOEnZb"><font color="#888888"></font></span>
<div><span class="HOEnZb"><font color="#888888"></font></span>
<div><span class="HOEnZb"><font color="#888888"></font></span>
<div><span class="HOEnZb"><font color="#888888"></font></span>
<div><span class="HOEnZb"><font color="#888888"></font></span>
<div><span class="HOEnZb"><font color="#888888">Joshua Wall<br>
</font></span></div>
<span class="HOEnZb"><font color="#888888">Doctoral Candidate<br>
</font></span></div>
<span class="HOEnZb"><font color="#888888">Department of Physics<br>
</font></span></div>
<span class="HOEnZb"><font color="#888888">Drexel University<br>
</font></span></div>
<span class="HOEnZb"><font color="#888888"><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g">3141 Chestnut Street</a><br>
</font></span></div>
<span class="HOEnZb"><font color="#888888"><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g">Philadelphia, PA 19104</a><br>
</font></span></div>
<span class="HOEnZb"><font color="#888888"></font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
--<br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
<span style="font-size:12.8px">Mark Richardson</span><br style="font-size:12.8px">
<div style="font-size:small"><span style="font-size:12.8px">MAT Postdoctoral Fellow</span>
<div style="font-size:12.8px">Department of Astrophysics</div>
<div><span style="font-size:12.8px">American Museum of Natural History</span>
<div style="font-size:12.8px"><a href="mailto:Mark.Richardson.Work@gmail.com" style="color:rgb(17,85,204)" target="_blank">Mark.Richardson.Work@gmail.com</a></div>
<div><a href="https://sites.google.com/site/marklarichardson/" style="color:rgb(17,85,204)" target="_blank">My Website</a><br>
<div style="font-size:12.8px">212 496 3432</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div></div>
</div>
</body>
</html>