<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Mohammad,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It might be helpful to send some information on the computer system you are using. What kind of mpi command are you using to run flash and what kind of job scheduler
is your computer using. Maybe you could copy your run command into the email as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jacob<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> flash-users-bounces@flash.uchicago.edu <flash-users-bounces@flash.uchicago.edu>
<b>On Behalf Of </b>Mohaghar, Mohammad<br>
<b>Sent:</b> Friday, September 27, 2019 5:29 PM<br>
<b>To:</b> flash-users@flash.uchicago.edu<br>
<b>Subject:</b> [FLASH-USERS] Regarding slow computation on multi-node<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div id="Signature">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Dear FLASH users, <o:p></o:p></span></p>
</div>
<div id="Signature">
<div>
<div>
<div id="Signature">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I figured out that the 3 dimensional computation on 1 node with N processors runs much faster than >1 node where each node has N processors. Hence, the computational time drops
on one node as I increase the number of processors (The maximum available processors for me at the moment is 24), however, as soon as I run on multi-node processors (even if I double or triple the number of processors) the computational time increases. The
computational time can increase even to 8 times the computational time for one node (so it's way too slow). Do you have any idea why this happens? I understand that it takes more time for communication between the nodes, but never thought it'd be this slow.
With this problem, there is no point point for me to run on multi-node since it takes way too long time for the 3D simulation to run. On one node, I can run the 3D simulation with lower resolution, but if I increase the resolution I get an "out of memory"
error.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I'd appreciate any thoughts that you may have to help me out.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Thank you.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Best,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Mohammad <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper">
<div id="divtagdefaultwrapper">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">----------------------------------------------------------------------<o:p></o:p></span></p>
</div>
<div id="divtagdefaultwrapper">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">----------------------------------------------------------------------<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Mohammad Mohaghar, Ph.D.<br>
Postdoctoral Fellow<br>
School of Civil and Environmental Engineering<br>
Georgia Institute of Technology<br>
Phone: (678) 900-5453 <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>