<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Jacob,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for the reply. I'm using clusters at the partnership for an advanced computing environment (PACE) system at Georgia Tech. Each node has 24 processors and total of 128 GB memory. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For Job submission, users must prepare a PBS script and submit that script using the
<b>qsub </b>command which is known to be faster and more robust.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is an example of PBS script that I'm submitting to the system:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><i>#!/usr/bin/env bash<br>
</i></span>
<div><i>#PBS -l nodes=N:ppn=24<br>
</i></div>
<div><i>#PBS -l walltime=35:06:00:00<br>
</i></div>
<div><i>#PBS -q NameOfQueue<br>
</i></div>
<div><i><br>
</i></div>
<div><i>cd ~/data/FLASH4.6/object<br>
</i></div>
<div><i>module purge; module load intel/19.0 hdf5/1.10.5<br>
</i></div>
<div><i>mpirun -np 24*N ./flash4<br>
</i></div>
<div><i><br>
</i></div>
<span><i>#End of script</i></span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So, the issue is that when I increase <i>N </i>to higher number than 1, then the computational time becomes much slower. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mohammad</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
----------------------------------------------------------------------</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
----------------------------------------------------------------------<br>
<div>
<div>Mohammad Mohaghar, Ph.D.<br>
<span>Postdoctoral Fellow</span><br>
School of Civil and Environmental Engineering<br>
Georgia Institute of Technology<br>
Phone: (678) 900-5453 </div>
<br>
<div><b><br>
</b></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> McFarland, Jacob A. <mcfarlandja@missouri.edu><br>
<b>Sent:</b> Friday, September 27, 2019 6:33 PM<br>
<b>To:</b> Mohaghar, Mohammad <mohaghar@gatech.edu>; flash-users@flash.uchicago.edu <flash-users@flash.uchicago.edu><br>
<b>Subject:</b> RE: Regarding slow computation on multi-node</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
span.x_EmailStyle19
{font-family:"Calibri",sans-serif;
color:#1F497D}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Mohammad,</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">It might be helpful to send some information on the computer system you are using. What kind of mpi command are you using to run flash and what kind of job
scheduler is your computer using. Maybe you could copy your run command into the email as well.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Thanks,</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Jacob</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> flash-users-bounces@flash.uchicago.edu <flash-users-bounces@flash.uchicago.edu>
<b>On Behalf Of </b>Mohaghar, Mohammad<br>
<b>Sent:</b> Friday, September 27, 2019 5:29 PM<br>
<b>To:</b> flash-users@flash.uchicago.edu<br>
<b>Subject:</b> [FLASH-USERS] Regarding slow computation on multi-node</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<div id="x_Signature">
<div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">Dear FLASH users, </span></p>
</div>
<div id="x_Signature">
<div>
<div>
<div id="x_Signature">
<div>
<div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">I figured out that the 3 dimensional computation on 1 node with N processors runs much faster than >1 node where each node has N processors. Hence, the computational time drops
on one node as I increase the number of processors (The maximum available processors for me at the moment is 24), however, as soon as I run on multi-node processors (even if I double or triple the number of processors) the computational time increases. The
computational time can increase even to 8 times the computational time for one node (so it's way too slow). Do you have any idea why this happens? I understand that it takes more time for communication between the nodes, but never thought it'd be this slow.
With this problem, there is no point point for me to run on multi-node since it takes way too long time for the 3D simulation to run. On one node, I can run the 3D simulation with lower resolution, but if I increase the resolution I get an "out of memory"
error.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">I'd appreciate any thoughts that you may have to help me out.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">Thank you.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">Best,</span></p>
</div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">Mohammad </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper">
<div id="x_divtagdefaultwrapper">
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">----------------------------------------------------------------------</span></p>
</div>
<div id="x_divtagdefaultwrapper">
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">----------------------------------------------------------------------</span></p>
<div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black">Mohammad Mohaghar, Ph.D.<br>
Postdoctoral Fellow<br>
School of Civil and Environmental Engineering<br>
Georgia Institute of Technology<br>
Phone: (678) 900-5453 </span></p>
</div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
<div>
<p class="x_MsoNormal"><span style="font-family:"Calibri",sans-serif; color:black"> </span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>