[FLASH-USERS] Regarding slow computation on multi-node

[Mohaghar, Mohammad]

Hi Jacob,

Thanks for the reply. I'm using clusters at the partnership for an advanced computing environment (PACE) system at Georgia Tech. Each node has 24 processors and total of 128 GB memory.
For Job submission, users must prepare a PBS script and submit that script using the qsub command which is known to be faster and more robust.

 This is an example of PBS script that I'm submitting to the system:

#!/usr/bin/env bash
#PBS -l nodes=N:ppn=24
#PBS -l walltime=35:06:00:00
#PBS -q NameOfQueue

cd ~/data/FLASH4.6/object
module purge; module load intel/19.0 hdf5/1.10.5
mpirun -np 24*N ./flash4

#End of script

So, the issue is that when I increase N to higher number than 1, then the computational time becomes much slower.

Best,
Mohammad
----------------------------------------------------------------------
----------------------------------------------------------------------
Mohammad Mohaghar, Ph.D.
Postdoctoral Fellow
School of Civil and Environmental Engineering
Georgia Institute of Technology
Phone: (678) 900-5453


________________________________
From: McFarland, Jacob A. <mcfarlandja at missouri.edu>
Sent: Friday, September 27, 2019 6:33 PM
To: Mohaghar, Mohammad <mohaghar at gatech.edu>; flash-users at flash.uchicago.edu <flash-users at flash.uchicago.edu>
Subject: RE: Regarding slow computation on multi-node


Mohammad,



It might be helpful to send some information on the computer system you are using. What kind of mpi command are you using to run flash and what kind of job scheduler is your computer using. Maybe you could copy your run command into the email as well.



Thanks,

Jacob



From: flash-users-bounces at flash.uchicago.edu <flash-users-bounces at flash.uchicago.edu> On Behalf Of Mohaghar, Mohammad
Sent: Friday, September 27, 2019 5:29 PM
To: flash-users at flash.uchicago.edu
Subject: [FLASH-USERS] Regarding slow computation on multi-node





Dear FLASH users,



I figured out that the 3 dimensional computation on 1 node with N processors runs much faster than >1 node where each node has N processors. Hence, the computational time drops on one node as I increase the number of processors (The maximum available processors for me at the moment is 24), however, as soon as I run on multi-node processors (even if I double or triple the number of processors) the computational time increases. The computational time can increase even to 8 times the computational time for one node (so it's way too slow). Do you have any idea why this happens? I understand that it takes more time for communication between the nodes, but never thought it'd be this slow. With this problem, there is no point point for me to run on multi-node since it takes way too long time for the 3D simulation to run. On one node, I can run the 3D simulation with lower resolution, but if I increase the resolution I get an "out of memory" error.



I'd appreciate any thoughts that you may have to help me out.



Thank you.



Best,

Mohammad



----------------------------------------------------------------------

----------------------------------------------------------------------

Mohammad Mohaghar, Ph.D.
Postdoctoral Fellow
School of Civil and Environmental Engineering
Georgia Institute of Technology
Phone: (678) 900-5453




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20190927/a0757621/attachment.htm>