<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div>Hi All,</div>
<div><br>
</div>
<div>I am running N-body + Hydro simulations of cluster mergers with FLASH. </div>
<div><br>
</div>
<div>My code, depending on the initial parameters used (such as the refinement or placement of the clusters) or the number of nodes I run over, either f<span style="font-size: 10pt;">ails straight away during the initialisation with a generic segmentation fault
or will run for a while and then "hang" i.e. not do anything code wise although it still runs on our HPC using up memory.</span></div>
<div><span style="font-size: 10pt;"><br>
</span></div>
<div><span style="font-size: 10pt;">If the job hangs, I can usually run from the previous checkpoint and it will run further along until it hangs again, so its a case of restarting the simulation every time it hangs until the simulation completes its run. Which
isn't ideal but at least I can get some work out of it.</span></div>
<div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;"><br>
</span></div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;">For example, if I run a hydrostatic test for a single galaxy cluster on 20 nodes it fails straight away but if I increase the number of nodes to 100 then the simulation runs all the way
to completion. </span><span style="font-size: 13.3333px;">I have done two hydrostatic tests now on a single cluster and these two simulations ran fine without any issues (I run for 10 Gyrs and produce a 1000 plot files). </span><span style="font-size: 13.3333px;">I
have also done a cluster merger simulation which on the first attempt reached 460 plot files before hanging while on the second attempt (starting from the beginning) it got all the way up to 862 plot files before hanging. Both simulations use the same flash4.</span></div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;"><br>
</span></div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;">Block wise, the simulations should be able to run on say 10 or 20 nodes, but they have only been running when I use 50 or more nodes where here it only uses between 2-6 blocks per processor.
Interestingly,</span><span style="font-size: 13.3333px;"> I have been running with debugging flags and optimisation flags of -O0 instead of the usual -O3 and the simulations appear to be running faster and get further along this way.</span></div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;"><br>
</span></div>
<div style="font-size: 13.3333px;"><span style="font-size: 13.3333px;">I have used an intel inspector debugging tool on a run where I used 20 nodes (when it fails straightaway) and it showed that there are 19 memory leaks and 1 memory deallocation issue:</span></div>
</div>
<br>
<div>
<p><img src="cid:9d902851-9e13-4662-8602-125a41d61c3c" originalwidth="1092" originalheight="493" rszimgcmd="fitToWindow" style="width: 560px; height: 252.821px;"></p>
<div><br>
</div>
<div><span style="font-size: 13.3333px;">The memory deallocation issue is only a warning and comes from the flash4 binary where it complains about all the allocations from files such as Grid_init.F90, Particles_init.F90, amr_initialize.F90 which is obviously
something I don't touch and would presume FLASH handles this allocation correctly anyway.</span></div>
<div><span style="font-size: 13.3333px;"><br>
</span></div>
<div><span style="font-size: 13.3333px;">The memory leak issues to do with lipmpi etc I don't know much about except that they are libraries from the MPI package. I have mentioned this to our HPC engineers and they sa</span><font size="2">id "<span style="text-indent: -24px;">if
these libraries have something wrong it should affect everyone using the same version of MPI on Viper, which is not the case. </span><span style="text-indent: -24px;">In some very unlikely cases, there might be a bug or an issue with one of these libraries
(that is more probable with omnipath)" . </span></font></div>
<div><br>
</div>
<div>I have been through my code many times by myself and with colleagues and we just can't see any issue there.<span style="font-size: 10pt;"> I am using standard FLASH modules so nothing has been altered in that regard. </span></div>
<div><br>
</div>
<div>Has anyone got an idea what is happening? Has anyone come across problems with FLASH before with memory leak issues like I have shown? Is this a problem with the mpi installation on the HPC or is it more likely there is a problem with my code somewhere?</div>
<div><br>
</div>
<div>I am currently using <span class="s1" style="color: rgb(80, 80, 80); font-family: Menlo; font-size: 11px;">intel/mpi/</span><span class="s2" style="font-family: Menlo; font-size: 11px;">64</span><span class="s1"><font face="Menlo" style="font-size: 11px; color: rgb(80, 80, 80);">/5.1.3.181</font><font size="2">
and</font><font face="Menlo" style="font-size: 11px; color: rgb(80, 80, 80);"> </font></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(80, 80, 80); font-family: Menlo; font-size: 11px;">hdf5/intel/intelmpi/1.8.16
</span><span style="font-variant-ligatures: no-common-ligatures;"><font size="2">to compile/run my jobs. I have also tried with </font></span><span style="color: rgb(80, 80, 80); font-family: Menlo; font-size: 11px;">hdf5/gcc/openmpi/1.8.16
</span><font size="2">and</font><span style="color: rgb(80, 80, 80); font-family: Menlo; font-size: 11px;"> </span><span style="color: rgb(80, 80, 80); font-family: Menlo; font-size: 11px;">openmpi/gcc/1.10.5
</span><font size="2">but I experience similar behaviour.</font></div>
<div><br>
</div>
<div>Any thoughts on this will be greatly appreciated!</div>
<div><br>
</div>
Many Thanks,</div>
<div><br>
</div>
<div>Alex</div>
<div><br>
<p></p>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-size:12px">
<hr>
</div>
<div style="font-size:12px"><b><font face="Arial">Mr Alex Sheardown</font></b></div>
<div style="font-size:12px"><font face="Arial">Postgraduate Research Student</font></div>
<div style="font-size:12px"><font face="Arial"><br>
</font></div>
<div style="font-size:12px"><font face="Arial">E.A. Milne Centre for Astrophysics</font></div>
<div style="font-size:12px"><font face="Arial">University of Hull</font></div>
<div style="font-size:12px"><font face="Arial">Cottingham Road</font></div>
<div style="font-size:12px"><font face="Arial">Kingston upon Hull</font></div>
<div style="font-size:12px"><font face="Arial">HU6 7RX</font></div>
<div style="font-size:12px"><font face="Arial"><br>
</font></div>
<div style="font-size:12px"><a href="https://mail.hull.ac.uk/owa/redir.aspx?REF=_wok6-STjTeTuQlVeEE3DYaVcvKXJXINIb2ho14u7UoAceEsmknTCAFodHRwOi8vd3d3Lm1pbG5lLmh1bGwuYWMudWs." target="_blank"><font face="Arial">www.milne.hull.ac.uk</font></a></div>
</div>
</div>
</div>
</div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #505050}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.s2 {font-variant-ligatures: no-common-ligatures; color: #c33720}
</style></div>
</body>
</html>