<div dir="ltr">Thanks to Lee for pointing this out and sorry to Pedro for glossing over your initial message where you mention that you tune iprocs, jprocs, nxb, and nyb to have about the same number of grid points (so indeed looking at strong scaling rather than weak scaling).<div><br></div><div>One more point regarding the sigbus issue is that perhaps adding mcmodel=large in your FFLAGS_* (in your Makefile.h) will help -- if you're on an Intel system anyway this enforces absolute addressing whereas the default mcmodel=small uses relative addressing. From what I recall I had a ~2% performance decrease switching from mcmodel=large (but from mcmodel=medium) but it helped with memory issues in the past at large core counts.</div><div><br></div><div>Best,<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">--------<div>Ryan</div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 5, 2023 at 5:06 PM Ryan Farber <<a href="mailto:rjfarber@umich.edu">rjfarber@umich.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Pedro,<div><br></div><div>One point of follow-up regarding your sigbus error - it looks like this is a memory access error. I'm wondering if you're requesting more logical cores than physical cores exist on the machine. I've found that to be problematic in the past.</div><div><br></div><div>It sounds like Lee might have the answer to your issue regarding too few zones per proc. One point I'm confused about though is whether you're studying strong or weak scaling. Based on your response to Paco and I checked the FLASH users guide, for uniform grid mode there's one block per processor -- and the number of zones per block are fixed at compile time (usually) so doesn't that mean you're increasing the amount of work proportional to the number of processors you use? In that case, seeing a constant "speedup" suggests good weak scaling.</div><div><br></div><div>Best,<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">--------<div>Ryan</div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 5, 2023 at 4:17 PM Leland Ellison <<a href="mailto:c.leland.ellison@gmail.com" target="_blank">c.leland.ellison@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Pedro,</div><div><br></div><div>I suspect your per-block zone counts are too low to see benefits of adding more procs at this point. The details will depend on your specific problem and hardware of course, but when I've done strong scaling studies I've found (rapidly) diminishing returns to adding more procs once I fall below about ~1000 zones per proc. I think this is what you're seeing in your nxb=nxy=16 and nxb=nxy=28 cases. If your scaling study continues up to several thousand zones per proc, I bet you'll see more of the expected behavior.</div><div><br></div><div>Hope this helps!</div><div>Lee</div><div><br></div><div> ________________</div><div>Leland Ellison PhD<div>Computational Physicist</div><div><a href="https://www.linkedin.com/in/clelandellison/" target="_blank">https://www.linkedin.com/in/clelandellison/</a><br></div><div><a href="https://scholar.google.com/citations?user=1rfcVWgAAAAJ" target="_blank">https://scholar.google.com/citations?user=1rfcVWgAAAAJ</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 5, 2023 at 6:46 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="ES">
<div>
<p class="MsoNormal"><span>Hi Paco,<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">As far as I know, using uniform grid fixes the number of blocks as one per processor. Am I wrong? Do you mean to fix nxb and nyb while varying the cores?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>De:</b> Francisco Holguin <<a href="mailto:opaco@umich.edu" target="_blank">opaco@umich.edu</a>> <br>
<b>Enviado el:</b> miércoles, 5 de abril de 2023 15:17<br>
<b>Para:</b> pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>><br>
<b>CC:</b> <a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<b>Asunto:</b> Re: [FLASH-USERS] Scaling problems<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Pedro,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">What if you just fix the number of blocks, and vary the cores?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-Paco<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 5, 2023 at 5:13 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to scale up on computational resources and I came across a few issues. First of all, I am running the same +ug example (a modification of 2D Zpinch
template) varying the number of cores, nxb and nyb but it shows no speed up as the number of cores increase (I am tuning Iprocs, Jprocs, nxb and nyb to always get an approximately equal grid).
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Furthermore, at a certain number of cores the program execution interrupts, and I get a SIGBUS error (which I attach to this message). Am I missing something?
Is there any additional thing to consider?</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I will also attach the log file of one successful run using 36 cores and nxb=nyb=16 (which shows little or no speed up in comparison to a run on 12 cores and
nxb=nyb=28). Thank you in advance for any help.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Pedro</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" rel="noreferrer" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><br>
</div></blockquote></div>
_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" rel="noreferrer" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><br>
</blockquote></div>
</blockquote></div>