<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Another point about running with uniform grid…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If you use +ug, then you are setting a fixed block size during the setup. You then have to set these parameters in your par file: iProcs, jProcs, kProcs. As mentioned, these determine the number of blocks
and the number of processors you must use.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">You can instead run setup with +ug +nofbs. This is “no fixed block size”. In this case, you set these parameters in your par file: IGridSize, jGridSize, kGridSize (in addition to iProcs, jProcs, kProcs). Now
the ‘Procs’ parameters still need to match the number of processors being used, but the block size is determined by iGridSize/iProcs, jGridSize/jProcs, kGridSize/kProcs. So, in some sense, this might make it simpler to do strong or weak scaling.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">For strong scaling, keep the ‘GridSize’ parameters constant and increase the ‘Procs’ parameters. Just don’t increase ‘Procs’ so much that your block size becomes < 8 cells (or start with larger ‘GridSize’
values).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">For weak scaling, increase the ‘GridSize’ and ‘Procs’ parameters so that GridSize/Procs remains constant.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">--</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Eddie Hansen</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Research Scientist</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Flash Center for Computational Science</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">flash-users <flash-users-bounces@flash.rochester.edu> on behalf of Ryan Farber <rjfarber@umich.edu><br>
<b>Date: </b>Wednesday, April 5, 2023 at 2:34 PM<br>
<b>To: </b>Leland Ellison <c.leland.ellison@gmail.com><br>
<b>Cc: </b>flash-users@flash.rochester.edu <flash-users@flash.rochester.edu><br>
<b>Subject: </b>Re: [FLASH-USERS] Scaling problems<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks to Lee for pointing this out and sorry to Pedro for glossing over your initial message where you mention that you tune iprocs, jprocs, nxb, and nyb to have about the same number of grid points (so indeed
looking at strong scaling rather than weak scaling).<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">One more point regarding the sigbus issue is that perhaps adding mcmodel=large in your FFLAGS_* (in your Makefile.h) will help -- if you're on an Intel system anyway this enforces absolute addressing whereas
the default mcmodel=small uses relative addressing. From what I recall I had a ~2% performance decrease switching from mcmodel=large (but from mcmodel=medium) but it helped with memory issues in the past at large core counts.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<br clear="all">
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">--------<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Ryan<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Wed, Apr 5, 2023 at 5:06 PM Ryan Farber <<a href="mailto:rjfarber@umich.edu">rjfarber@umich.edu</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">One point of follow-up regarding your sigbus error - it looks like this is a memory access error. I'm wondering if you're requesting more logical cores than physical cores exist on the machine. I've found
that to be problematic in the past.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">It sounds like Lee might have the answer to your issue regarding too few zones per proc. One point I'm confused about though is whether you're studying strong or weak scaling. Based on your response to Paco
and I checked the FLASH users guide, for uniform grid mode there's one block per processor -- and the number of zones per block are fixed at compile time (usually) so doesn't that mean you're increasing the amount of work proportional to the number of processors
you use? In that case, seeing a constant "speedup" suggests good weak scaling.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<br clear="all">
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">--------<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Ryan<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Wed, Apr 5, 2023 at 4:17 PM Leland Ellison <<a href="mailto:c.leland.ellison@gmail.com" target="_blank">c.leland.ellison@gmail.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">I suspect your per-block zone counts are too low to see benefits of adding more procs at this point. The details will depend on your specific problem and hardware of course, but when I've done strong scaling
studies I've found (rapidly) diminishing returns to adding more procs once I fall below about ~1000 zones per proc. I think this is what you're seeing in your nxb=nxy=16 and nxb=nxy=28 cases. If your scaling study continues up to several thousand zones per
proc, I bet you'll see more of the expected behavior.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hope this helps!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Lee<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> ________________<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Leland Ellison PhD<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Computational Physicist<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://www.linkedin.com/in/clelandellison/" target="_blank">https://www.linkedin.com/in/clelandellison/</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://scholar.google.com/citations?user=1rfcVWgAAAAJ" target="_blank">https://scholar.google.com/citations?user=1rfcVWgAAAAJ</a><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Wed, Apr 5, 2023 at 6:46 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">Hi Paco,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">As far as I know, using uniform grid fixes the number of blocks as one per processor. Am I wrong? Do you mean to fix nxb and nyb while varying the
cores?</span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="ES" style="font-size:11.0pt">De:</span></b><span lang="ES" style="font-size:11.0pt"> Francisco Holguin <<a href="mailto:opaco@umich.edu" target="_blank">opaco@umich.edu</a>>
<br>
<b>Enviado el:</b> miércoles, 5 de abril de 2023 15:17<br>
<b>Para:</b> pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>><br>
<b>CC:</b> <a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<b>Asunto:</b> Re: [FLASH-USERS] Scaling problems<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">What if you just fix the number of blocks, and vary the cores?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">-Paco<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">On Wed, Apr 5, 2023 at 5:13 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>>
wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I am trying to scale up on computational resources and I came across a few issues. First of all, I am running the same +ug example (a modification
of 2D Zpinch template) varying the number of cores, nxb and nyb but it shows no speed up as the number of cores increase (I am tuning Iprocs, Jprocs, nxb and nyb to always get an approximately equal grid).
</span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Furthermore, at a certain number of cores the program execution interrupts, and I get a SIGBUS error (which I attach to this message). Am I missing
something? Is there any additional thing to consider?</span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I will also attach the log file of one successful run using 36 cores and nxb=nyb=16 (which shows little or no speed up in comparison to a run on 12
cores and nxb=nyb=28). Thank you in advance for any help.</span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Pedro</span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><span lang="ES" style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="ES" style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>