<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EstiloCorreo20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ES" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Thank you very much for your replies. I will try with a higher number of zones per processor and if the sigbus error persists after some tests, I will message again.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Thank you all, once again.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Pedro<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt">De:</span></b><span style="font-size:11.0pt"> flash-users <flash-users-bounces@flash.rochester.edu>
<b>En nombre de </b>Hansen, Eddie<br>
<b>Enviado el:</b> jueves, 6 de abril de 2023 1:56<br>
<b>Para:</b> Ryan Farber <rjfarber@umich.edu>; Leland Ellison <c.leland.ellison@gmail.com><br>
<b>CC:</b> flash-users@flash.rochester.edu<br>
<b>Asunto:</b> Re: [FLASH-USERS] Scaling problems<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hello all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Another point about running with uniform grid…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">If you use +ug, then you are setting a fixed block size during the setup. You then have to set these parameters in your par file: iProcs, jProcs, kProcs. As mentioned, these determine the number
of blocks and the number of processors you must use.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">You can instead run setup with +ug +nofbs. This is “no fixed block size”. In this case, you set these parameters in your par file: IGridSize, jGridSize, kGridSize (in addition to iProcs, jProcs,
kProcs). Now the ‘Procs’ parameters still need to match the number of processors being used, but the block size is determined by iGridSize/iProcs, jGridSize/jProcs, kGridSize/kProcs. So, in some sense, this might make it simpler to do strong or weak scaling.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">For strong scaling, keep the ‘GridSize’ parameters constant and increase the ‘Procs’ parameters. Just don’t increase ‘Procs’ so much that your block size becomes < 8 cells (or start with larger
‘GridSize’ values).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">For weak scaling, increase the ‘GridSize’ and ‘Procs’ parameters so that GridSize/Procs remains constant.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:black">--</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Eddie Hansen</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Research Scientist</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#181A1B">Flash Center for Computational Science</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span lang="EN-US" style="font-size:12.0pt;color:black">From:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black">flash-users <<a href="mailto:flash-users-bounces@flash.rochester.edu">flash-users-bounces@flash.rochester.edu</a>> on behalf of Ryan Farber <<a href="mailto:rjfarber@umich.edu">rjfarber@umich.edu</a>><br>
<b>Date: </b>Wednesday, April 5, 2023 at 2:34 PM<br>
<b>To: </b>Leland Ellison <<a href="mailto:c.leland.ellison@gmail.com">c.leland.ellison@gmail.com</a>><br>
<b>Cc: </b><a href="mailto:flash-users@flash.rochester.edu">flash-users@flash.rochester.edu</a> <<a href="mailto:flash-users@flash.rochester.edu">flash-users@flash.rochester.edu</a>><br>
<b>Subject: </b>Re: [FLASH-USERS] Scaling problems<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Thanks to Lee for pointing this out and sorry to Pedro for glossing over your initial message where you mention that you tune iprocs, jprocs, nxb, and nyb to have about the same number of grid
points (so indeed looking at strong scaling rather than weak scaling).<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">One more point regarding the sigbus issue is that perhaps adding mcmodel=large in your FFLAGS_* (in your Makefile.h) will help -- if you're on an Intel system anyway this enforces absolute addressing
whereas the default mcmodel=small uses relative addressing. From what I recall I had a ~2% performance decrease switching from mcmodel=large (but from mcmodel=medium) but it helped with memory issues in the past at large core counts.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Best,<br clear="all">
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">--------<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Ryan<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">On Wed, Apr 5, 2023 at 5:06 PM Ryan Farber <<a href="mailto:rjfarber@umich.edu">rjfarber@umich.edu</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">One point of follow-up regarding your sigbus error - it looks like this is a memory access error. I'm wondering if you're requesting more logical cores than physical cores exist on the machine.
I've found that to be problematic in the past.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">It sounds like Lee might have the answer to your issue regarding too few zones per proc. One point I'm confused about though is whether you're studying strong or weak scaling. Based on your response
to Paco and I checked the FLASH users guide, for uniform grid mode there's one block per processor -- and the number of zones per block are fixed at compile time (usually) so doesn't that mean you're increasing the amount of work proportional to the number
of processors you use? In that case, seeing a constant "speedup" suggests good weak scaling.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Best,<br clear="all">
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">--------<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Ryan<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">On Wed, Apr 5, 2023 at 4:17 PM Leland Ellison <<a href="mailto:c.leland.ellison@gmail.com" target="_blank">c.leland.ellison@gmail.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">I suspect your per-block zone counts are too low to see benefits of adding more procs at this point. The details will depend on your specific problem and hardware of course, but when I've done
strong scaling studies I've found (rapidly) diminishing returns to adding more procs once I fall below about ~1000 zones per proc. I think this is what you're seeing in your nxb=nxy=16 and nxb=nxy=28 cases. If your scaling study continues up to several thousand
zones per proc, I bet you'll see more of the expected behavior.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hope this helps!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Lee<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> ________________<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Leland Ellison PhD<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Computational Physicist<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><a href="https://www.linkedin.com/in/clelandellison/" target="_blank">https://www.linkedin.com/in/clelandellison/</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><a href="https://scholar.google.com/citations?user=1rfcVWgAAAAJ" target="_blank">https://scholar.google.com/citations?user=1rfcVWgAAAAJ</a><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">On Wed, Apr 5, 2023 at 6:46 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi Paco,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">As far as I know, using uniform grid fixes the number of blocks as one per processor. Am I wrong? Do you mean to fix nxb and nyb while
varying the cores?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt">De:</span></b><span style="font-size:11.0pt"> Francisco Holguin <<a href="mailto:opaco@umich.edu" target="_blank">opaco@umich.edu</a>>
<br>
<b>Enviado el:</b> miércoles, 5 de abril de 2023 15:17<br>
<b>Para:</b> pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>><br>
<b>CC:</b> <a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<b>Asunto:</b> Re: [FLASH-USERS] Scaling problems<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi Pedro,<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">What if you just fix the number of blocks, and vary the cores?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">-Paco<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Wed, Apr 5, 2023 at 5:13 AM pedro romero <<a href="mailto:pedro.romero@greentownsbyfusion.com" target="_blank">pedro.romero@greentownsbyfusion.com</a>>
wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">I am trying to scale up on computational resources and I came across a few issues. First of all, I am running the same +ug example (a
modification of 2D Zpinch template) varying the number of cores, nxb and nyb but it shows no speed up as the number of cores increase (I am tuning Iprocs, Jprocs, nxb and nyb to always get an approximately equal grid).
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Furthermore, at a certain number of cores the program execution interrupts, and I get a SIGBUS error (which I attach to this message).
Am I missing something? Is there any additional thing to consider?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">I will also attach the log file of one successful run using 36 cores and nxb=nyb=16 (which shows little or no speed up in comparison
to a run on 12 cores and nxb=nyb=28). Thank you in advance for any help.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Pedro</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu" target="_blank">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a href="https://flash.rochester.edu/mailman/listinfo/flash-users" target="_blank">https://flash.rochester.edu/mailman/listinfo/flash-users</a><o:p></o:p></span></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>