<div dir="ltr">Hi Malia :<div><br></div><div> To add to the other comments, the key point is that even though FLASH employs adaptive mesh refinement, it allocates all memory for all possible blocks that will ever exist in the simulation at build time. This is in contrast to other packages where the memory allocation is also fully dynamic.</div><div><br></div><div> To circumvent your memory issues, if you are certain you will only be using a 512^2 mesh, you should either employ the uniform grid option as I suggested earlier (+ug on setup) or otherwise dial your MAXBLOCKS may down. Additionally, as John suggested, you can experiment with the block size to optimize performance.</div><div><br></div><div> -B</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 1:00 PM, John ZuHone <span dir="ltr"><<a href="mailto:jzuhone@gmail.com" target="_blank">jzuhone@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">From a computational perspective, having blocks with that many zones is pretty impractical. FLASH’s performance is much better with far smaller sizes (such as 16x16). <div><br></div><div>I’d recommend picking nblock[xy] in the parameter file such that when you multiply it by whatever you pick for -n[xy]b (something like 16 or 32) yields 512. It should be fairly easy that writes a setup that maps from the initial condition to that pattern of blocks.</div><div><div class="h5"><div><br><div><blockquote type="cite"><div>On May 24, 2017, at 12:56 PM, Jenks, Malia T. <<a href="mailto:mjenks@ou.edu" target="_blank">mjenks@ou.edu</a>> wrote:</div><br class="m_-7378289271474074117Apple-interchange-newline"><div><div id="m_-7378289271474074117divtagdefaultwrapper" dir="ltr" style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="margin-top:0px;margin-bottom:0px">Tomek,</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Where does the 2000 come from?</div><div style="margin-top:0px;margin-bottom:0px">Maxblocks is auto set to 1000.</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">The point is I am reading in an external grid that is 512x512 and I want the FLASH grid to match it exactly so that I can do a trivial map of the physical quantities.</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">I'm missing something.</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Malia<br></div></div><hr style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline-block;width:628.171875px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important"></span><div id="m_-7378289271474074117divRplyFwdMsg" dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Jenks, Malia T.<br><b>Sent:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Wednesday, May 24, 2017 10:32:34 AM<br><b>To:</b><span class="m_-7378289271474074117Apple-converted-space"> </span><a href="mailto:flash-users@flash.uchicago.edu" target="_blank">flash-users@flash.<wbr>uchicago.edu</a><br><b>Subject:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Re: Question about grid size</font><div> </div></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div id="m_-7378289271474074117divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="margin-top:0px;margin-bottom:0px">Hmm, somehow Flash is statically allocating more than the Linux 2GB limit. I'm surprised that anything is statically allocated.<span class="m_-7378289271474074117Apple-converted-space"> </span><br></div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">I did a parallel compile but assuming (for debugging purposes) that I would just use 1 CPU. Thus, I set iProcs =1</div><div style="margin-top:0px;margin-bottom:0px">jProcs=1 and</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">setup -2d +cylindrical +usm -nxb=512 -nyb=512 -auto</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Then at link time I get:</div><div style="margin-top:0px;margin-bottom:0px"><br></div><p style="margin-top:0px;margin-bottom:0px"></p><div>Burn.o: In function `burn':<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:107:(.text+0xe3): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_useburn_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:115:(.text+0x2ad): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_useshockburn_<wbr>' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:145:(.text+0x604): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_useshockburn_<wbr>' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:172:(.text+0x897): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_<wbr>nucleartempmin_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:175:(.text+0x8bc): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_<wbr>nucleartempmax_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:174:(.text+0x8dc): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_useshockburn_<wbr>' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:173:(.text+0x8f1): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_<wbr>nucleardensmin_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:173:(.text+0x8f9): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_<wbr>nucleardensmax_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn.F90:181:(.text+0x961): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_<wbr>nuclearni56max_' defined in COMMON section in Burn_data.o<br>Burn_computeDt.o: In function `burn_computedt':<br>/home/mjenks/FLASH4.4/object/<wbr>Burn_computeDt.F90:108:(.text+<wbr>0x16): relocation truncated to fit: R_X86_64_PC32 against symbol `burn_data_mp_bn_useburn_' defined in COMMON section in Burn_data.o<br>/home/mjenks/FLASH4.4/object/<wbr>Burn_computeDt.F90:141:(.text+<wbr>0x69): additional relocation overflows omitted from the output<br>make: *** [flash4] Error 1<br><br>which means that I have > 2GB of static data see<span class="m_-7378289271474074117Apple-converted-space"> </span><span><a href="https://software.intel.com/en-us/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/268374" class="m_-7378289271474074117OWAAutoLink" id="m_-7378289271474074117LPlnk525102" target="_blank">https://software.intel.<wbr>com/en-us/forums/intel-<wbr>fortran-compiler-for-linux-<wbr>and-mac-os-x/topic/268374</a><br><br>which shouldn't be happening.<span class="m_-7378289271474074117Apple-converted-space"> </span><br><br>Ideas?<br><br>Malia<br></span><div id="m_-7378289271474074117LPBorder_GT_14956434990920.870054988879386" style="margin-bottom:20px;overflow:auto;width:641px;text-indent:0px"><table id="m_-7378289271474074117LPContainer_14956434990880.664723062318314" cellspacing="0" style="width:576.890625px;background-color:rgb(255,255,255);overflow:auto;padding-top:20px;padding-bottom:20px;margin-top:20px;border-top-width:1px;border-top-style:dotted;border-top-color:rgb(200,200,200);border-bottom-width:1px;border-bottom-style:dotted;border-bottom-color:rgb(200,200,200)"><tbody><tr valign="top" style="border-spacing:0px"><td id="m_-7378289271474074117TextCell_14956434990890.30660771925562713" colspan="2" style="vertical-align:top;padding:0px;display:table-cell"><div id="m_-7378289271474074117LPRemovePreviewContainer_14956434990890.3412652029947264"></div><div id="m_-7378289271474074117LPTitle_14956434990890.013272616547162386" style="color:rgb(0,120,215);font-weight:400;font-size:21px;font-family:wf_segoe-ui_light,'Segoe UI Light','Segoe WP Light','Segoe UI','Segoe WP',Tahoma,Arial,sans-serif;line-height:21px"><a id="m_-7378289271474074117LPUrlAnchor_14956434990900.6708587305907193" href="https://software.intel.com/en-us/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/268374" style="text-decoration:none" target="_blank">relocation truncated to fit: R_X86_64_PC32 - Intel® Software</a></div><div id="m_-7378289271474074117LPMetadata_14956434990910.920320941180578" style="margin:10px 0px 16px;color:rgb(102,102,102);font-weight:400;font-family:wf_segoe-ui_normal,'Segoe UI','Segoe WP',Tahoma,Arial,sans-serif;font-size:14px;line-height:14px"><a href="http://software.intel.com" target="_blank">software.intel.com</a></div><div id="m_-7378289271474074117LPDescription_14956434990920.15876753967591728" style="display:block;color:rgb(102,102,102);font-weight:400;font-family:wf_segoe-ui_normal,'Segoe UI','Segoe WP',Tahoma,Arial,sans-serif;font-size:14px;line-height:20px;max-height:100px;overflow:hidden">Hi, I'm trying to compile a big code that so far has been running fine.Now, we have surpassed the 2GB in statically allocated data limit. I knew that would bring us ...</div></td></tr></tbody></table></div><br></div><br><p style="margin-top:0px;margin-bottom:0px"></p></div><hr style="display:inline-block;width:628.171875px"><div id="m_-7378289271474074117divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Jenks, Malia T.<br><b>Sent:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Tuesday, May 23, 2017 4:29:17 PM<br><b>To:</b><span class="m_-7378289271474074117Apple-converted-space"> </span><a href="mailto:flash-users@flash.uchicago.edu" target="_blank">flash-users@flash.<wbr>uchicago.edu</a><br><b>Subject:</b><span class="m_-7378289271474074117Apple-converted-space"> </span>Question about grid size</font><div> </div></div><div><div id="m_-7378289271474074117divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="margin-top:0px;margin-bottom:0px">I tried to increase the number of cells in my 2d model to 512 by 512. It will not compile at this size and I get an error about static memory. Is there a way I can solve this?</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Malia Jenks</div></div></div></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Dr. Robert Fisher<br>Associate Professor / Graduate Program Director<br>University of Massachusetts/Dartmouth<br>Department of Physics<br>285 Old Westport Road<br>North Dartmouth, Massachusetts 02747<br><a href="mailto:robert.fisher@umassd.edu" target="_blank">robert.fisher@umassd.edu</a><br><a href="http://www.novastella.org" target="_blank">http://www.novastella.org</a></div></div>
</div>