<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.aqj
{mso-style-name:aqj;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">Hi,<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">I have been using FLASH for my simulations with diffusion (FALSH4). I want to run a simulation with resolution of 512 (i.e. nxb=nyb=nzb=8, AMRmin = AMRmax=7), the total size
of the simulation will be (512)x(512)x(512x12). I receive a memory errors on Stampede and Mira, although I account for max number of blocks per processor, and number of processors that I run on. I started with lower AMR on given number of processors and increased both of
them progressively. For example:<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">AMR levels number of procs requested number of procs ran on<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">min = 5 64 64<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">max = 5 <o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">min=6 512 512<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">max=6<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">min=7 4096 4096<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">max=7<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">The cases with AMR min=max=5 and 6 run, but AMR min=max=7 crashes with memory issue. Following are the error message:<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">forrtl: severe (41): insufficient virtual memory</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">Image PC Routine Line Source</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libintlc.so.5 00002B9B18CFFA1E Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libintlc.so.5 00002B9B18CFE4B6 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libifcore.so.5 00002B9B185CC01E Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libifcore.so.5 00002B9B1853BB1E Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libifcore.so.5 00002B9B1858A9B9 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 00000000006FFA85 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 000000000078D82F Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 000000000046BD9C Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 00000000004810DD Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 000000000061B5C1 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 00000000004800A6 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 000000000044CF96 Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 00000000004573FA Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">flash4 000000000042D86C Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">libc.so.6 0000003CA6A1ED5D Unknown Unknown Unknown</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">flash4 000000000042D769 Unknown Unknown Unknown</span><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">I have also, run AMR min=max= 7 on another Mira on 8192 processors, which runs into another error:<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">"mpi_amr_comm_setup.F90", line 504: 1525-108 Error encountered while attempting to allocate a data object. The program will stop.</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span class="aqj"><span style="font-size:10.0pt;color:#1F497D">2015-06-09 00:45:41.006</span></span><span style="font-size:10.0pt;color:#1F497D"> (INFO ) [0x40001aebde0] MIR-04000-37331-512:1211855:ibm.runjob.client.Job: exited with status 1</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span class="aqj"><span style="font-size:10.0pt;color:#1F497D">2015-06-09 00:45:41.006</span></span><span style="font-size:10.0pt;color:#1F497D"> (WARN ) [0x40001aebde0] MIR-04000-37331-512:1211855:ibm.runjob.client.Job: normal termination with status 1 from
rank 1538</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span class="aqj"><span style="font-size:10.0pt;color:#1F497D">2015-06-09 00:45:41.006</span></span><span style="font-size:10.0pt;color:#1F497D"> (INFO ) [0x40001aebde0] tatu.runjob.client: task exited with status 1</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span class="aqj"><span style="font-size:10.0pt;color:#1F497D">2015-06-09 00:45:41.006</span></span><span style="font-size:10.0pt;color:#1F497D"> (INFO ) [0x400016334e0] 15540:tatu.runjob.monitor: monitor terminating</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span class="aqj"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2015-06-09 00:45:41.012</span></span><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
(INFO ) [0x40001aebde0] tatu.runjob.client: monitor completed</span><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black">where, the last lines of the log file are:<o:p></o:p></span></p>
<p style="background:white"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">[ <span class="aqj">06-09-2015 00:42:30.891</span> ] [GRID amr_refine_derefine]: refinement complete</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">[ <span class="aqj">06-09-2015 00:45:08.546</span> ] [GRID gr_expandDomain]: iteration=6, create level=7</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-size:10.0pt;color:#1F497D">[ <span class="aqj">06-09-2015 00:45:39.352</span> ] [mpi_amr_comm_setup]: buffer_dim_send=4081925, buffer_dim_recv=3988369</span><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
</div>
</body>
</html>