<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title></title>
</head>
<body>
<div name="messageBodySection">
<div dir="auto">The log file will tell you how many leaf and total blocks there are after each refinement event. I’ve found that you can bring maxblocks down pretty close to max(totBlks)/ranks. But, you still want to have a little bit of head room for the AMR
to do its thing.<br>
<br>
Sean</div>
</div>
<div name="messageSignatureSection"><br>
<div class="matchFont">
<div dir="auto">----------------------------------------------------------------------
<div>Sean M. Couch, Ph.D.</div>
<div>Assistant Professor</div>
<div>Department of Physics and Astronomy</div>
<div>Department of Computational Mathematics, Science, and Engineering</div>
<div>Facility for Rare Isotope Beams</div>
<div>Michigan State University</div>
<div>567 Wilson Rd, 3260 BPS</div>
<div>East Lansing, MI 48824</div>
<div>(517) 884-5035 --- couch@pa.msu.edu --- www.pa.msu.edu/~couch</div>
</div>
</div>
</div>
<div name="messageReplySection">On Sep 24, 2019, 9:30 AM -0400, Slavin, Jonathan <jslavin@cfa.harvard.edu>, wrote:<br>
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Hi Sean,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Thanks, that's a good idea. I think that I may have been using more blocks than I needed. I'll try that out.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Is there a way, after the fact to determine how many blocks were necessary for a run?</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Thanks,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Jon</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 24, 2019 at 9:15 AM Couch, Sean <<a href="mailto:scouch@msu.edu">scouch@msu.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div>
<div name="messageBodySection">
<div dir="auto">Hi Jon,
<div dir="auto"><br>
</div>
<div dir="auto">There are various temporary/scratch arrays that are allocated at runtime. It could be the machine is running out of memory. Some arrays are sized according to maxblocks (or a _multiple_ thereof). Have you tried reducing maxblocks compiled into
the application? Also, if you can setup the code with `useFortran2003=True`, you will get some very useful memory usage statistics in the log file just before the main iteration loop info starts printing.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Sean</div>
</div>
</div>
<div name="messageSignatureSection"><br>
<div class="gmail-m_-2006631231577343930matchFont">
<div dir="auto">----------------------------------------------------------------------
<div>Sean M. Couch, Ph.D.</div>
<div>Assistant Professor</div>
<div>Department of Physics and Astronomy</div>
<div>Department of Computational Mathematics, Science, and Engineering</div>
<div>Facility for Rare Isotope Beams</div>
<div>Michigan State University</div>
<div>567 Wilson Rd, 3260 BPS</div>
<div>East Lansing, MI 48824</div>
<div>(517) 884-5035 --- <a href="mailto:couch@pa.msu.edu" target="_blank">couch@pa.msu.edu</a> ---
<a href="http://www.pa.msu.edu/~couch" target="_blank">www.pa.msu.edu/~couch</a></div>
</div>
</div>
</div>
<div name="messageReplySection">On Sep 23, 2019, 11:27 AM -0400, Slavin, Jonathan <<a href="mailto:jslavin@cfa.harvard.edu" target="_blank">jslavin@cfa.harvard.edu</a>>, wrote:<br>
<blockquote type="cite" class="gmail-m_-2006631231577343930spark_quote spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Hi Marissa,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Thanks for sharing your experience. I'm currently letting it run, as I mentioned, having changed the refinement criteria to remove pressure as a refine_var. Before I try anything more I'd like to see how that turns out. Running under gdb could take some time
since even using mpi with 10 cpus on my desktop it takes a couple hours before it starts getting into that mode where it keeps increasing the refinement. I could, however, compile with various checks enabled and at low optimization, which could produce useful
information.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
As for the units used, here they are (note that the drag unit under Particles/ParticlesForces and the dust unit under Particles/ParticlesMain are units that I have written):</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
FLASH Units used:</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Driver/DriverMain/Split<br>
Grid/GridBoundaryConditions<br>
Grid/GridMain/paramesh/interpolation/Paramesh4/prolong<br>
Grid/GridMain/paramesh/interpolation/prolong<br>
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/headers<br>
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source<br>
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/source<br>
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/utilities/multigrid<br>
Grid/GridParticles/GridParticlesMapFromMesh<br>
Grid/GridParticles/GridParticlesMapToMesh/Paramesh/MoveSieve<br>
Grid/GridParticles/GridParticlesMove/Sieve/BlockMatch<br>
Grid/GridParticles/GridParticlesMove/paramesh<br>
Grid/GridSolvers/HYPRE/paramesh<br>
IO/IOMain/hdf5/serial/PM<br>
IO/IOParticles/hdf5/serial<br>
Particles/ParticlesForces/shortRange/drag<br>
Particles/ParticlesInitialization<br>
Particles/ParticlesMain/active/dust<br>
Particles/ParticlesMapping/Quadratic<br>
Particles/ParticlesMapping/meshWeighting/MapToMesh<br>
PhysicalConstants/PhysicalConstantsMain<br>
RuntimeParameters/RuntimeParametersMain<br>
Simulation/SimulationMain<br>
flashUtilities/contiguousConversion<br>
flashUtilities/general<br>
flashUtilities/interpolation/oneDim<br>
flashUtilities/nameValueLL<br>
flashUtilities/rng<br>
flashUtilities/sorting/quicksort<br>
flashUtilities/system/memoryUsage/legacy<br>
monitors/Logfile/LogfileMain<br>
monitors/Timers/TimersMain/MPINative<br>
physics/Diffuse/DiffuseMain/Unsplit<br>
physics/Eos/EosMain/Gamma<br>
physics/Hydro/HydroMain/split/PPM/PPMKernel<br>
physics/materialProperties/Conductivity/ConductivityMain/PowerLaw<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Also note that this run uses Flash 4.3.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Regards,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Jon</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 23, 2019 at 11:02 AM Marissa Adams <<a href="mailto:madams@pas.rochester.edu" target="_blank">madams@pas.rochester.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote gmail-m_-2006631231577343930spark_quote spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #d35400;">
<div>
<div dir="ltr">Hi Jonathan,
<div><br>
</div>
<div>I've encountered something quite similar earlier this past summer. When I ran with AMR on a supercomputer, it would refine all the way, then crash via segfault. I then tried running the same executable under gdb, and it ran just fine. However, while running,
it wouldn't even register the refinement! And just pushed through the time step where it refined/crashed as if it were a pseudo-fixed grid. It would do this, even when I asked for only one level of refinement, or two. I am wondering if you can run yours under
gdb and see if it does something similar? Then perhaps we know we are dealing with the same "heisenbug" I've encountered. </div>
<div><br>
</div>
<div>After moving back to my local machine to debug further, I sorted through some uninitialized variables that may have been eating at the memory and causing that sort of crash. I added debug -O0 and additional -W flags to make sure it crashed appropriataely
for me to sort through the weeds. It was a process.... but perhaps list what units you're using and I can tell you if I have any cross over, and what variables I initialized that seemed to fix the problem. </div>
<div><br>
</div>
<div>Best,<br>
Marissa</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 23, 2019 at 9:05 AM Slavin, Jonathan <<a href="mailto:jslavin@cfa.harvard.edu" target="_blank">jslavin@cfa.harvard.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote gmail-m_-2006631231577343930spark_quote spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #34495e;">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Hi,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I've run into an issue with a simulation I'm running where, after evolving just fine, it begins to take more and more iterations during grid refinement until it fails. It's strange because I ran the same simulation on a different system with the same code and
same parameters and it worked just fine. I did use different versions of the Intel fortran compiler and the hardware (cpus) were a bit different. But in terms of the software it is the same.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
I'm currently trying again with refinement on pressure removed - previously I refined on density and pressure now I'm trying just refining on density.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
If anyone has any other suggestions, I'd like to hear them.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Thanks,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">
Jon</div>
<div><br>
</div>
--<br>
<div dir="ltr" class="gmail-m_-2006631231577343930gmail-m_4148645980993955406gmail-m_3464243699937507252gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="font-size: 12.6667px; font-family: Arial; white-space: pre-wrap; border: none;"></div>
<div dir="ltr">Jonathan D. Slavin</div>
<div dir="ltr">Astrophysicist - High Energy Astrophysics Division</div>
<div dir="ltr">Center for Astrophysics | Harvard & Smithsonian</div>
<div dir="ltr">Office: (617) 496-7981 | Cell: (781) 363-0035<br>
60 Garden Street | MS 83 | Cambridge, MA 02138</div>
</div>
<div dir="ltr"><span>
<p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="border: none;"></span><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br>
</span></p>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
--<br>
<div dir="ltr" class="gmail-m_-2006631231577343930gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="font-size: 12.6667px; font-family: Arial; white-space: pre-wrap; border: none;"></div>
<div dir="ltr">Jonathan D. Slavin</div>
<div dir="ltr">Astrophysicist - High Energy Astrophysics Division</div>
<div dir="ltr">Center for Astrophysics | Harvard & Smithsonian</div>
<div dir="ltr">Office: (617) 496-7981 | Cell: (781) 363-0035<br>
60 Garden Street | MS 83 | Cambridge, MA 02138</div>
</div>
<div dir="ltr"><span>
<p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="border: none;"></span><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br>
</span></p>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
--<br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="font-size: 12.6667px; font-family: Arial; white-space: pre-wrap; border: none;"></div>
<div dir="ltr">Jonathan D. Slavin</div>
<div dir="ltr">Astrophysicist - High Energy Astrophysics Division</div>
<div dir="ltr">Center for Astrophysics | Harvard & Smithsonian</div>
<div dir="ltr">Office: (617) 496-7981 | Cell: (781) 363-0035<br>
60 Garden Street | MS 83 | Cambridge, MA 02138</div>
</div>
<div dir="ltr"><span>
<p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/mJuf9zI6QC5w6HPVCinnLSkpp-RgufOfl7nw1E_aghDflNhusoRbZTuP_0x-RkECQRhIfvPxGMe6zSt6chyXZ7D5DdqtkZPp0q-TE8itKny8VWObWAiZ1aurYWDO0CtZgvJEXfzu" width="350" height="2" style="border: none;"></span><span style="font-size:9.5pt;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br>
</span></p>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>