<div dir="ltr"><div>Hi Josh,</div><div><br></div><div>I solved the "perfmon" error by copying Timers_data.F90 to my problem directory and setting tmr_maxTimerParents = 1000.</div><div><br></div><div>Explanation: I noticed that the strings in call Timers_start/stop seemed to be much greater than 30 (for me it is of order 100). <br></div><div>The "perfmon" message is in Timers_start.F90 for the case that j < 0 where j gets its value from tmr_stackListAdd().</div><div>tmr_stackListAdd is a subroutine in tmr_stackLib.F90 which passes -1 if the referenced string == tmr_maxTimerParents.<br></div><div><br></div><div class="gmail_extra">However, the "perfmon" message doesn't actually throw an exception. I forgot that I've had runs (probably the static mesh refinement ones too but I can't check with Stampede2 down for maintenance today) that run okay even with this perfmon error.</div><div class="gmail_extra"><br></div><div class="gmail_extra">My actual error message (which I'll still have to debug) is:</div><div>DRIVER_ABORT: Error: Grid_getBlkBoundBox, blockId out of bounds</div><div>blockID = 0</div><div><br></div><div>I'm guessing your error is different.</div><div><br></div><div>Still, that solves one mystery.</div><div><br></div><div>Best,<br></div><div><div class="gmail_extra"><div><div class="m_3127045564360021768gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">--------<div>Ryan</div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Sep 11, 2018 at 4:34 PM, Joshua Wall <span dir="ltr"><<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Ryan,</div><div><br></div><div> Another clue I just found that probably supports Klaus's theory of some timer started that didn't get stopped is the following I found in my log file:</div><div><br></div><div>[ 09-10-2018 23:31:07.824 ] [Timers_getSummary]: Not writing timer max/min/avg values because not all processors had same timers</div><div><br></div><div>In case you find the same. To be clear on what I'm running with exactly:</div><div><br></div><div>Hydro: USM<br></div><div>Grav: MG</div><div>+cube16 <br></div><div>AMR</div><div>+pm4dev</div><div>+supportPPMupwind <br></div><div>maxblocks = 50</div><div>-a -3d</div><div>optimization: -03<br></div><div>MPI: OpenMPI 1.10.02</div><div>compiler: gnu 4.8.0 (with the -O0 fix for mpi_amr_1blk_guardcell.o)</div><div><br></div><div><br></div><div>Klaus,</div><div><br></div><div> Would it be possible to call Timer_getSummary() after each call to the individual units in Driver_evolveFlash in an attempt to find the unit responsible by "bisection"? Essentially the same log message as above should print right after some unit gets the timers on the different processors out of sync. Does that seem okay to try?<br></div><div><br></div><div>Cordially,<br></div><div><br></div><div>Josh<br></div><div><div class="m_3127045564360021768h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Sep 11, 2018 at 3:03 PM Joshua Wall <<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello Ryan and Klaus,</div><div><br></div><div> Ryan, yes I am running in AMR. Per Klaus's recommendation I just did:</div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">josh@iris2:~/flash/src/flash4.<wbr>2.2-rad/object$ grep -iIn "call Timers" * > timers_grep.txt <br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"></span>Which will give me a list of calls to Timers_start and Timers_stop in one text file. After I check this file to ensure that all have pairs, I'll have to check each FLASH file for "if () then" and "#ifdef" statements that might have isolated one of the timer calls. Feel free to also do this as a check in case I miss anything. Whoever finds it first can post what we find here (or if we don't find anything).<br></div><div><br></div><div>Cordially,</div><div><br></div><div>Josh<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Sep 11, 2018 at 2:54 PM Ryan Farber <<a href="mailto:rjfarber@umich.edu" target="_blank">rjfarber@umich.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Josh/Klaus,</div><div><br></div><div>I'm also suffering this problem. It only appears for me when I run my problem in AMR. That is, it doesn't happen if I have nrefs=100000. Josh, do you know if that is also the case for you?<br></div><div><br></div><div>Klaus, thanks for the explanation I will look for Timers_start/stop pairs to see if that is the issue.<br></div><div><br></div><div class="gmail_extra">Best,<br clear="all"></div><div class="gmail_extra"><div><div class="m_3127045564360021768m_-9132901358664290924m_880814478092115241m_8643547799757267749gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">--------<div>Ryan</div></div></div></div></div></div></div></div></div><div dir="ltr"><div class="gmail_extra">
<br><div class="gmail_quote">On Tue, Sep 11, 2018 at 2:44 PM, Klaus Weide <span dir="ltr"><<a href="mailto:klaus@flash.uchicago.edu" target="_blank">klaus@flash.uchicago.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, 11 Sep 2018, Joshua Wall wrote:<br>
<br>
> Hello FLASH users,<br>
> <br>
> I'm attempting to track down a strange occurrence in one of my runs (using<br>
> FLASH 4.2.2), where I get the following error related to timing of the<br>
> multigrid solver:<br>
> <br>
> 6043 perfmon: ran out of space for timer, "gr_hgBndry", cannot time<br>
> this timer with perfmon<br>
> 6044 perfmon: ran out of space for timer, "work copy", cannot time<br>
> this timer with perfmon<br>
> 6045 perfmon: ran out of space for timer, "gr_hgGuardCell", cannot<br>
> time this timer with perfmon<br>
<br>
</span>This could happen if Timers_start / Timers_stop call pairs are incomplete,<br>
or not properly nested.<br>
<br>
It is possible that the offending unbalanced code is not in the multigrid <br>
solver at all, but in an entirely different part of the code - the error <br>
could just happen to cause an overflow of the Timers stack there first.<br>
<span class="m_3127045564360021768m_-9132901358664290924m_880814478092115241m_8643547799757267749HOEnZb"><font color="#888888"><br>
Klaus<br>
</font></span></blockquote></div><br></div></div></blockquote></div>-- <br><div dir="ltr" class="m_3127045564360021768m_-9132901358664290924m_880814478092115241gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div><div><div><div>Joshua Wall<br></div>Doctoral Candidate<br></div>Department of Physics<br></div>Drexel University<br></div><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g" target="_blank">3141 Chestnut Street</a><br></div><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g" target="_blank">Philadelphia, PA 19104</a><br></div></div>
</blockquote></div></div></div></div><div class="m_3127045564360021768HOEnZb"><div class="m_3127045564360021768h5">-- <br><div dir="ltr" class="m_3127045564360021768m_-9132901358664290924gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div><div><div><div>Joshua Wall<br></div>Doctoral Candidate<br></div>Department of Physics<br></div>Drexel University<br></div><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g" target="_blank">3141 Chestnut Street</a><br></div><a href="https://maps.google.com/?q=3141+Chestnut+Street+Philadelphia,+PA+19104&entry=gmail&source=g" target="_blank">Philadelphia, PA 19104</a><br></div></div>
</div></div></blockquote></div><br></div></div></div>