<div dir="ltr">Dear FLASH users,<div><br></div><div>I'm seeing a problem for quite some time on our local cluster which I don't seem able to get rid of. Whenever I run a sufficiently large simulation, it will fail sooner or later while paramesh writes this to the output:</div><div><br></div><div><font face="monospace, monospace"> 45251 8.5418E-01 5.7059E-06 ( 7.080E-02, 8.350E-02, 0.000E+00) | 5.706E-06</font><br></div><div><div><span style="font-family:monospace,monospace"> iteration, no. not moved = 0 4629</span><br></div><div><font face="monospace, monospace"> iteration, no. not moved = 1 3</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 2 1</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 3 1</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 4 1</font></div><div><font face="monospace, monospace">[...]</font></div><div><div><font face="monospace, monospace"> iteration, no. not moved = 98 1</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 99 1</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 100 1</font></div><div><font face="monospace, monospace"> ERROR: could not move all blocks in amr_redist_blk</font></div><div><font face="monospace, monospace"> Try increasing maxblocks or use more processors</font></div><div><font face="monospace, monospace"> nm2_old, nm2 = 1 1</font></div><div><font face="monospace, monospace"> ABORTING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</font></div></div><div><br></div><div>My current simulation (2 dimensions) runs on 64 cores with maxblocks set to 2'000, allowing a total maximum number of blocks of 128'000. With the current simulation I have 26'470 blocks in total (19'853 leaf blocks), so I'm well below that limit.</div><div><br></div><div>Accordingly, increasing maxblocks doesn't make any difference, neither does allowing "nit" to go to a higher value than 100. Running on an even more cores is no solution, either (initially, I have been running on only 16, then 32 and now 64 cores).</div><div><br></div><div>It may or not be interesting that I'm using the Intel compilers (v17.0) along with Intel MPI (v5.0.3).</div><div><br></div><div>It happens both with FLASH 4.3 and 4.4 (I have not tried other versions of FLASH). I cannot reporduce this issue reliably. Restarting from one of the recent checkpoints lets it run through perfectly fine:</div><div><br></div><div><font face="monospace, monospace"> 45251 8.5418E-01 5.7059E-06 ( 7.080E-02, 8.350E-02, 0.000E+00) | 5.706E-06</font><br></div><div><div><font face="monospace, monospace"> iteration, no. not moved = 0 4629</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 1 2</font></div><div><font face="monospace, monospace"> iteration, no. not moved = 2 0</font></div><div><font face="monospace, monospace"> refined: total leaf blocks = 19853</font></div><div><font face="monospace, monospace"> refined: total blocks = 26470</font></div></div><div><font face="monospace, monospace"> 45252 8.5419E-01 5.7065E-06 ( 7.080E-02, 8.350E-02, 0.000E+00) | 5.707E-06</font><br></div><div><br></div><div>Do you know how to prevent this error from happening or - if not - if it is safe to remove the corresponding MPI_ABORT entirely and just work with one block not being shifted around correctly?</div><div>Best regards,</div><div>Dominik</div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Dominik Derigs<br>I. Physikalisches Institut<br>Universität zu Köln<br>Zülpicher Straße 77<br>50937 Köln<br>GERMANY</div><div dir="ltr"><br></div><div dir="ltr"><a href="https://hera.ph1.uni-koeln.de/~derigs/" target="_blank">https://hera.ph1.uni-koeln.de/~derigs/</a><br><br>Tel. (+49|0) 221 470-8352<br>Fax. (+49|0) 221 470-5162<br><br>Diese Email ist vertraulich und nur für den angegebenen Empfänger bestimmt. Zugang, Freigabe, die Kopie, die Verteilung oder Weiterleitung durch jemand anderen außer dem Empfänger selbst ist verboten und kann eine kriminelle Handlung sein. Bitte löschen Sie die Email, wenn Sie sie durch einen Fehler erhalten haben und informieren Sie den Absender.<br><br>This email and any files transmitted with it may contain confidential and/or privileged material and is intended only for the person or entity to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately and delete this material from all known records.</div></div></div></div></div></div></div></div>
</div></div>