<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Dear FLASH users,<br>
<br>
Following up on the previous suggestions, I tried those parameters,
but I have now run into a different issue. One of the flash4
processes terminates with an MPI_ABORT, leaving no error message—not
even in the flash.log. This happens every time immediately after
grid refinement, and specifically only when I dynamically change the
maximum refinement level over time.<br>
<br>
To troubleshoot, I compiled the code in debug mode and ran it from a
checkpoint just before the crash. I used the following Open MPI
parameters to keep the crashed process alive in memory:<br>
<br>
export OMPI_MCA_opal_abort_print_stack=1<br>
export OMPI_MCA_opal_abort_delay=10000<br>
<br>
This allowed me to attach GDB to the hanging process. I investigated
the process that died and obtained this backtrace:
<p>#0 0x00007fdcd9bd69fd in nanosleep () from /lib64/libc.so.6<br>
#1 0x00007fdcd9bd6894 in sleep () from /lib64/libc.so.6<br>
#2 0x00007fdcd91362de in opal_delay_abort () from
/home/zajanond/openmpi/lib/libopen-pal.so.80<br>
#3 0x00007fdcdad789e5 in ompi_mpi_abort () from
/home/zajanond/openmpi/lib/libmpi.so.40<br>
#4 0x000000000104d5f1 in hypre_CAlloc ()<br>
#5 0x00000000010f54b9 in hypre_ParCSRCommPkgCreateApart_core ()<br>
#6 0x00000000010f5e58 in hypre_ParCSRCommPkgCreateApart ()<br>
#7 0x0000000000fb32b7 in hypre_MatvecCommPkgCreate ()<br>
#8 0x0000000000fc6373 in hypre_ParCSRMatrixMatvecOutOfPlaceHost
()<br>
#9 0x0000000000ff7948 in hypre_SStructMatvecCompute ()<br>
#10 0x0000000000ff7acd in hypre_SStructMatvec ()<br>
#11 0x0000000000ff1469 in HYPRE_SStructMatrixMatvec ()<br>
#12 0x0000000000febf9b in hypre_sstructmatrixmatvec_ ()<br>
#13 0x00000000004c6c53 in grid_advanceimplicit (problem=...,
dt=6.1905819767663258e-13) at Grid_advanceImplicit.F90:165<br>
#14 0x0000000000447493 in diffuse_solve (problem=...,
dt=6.1905819767663258e-13) at Diffuse_solve.F90:24<br>
#15 0x000000000071d0b2 in diff_advancevisc (blockcount=16,
blocklist=..., dt=6.1905819767663258e-13, pass=<error reading
variable: Cannot access memory at a
ddress 0x0>) at diff_advanceVisc.F90:258<br>
#16 0x0000000000409b6c in diffuse (blockcount=16, blocklist=...,
dt=6.1905819767663258e-13, pass=<error reading variable: Cannot
access memory at address 0x
0>) at Diffuse.F90:79<br>
#17 0x0000000000452647 in driver_evolveflash () at
Driver_evolveFlash.F90:306<br>
#18 0x00000000004b8d78 in flash () at Flash.F90:51<br>
#19 0x00000000004b8dd7 in main (argc=1, argv=0x7fff41e1ef00) at
Flash.F90:43<br>
#20 0x00007fdcd9b33555 in __libc_start_main () from
/lib64/libc.so.6<br>
#21 0x0000000000407139 in _start ()<br>
<br>
</p>
<p>The backtrace is consistent with every failure. </p>
The crash occurs at hypre_CAlloc(), which clearly points to a memory
allocation issue. However, during the failure, the node still has
around 80 GB of free RAM, 100 GB on HDD. I tried doubling the
maximum number of blocks, but this did not resolve the issue.
Sometimes the crash happens at a different simulation time, but in
all my tests, it consistently occurs right after refinement.<br>
<br>
Thank you in advance for your time and any ideas on how to solve
this.<br>
<br>
Sincerely,<br>
<br>
Ondřej Zajan<br>
Ph.D. Student<br>
Faculty of Electrical Engineering, Czech Technical University in
Prague
<p> </p>
<p><br>
</p>
<div class="moz-cite-prefix">Dne 15.04.2026 v 15:58 ERNESTO
ZURBRIGGEN napsal(a):<br>
</div>
<blockquote type="cite"
cite="mid:CAE7HDOLU=uRXX+od44vwWcNAJ6gnJGtWOAs5EcCP4S_nOU1yBA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Dear Ondrej,
<div>an option for modifying the maximum refinement level during
runtime is using the runtime parameters: </div>
<div><font face="monospace">gr_lrefineMaxByTime;
gr_lrefineMaxRedTRef; gr_lrefmaxTimeValue_*
and gr_lrefmaxTime_*</font>. </div>
<div><br>
</div>
<div>Hope this helps. </div>
<div>Best, </div>
<div>Ernesto</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Wed, 15 Apr 2026 at 03:41,
Ondřej Zajan <<a href="mailto:zajanond@fel.cvut.cz"
moz-do-not-send="true" class="moz-txt-link-freetext">zajanond@fel.cvut.cz</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear
FLASH Users,<br>
<br>
During an ongoing simulation, I forced FLASH 4.8 to reduce the
maximum <br>
grid refinement level (lrefine_max) by adjusting the
refinement cutoffs. <br>
I saved this state to a checkpoint file and subsequently
initialized a <br>
new simulation with the reduced maximum refinement level.<br>
<br>
The simulation runs temporarily but eventually terminates with
the <br>
following error: "DRIVER_ABORT: Got nonzero ierr from <br>
HYPRE_SStructMatrixSetValues!".<br>
<br>
I suspect that the PARAMESH block tree requires rebuilding to
properly <br>
eliminate the highest refinement level inherited from the
initial <br>
simulation. I believe the failure occurs within the following
file: <br>
FLASH4.8\source\Grid\GridSolvers\HYPRE\MHD\paramesh\gr_hypreCreateMatrix1BlkMag.F90.<br>
<br>
Could anyone advise on how to accomplish this tree rebuild, or
suggest <br>
an alternative method for reducing the grid resolution during
runtime?<br>
<br>
Thank you in advance for your time and assistance.<br>
<br>
Sincerely,<br>
<br>
Ondřej Zajan<br>
Ph.D. Student<br>
Faculty of Electrical Engineering, Czech Technical University
in Prague<br>
_______________________________________________<br>
flash-users mailing list<br>
<a href="mailto:flash-users@flash.rochester.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">flash-users@flash.rochester.edu</a><br>
<br>
For list info, including unsubscribe:<br>
<a
href="https://flash.rochester.edu/mailman/listinfo/flash-users"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://flash.rochester.edu/mailman/listinfo/flash-users</a><br>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
flash-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:flash-users@flash.rochester.edu">flash-users@flash.rochester.edu</a>
For list info, including unsubscribe:
<a class="moz-txt-link-freetext" href="https://flash.rochester.edu/mailman/listinfo/flash-users">https://flash.rochester.edu/mailman/listinfo/flash-users</a>
</pre>
</blockquote>
</body>
</html>