[FLASH-USERS] AMR grid with paramesh: jump to lower refinement level during simulation

Thu Apr 23 02:17:48 EDT 2026

Dear FLASH users,

Following up on the previous suggestions, I tried those parameters, but 
I have now run into a different issue. One of the flash4 processes 
terminates with an MPI_ABORT, leaving no error message—not even in the 
flash.log. This happens every time immediately after grid refinement, 
and specifically only when I dynamically change the maximum refinement 
level over time.

To troubleshoot, I compiled the code in debug mode and ran it from a 
checkpoint just before the crash. I used the following Open MPI 
parameters to keep the crashed process alive in memory:

export OMPI_MCA_opal_abort_print_stack=1
export OMPI_MCA_opal_abort_delay=10000

This allowed me to attach GDB to the hanging process. I investigated the 
process that died and obtained this backtrace:

#0  0x00007fdcd9bd69fd in nanosleep () from /lib64/libc.so.6
#1  0x00007fdcd9bd6894 in sleep () from /lib64/libc.so.6
#2  0x00007fdcd91362de in opal_delay_abort () from 
/home/zajanond/openmpi/lib/libopen-pal.so.80
#3  0x00007fdcdad789e5 in ompi_mpi_abort () from 
/home/zajanond/openmpi/lib/libmpi.so.40
#4  0x000000000104d5f1 in hypre_CAlloc ()
#5  0x00000000010f54b9 in hypre_ParCSRCommPkgCreateApart_core ()
#6  0x00000000010f5e58 in hypre_ParCSRCommPkgCreateApart ()
#7  0x0000000000fb32b7 in hypre_MatvecCommPkgCreate ()
#8  0x0000000000fc6373 in hypre_ParCSRMatrixMatvecOutOfPlaceHost ()
#9  0x0000000000ff7948 in hypre_SStructMatvecCompute ()
#10 0x0000000000ff7acd in hypre_SStructMatvec ()
#11 0x0000000000ff1469 in HYPRE_SStructMatrixMatvec ()
#12 0x0000000000febf9b in hypre_sstructmatrixmatvec_ ()
#13 0x00000000004c6c53 in grid_advanceimplicit (problem=..., 
dt=6.1905819767663258e-13) at Grid_advanceImplicit.F90:165
#14 0x0000000000447493 in diffuse_solve (problem=..., 
dt=6.1905819767663258e-13) at Diffuse_solve.F90:24
#15 0x000000000071d0b2 in diff_advancevisc (blockcount=16, 
blocklist=..., dt=6.1905819767663258e-13, pass=<error reading variable: 
Cannot access memory at a                      ddress 0x0>) at 
diff_advanceVisc.F90:258
#16 0x0000000000409b6c in diffuse (blockcount=16, blocklist=..., 
dt=6.1905819767663258e-13, pass=<error reading variable: Cannot access 
memory at address 0x              0>) at Diffuse.F90:79
#17 0x0000000000452647 in driver_evolveflash () at 
Driver_evolveFlash.F90:306
#18 0x00000000004b8d78 in flash () at Flash.F90:51
#19 0x00000000004b8dd7 in main (argc=1, argv=0x7fff41e1ef00) at Flash.F90:43
#20 0x00007fdcd9b33555 in __libc_start_main () from /lib64/libc.so.6
#21 0x0000000000407139 in _start ()

The backtrace is consistent with every failure.

The crash occurs at hypre_CAlloc(), which clearly points to a memory 
allocation issue. However, during the failure, the node still has around 
80 GB of free RAM, 100 GB on HDD. I tried doubling the maximum number of 
blocks, but this did not resolve the issue. Sometimes the crash happens 
at a different simulation time, but in all my tests, it consistently 
occurs right after refinement.

Thank you in advance for your time and any ideas on how to solve this.

Sincerely,

Ondřej Zajan
Ph.D. Student
Faculty of Electrical Engineering, Czech Technical University in Prague

Dne 15.04.2026 v 15:58 ERNESTO ZURBRIGGEN napsal(a):
> Dear Ondrej,
> an option for modifying the maximum refinement level during runtime is 
> using the runtime parameters:
> gr_lrefineMaxByTime; gr_lrefineMaxRedTRef; gr_lrefmaxTimeValue_* 
> and gr_lrefmaxTime_*.
>
> Hope this helps.
> Best,
> Ernesto
>
> On Wed, 15 Apr 2026 at 03:41, Ondřej Zajan <zajanond at fel.cvut.cz> wrote:
>
>     Dear FLASH Users,
>
>     During an ongoing simulation, I forced FLASH 4.8 to reduce the
>     maximum
>     grid refinement level (lrefine_max) by adjusting the refinement
>     cutoffs.
>     I saved this state to a checkpoint file and subsequently
>     initialized a
>     new simulation with the reduced maximum refinement level.
>
>     The simulation runs temporarily but eventually terminates with the
>     following error: "DRIVER_ABORT: Got nonzero ierr from
>     HYPRE_SStructMatrixSetValues!".
>
>     I suspect that the PARAMESH block tree requires rebuilding to
>     properly
>     eliminate the highest refinement level inherited from the initial
>     simulation. I believe the failure occurs within the following file:
>     FLASH4.8\source\Grid\GridSolvers\HYPRE\MHD\paramesh\gr_hypreCreateMatrix1BlkMag.F90.
>
>     Could anyone advise on how to accomplish this tree rebuild, or
>     suggest
>     an alternative method for reducing the grid resolution during runtime?
>
>     Thank you in advance for your time and assistance.
>
>     Sincerely,
>
>     Ondřej Zajan
>     Ph.D. Student
>     Faculty of Electrical Engineering, Czech Technical University in
>     Prague
>     _______________________________________________
>     flash-users mailing list
>     flash-users at flash.rochester.edu
>
>     For list info, including unsubscribe:
>     https://flash.rochester.edu/mailman/listinfo/flash-users
>
>
> _______________________________________________
> flash-users mailing list
> flash-users at flash.rochester.edu
>
> For list info, including unsubscribe:
> https://flash.rochester.edu/mailman/listinfo/flash-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20260423/8dc03977/attachment.htm>