[FLASH-USERS] FLASH increasing AMR through lrefine_max and running into MAXBLOCKS issue

[Leland Ellison]

Supplementing the ideas around increasing MAXBLOCKS:

If you're running a convergence study, I would increase nblockx and nblocky
instead of lrefinemax. lrefinemax will (hopefully) add more resolution near
interesting features, which might be a computationally efficient way to
increase resolution near the largest discretization errors, but to get
increased resolution throughout your problem and ensure convergence
everywhere you should just increase the number of zones on the lowest level
by increasing the number of blocks on the lowest level via nblockx and
nblocky.

Additionally, you probably want to add more processors as you're adding
more blocks. For good performance, you probably want a similar number of
zones per processor throughout your convergence study. So I would e.g.
double nblockx, double nblocky, and quadruple the number of processors and
you shouldn't run into issues with number of blocks per processor this way.

Best,
Lee Ellison


On Tue, Aug 1, 2023 at 5:15 PM Kaldawi, Thomas <Tkal at lle.rochester.edu>
wrote:

> Not sure if it would fix the other issues, but just keep increasing the
> max blocks in setup until you cannot run the simulation anymore. I run it
> on a supercomputer, but I usually cap it around 10,000 blocks.
>
> Sincerely,
>
> Thomas Kaldawi
>
> On Aug 1, 2023, at 18:39, Sharma, Divyanshu Raj <sharma22 at llnl.gov> wrote:
>
> 
>
> Hi Flash Users,
>
>
>
> I am trying to increase lrefinemax and see where the solution converges on
> the LaserSlab example. When doing this and taking lrefinemax from 9 to 16 I
> ran into a bunch of inconsistent errors. The main errors are shown below.
> It is just weird because I know that MAXBLOCKS is set to 1000 by default
> and I can increase it but would that take care of the other errors. I also
> was wondering if there was any intuition on how much should MAXBLOCKS be
> changed by and if that would have an impact on the runtime of the
> simulations or is it just once in allocating the memory.
>
>
>
> Lrefine_max = 9:
>
> ERROR: could not move all blocks in amr_redist_blk
>
>   Try increasing maxblocks or use more processors
>
>
>
> Lrefine_max = 10:
>
> Time limit exceeded (weird cause it didn’t burnout like 9)
>
>
>
> Lrefine_max = 11:
>
> Same as 10
>
>
>
> Lrefine_max = 12:
>
> Same as 10
>
>
>
> Lrefine_max = 13:
>
> [hy_uhd_unsplitUpdateMultiTemp] Negative 3T internal energy, CHECK LOG
>
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 46
>
>
>
> Lrefine_max = 14:
>
> Too many blocks!  Increase MAXBLOCKS or use more processors.
>
> application called MPI_Abort(comm=0x84000006, 32767) - process 140
>
>
>
> Lrefine_max = 15:
>
> [hy_uhd_unsplitUpdateMultiTemp] Negative 3T internal energy, CHECK LOG
>
>
>
> Lrefine_max = 16:
>
> Too many blocks!  Increase MAXBLOCKS or use more processors.
>
> application called MPI_Abort(comm=0x84000006, 32767) - process 117
> _______________________________________________
> flash-users mailing list
> flash-users at flash.rochester.edu
>
> For list info, including unsubscribe:
> https://flash.rochester.edu/mailman/listinfo/flash-users
>
> _______________________________________________
> flash-users mailing list
> flash-users at flash.rochester.edu
>
> For list info, including unsubscribe:
> https://flash.rochester.edu/mailman/listinfo/flash-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20230801/15462a9f/attachment-0001.htm>