[FLASH-USERS] ERROR in mpi_morton_bnd_prol

[Christopher Daley]

Hi Marco,

You should increase maxblocks because a value of maxblocks=4 is too
low.  You are using large blocks (32^3) and so memory usage prevents
you setting maxblocks too high, but I would suggest for this problem
you need a value of at least 10 for maxblocks.

The specific error you show is happening because data cannot be found
in a array of size maxblocks_alloc, where the default value of
maxblocks_alloc is maxblocks * 10.  Internally Paramesh has many
arrays of size maxblocks_alloc which hold e.g. various information
about the local view of the oct-tree.  A technique we have used in the
past when there is insufficient memory to make maxblocks much larger
and we need to avoid errors like you show is to make maxblocks_alloc
larger in amr_initialize.F90, e.g. maxblocks_alloc = maxblocks * 20.
You should also change maxblocks_tr to be the same size as
maxblocks_alloc.

Finally, if you don't need AMR then you should use the FLASH uniform
grid (you have a fully refined domain at level 5).  Memory usage will
be less and guard cell fills will be much faster.

Chris


On 01/18/2013 10:47 AM, Marco Mazzuoli wrote:
>     Dear Flash users,
>
> I am trying to run Flash on a bluegene-type supercomputer.
> The details of the present run are:
>
> #procs=1024 on #64 nodes (#16procs per node).
>
> The domain is rectangular.
> Block size = 32x32x32 computational points
> Max refinement level = 5
> The whole domain is refined at level = 5 such that 
> N°blocks=1+8+64+512+4096=4681
> Max_blocks per core = 4
>
> Do you know what the initialization error visualized in the standard 
> output and proposed in the following, could depend on?
>
> -----------------------------------------------------------------------------------------------------------------------
>  RuntimeParameters_read:  ignoring unknown parameter "igrav"...
>  RuntimeParameters_read:  ignoring unknown parameter 
> "mgrid_max_iter_change"...
>  RuntimeParameters_read:  ignoring unknown parameter 
> "mgrid_solve_max_iter"...
>  RuntimeParameters_read:  ignoring unknown parameter "mgrid_print_norm"...
>  RuntimeParameters_read:  ignoring unknown parameter "msgbuffer"...
>  RuntimeParameters_read:  ignoring unknown parameter "eint_switch"...
>  RuntimeParameters_read:  ignoring unknown parameter "order"...
>  RuntimeParameters_read:  ignoring unknown parameter "slopeLimiter"...
>  RuntimeParameters_read:  ignoring unknown parameter "LimitedSlopeBeta"...
>  RuntimeParameters_read:  ignoring unknown parameter "charLimiting"...
>  RuntimeParameters_read:  ignoring unknown parameter "use_avisc"...
>  RuntimeParameters_read:  ignoring unknown parameter "use_flattening"...
>  RuntimeParameters_read:  ignoring unknown parameter "use_steepening"...
>  RuntimeParameters_read:  ignoring unknown parameter "use_upwindTVD"...
>  RuntimeParameters_read:  ignoring unknown parameter "RiemannSolver"...
>  RuntimeParameters_read:  ignoring unknown parameter "entropy"...
>  RuntimeParameters_read:  ignoring unknown parameter "shockDetect"...
>  MaterialProperties initialized
>  Cosmology initialized
>  Source terms initialized
>  Cosmology initialized
>  Source terms initialized
>   iteration, no. not moved =  0 0
>  refined: total leaf blocks =  1
>  refined: total blocks =  1
>   starting MORTON ORDERING
>   tot_blocks after  1
>   max_blocks 2 1
>   min_blocks 2 0
>  Finished initialising block: 1
>  INFO: Grid_fillGuardCells is ignoring masking.
>   iteration, no. not moved =  0 0
>  refined: total leaf blocks =  8
>  refined: total blocks =  9
>   iteration, no. not moved =  0 7
>   iteration, no. not moved =  1 0
>  refined: total leaf blocks =  64
>  refined: total blocks =  73
>   iteration, no. not moved =  0 70
>   iteration, no. not moved =  1 7
>   iteration, no. not moved =  2 0
>  refined: total leaf blocks =  512
>  refined: total blocks =  585
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -3  not larger than lnblocks 1  processor no.  8 
> maxblocks_alloc  40
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -3  not larger than lnblocks 1  processor no. 496  
> maxblocks_alloc  40
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -3  not larger than lnblocks 1  processor no. 569  
> maxblocks_alloc  40
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -3  not larger than lnblocks 1  processor no. 172  
> maxblocks_alloc  40
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -12  not larger than lnblocks 1  processor no. 368  
> maxblocks_alloc  40
>  ERROR in mpi_morton_bnd_prol                      : guard block 
> starting index -12  not larger than lnblocks 1  processor no. 189  
> maxblocks_alloc  40
> ...
> ...
> ...
> Abort(1076419107) on node 442 (rank 442 in comm -2080374784): 
> application called MPI_Abort(comm=0x84000000, 1076419107) - process 442
> -----------------------------------------------------------------------------------------------------------------------
>
> Thank you in advance.
>
> Sincerely,
>
>     Marco Mazzuoli
>
>
> Ing. Marco Mazzuoli
> Dipartimento di Ingegneria
> delle Costruzioni, dell'Ambiente e
> del Territorio (DICAT)
> via Montallegro 1
> 16145 GENOVA-ITALY
> tel.  +39 010 353 2497
> cell. +39 338 7142904
> e-mail marco.mazzuoli at unige.it
>         marco.mazzuoli84 at gmail.com
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20130123/a2a6dede/attachment.htm>