[FLASH-USERS] Flash3 hanging in refinement

[Dave]

Hi,

I am finding that FLASH hangs in the refinement step. The final output 
in the log file is:

----
  [ 05-13-2012  03:56:30.023 ] step: n=2132 t=4.078276E+12 dt=6.783726E+08
  [ 05-13-2012  04:00:08.019 ] [mpi_amr_comm_setup]: 
buffer_dim_send=1519097, buffer_dim_recv=1485373
  [ 05-13-2012  04:00:38.738 ] [GRID amr_refine_derefine]: initiating 
refinement
----

and the final output in the standard dump is:

----
   iteration, no. not moved =            0      173352
   iteration, no. not moved =            1      169097
   iteration, no. not moved =            2      165742
   iteration, no. not moved =            3      162432
<etc etc>
   iteration, no. not moved =           98       19621
   iteration, no. not moved =           99       18939
   iteration, no. not moved =          100       18282
----

The code isn't crashing, and it appears that the processors are still 
running according to qstat etc, but it hasn't produced any output at all 
in over 24 hours.

It looks like it isn't redistributing blocks properly, but I would have 
assumed that would trigger an error message in mpi_amr_redist_blk.F90 :

----
       if (nm2_old.eq.nm2.and.nm2.ne.0.and.nit>=100) then
          if (mype.eq.0) then
           print *,' ERROR: could not move all blocks in amr_redist_blk'
           print *,' Try increasing maxblocks or use more processors'
           print *,' nm2_old, nm2 = ',nm2_old,nm2
           print *,' ABORTING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
          end if
          call MPI_ABORT(MPI_COMM_WORLD,errorcode,ierr)
       end if
----

Any clues or suggestions?

Cheers,
David Williamson

PhD Candidate
St. Mary's University
Halifax, NS