[FLASH-BUGS] hangups on IBM regatta

[Sebastian Heinz]

Dear FLASH team,


I am having problems getting FLASH to run on the new IBM regatta system at
the IPP in Garching, Germany. A description can be found at

  http://www.rzg.mpg.de/computing/IBM_P/

Here's a brief description of the problem:

Currently I am running a 2D hydro simulation in cylindrical coordinates at
relatively high resolution (but the same problem occurs also in cartesian
3D simulations). 

After running for a while, the system simply hangs up without producing
any errors. This only happens when I run the process on more than two
processors, while it runs fine on one or two processors (i.e., this is not
a maxblocks problem: I compiled using maxblocks=12000 and at the
corresponding timestep the simulation has 4478 blocks). After some
back-tracking, I found that the system hangs in the routine

  amr_redist_blk.F90

at the MPI_ALLREDUCE call in line 102. Some processors make it past this
call, while the others don't.

I compiled using the flags

  ABI  = -q32 -qmaxmem=-1
  OPT1 = -O3 -qstrict -bmaxdata:0x80000000
  OPT2 = -qfloat=nomaf -bloadmap:makelog
  OPT3 = -qcache=auto -qtune=auto -qarch=auto -qthreaded -qnosave
  OPTS = $(OPT1) $(OPT2) $(OPT3)

Any ideas what this could be? (From what I read on the FLASH website, I
don't seem to be the only one with problems on IBM systems).

I tried to compile the code with 64 bits, but I couldn't get that to work
- any input on what compiler options to use for 64 bits (and which
libraries I need to link) would also be highly welcome.


Thanks for your help,

Best regards,

Sebastian


P.S.: I also found a small bug in MarkRefLib.F90, which I will send you in
a separate E-mail