[FLASH-USERS] FLASH Not Recognizing Number of Processors

[Matthew Gilmer]

Hi all, I'm fairly new to FLASH but have been successfully running
simulations since late 2014. Recently I've installed FLASH4.2 on a new
machine and everything is working fine except that I'm severely limited in
resolution because FLASH always thinks the simulation is being run on one
processor.

The variable in Flash.h, MAXBLOCKS, determines the maximum number of blocks
per processor. For a 3d simulation this is set by default to 200, but I've
found that I can increase it up to 11697 without complaint. But, no matter
how many processors I run on with mpirun, the logfile always says (at the
top) 'Number of MPI tasks:     1'.

Using htop, I can clearly see that, while the simulation is still in the
initial refinement phase, all requested processors are being used, but as
soon as FLASH tries to refine past 11697 blocks, I get:

 Too many blocks!  Increase MAXBLOCKS or use more processors.
application called MPI_Abort(comm=0x84000000, 0) - process 0

Increasing MAXBLOCKS past 11697 gives the make error:
/shyne/msgilmer/FLASH4.2/pisn3d/Driver_abortFlashC.c:80: relocation
truncated to fit: R_X86_64_PC32 against symbol `Driver_abortFlashC_myPE'
defined in COMMON section in Driver_abortFlashC.o
Driver_abortFlashC.o: In function `Driver_abortFlashC':
/shyne/msgilmer/FLASH4.2/pisn3d/Driver_abortFlashC.c:54: relocation
truncated to fit: R_X86_64_PC32 against symbol `Driver_abortFlashC_myPE'
defined in COMMON section in Driver_abortFlashC.o
Eos_getParameters.o: In function `eos_getparameters_':
/shyne/msgilmer/FLASH4.2/pisn3d/Eos_getParameters.F90:76: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__eos_helmdata_MOD_eos_forceconstantinput' defined in .bss section in
eos_helmData.o
Grid_GCPutScratch.o: In function `grid_gcputscratch_':
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:132: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrblklist' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:129: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrblkcnt' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:132: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrblklist' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:133: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrindlist' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:130: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrindcnt' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:134: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrblkoffsets' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:133: relocation
truncated to fit: R_X86_64_PC32 against symbol
`__gr_gcscratchdata_MOD_gr_gcctrindlist' defined in .bss section in
gr_GCScratchData.o
/shyne/msgilmer/FLASH4.2/pisn3d/Grid_GCPutScratch.F90:131: additional
relocation overflows omitted from the output
collect2: ld returned 1 exit status
make: *** [flash4] Error 1

The 'Too many blocks!' message is issued by mpi_amr_refine_derefine.F90
(part of the paramesh4 source code) when the following if statement is
satisfied:
if (tot_blocks > MAXBLOCKS * nprocs) then

nprocs is determined with: Call MPI_COMM_SIZE
(amr_mpi_meshComm,nprocs,ierr).

So I tried manually setting nprocs after this call, and then the error is:
Fatal error in MPI_Ssend: Invalid rank, error stack:
MPI_Ssend(152): MPI_Ssend(buf=0x7fffffff2b80, count=1, MPI_INTEGER, dest=1,
tag=1, comm=0x84000000) failed
MPI_Ssend(94).: Invalid rank has value 1 but must be nonnegative and less
than 1

I could keep trying to circumvent the problem but it seems like a time
sink, and I'd like to know why this is happening.

I'm curious whether anybody has encountered a similar problem, or if
anybody has any ideas about what might be wrong. Please ask if you want
more information.


Thanks,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20150518/5df6eaf1/attachment.htm>