[FLASH-BUGS] MPI bug on SGI

[Markus Gross]

Hi!

I belive I have just encountered a bug in at least one FLASH routine.

Our System: 

SGI origin IRIX64 6.5 07091542 IP35
MPI 3.2.0.7 (MPT 1.4) -mpi Version 1266208220
MIPSpro Compilers: Version 7.4

Problem:

3-D hydro starting with 5x5x3 blocks, maxref we get a:

*** MPI has run out of request entries.
*** The current allocation level is:
***     MPI_REQUEST_MAX IOT Trap
MPI: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
MPI: aborting job

when:

 [04-17-2003  18:48.39] <<< refined: tot_blocks  [04-17-2003  18:48.59] *** plot file:  gas_pre_hdf_plt_cnt_0009
 [04-17-2003  18:49.08] *** wrote to gas_pre_hdf_plt_cnt_0009
 [04-17-2003  18:49.41] step      91  t [04-17-2003  18:50.12] step      92  t
MPI_REQUEST_MAX decreased using the enivronment variable.


I tracked that down to at least one non-blocking send to a blocking receive 
where the send reqest was not freed. This happens at least is:

line 241 AMR_redist_blk.F90 FLASH 2.2 & 2.1


according to the MPI standard that should be fixed by having a MPI_WAIT 
following the non-blocking send immediately. 

I attached a test program to this email which illustrates this point (at 
least on our system). Version as supplied should fails, if you uncomment the 
MPI_wait it should run until forever.

Please let me know what you think about this and if I am right, if there are 
more if these problems in the code (I presume you would know from the top of 
your head). I try to get through it tonight, but Flash is a little bit 
bigger, so help would be appreciated.

Regards,

Markus.



-- 
_______________________________________________________________

Markus Gross AMIMechE BEng (Hons.) Mechanical Engineering

Heriot Watt University Edinburgh
Department of Mechanical and Chemical Engineering

Associate member of the Institution of Mechanical Engineers
Member of the SPIE
_______________________________________________________________

further contact:

Phone   : +44 (0) 131 449 5111 etx. 4737

UNiX talk: talk markus at lasersim.mce.hw.ac.uk

_______________________________________________________________

"Plans are a place to begin," Grove said. "They rarely deliver
you to where you expect. Make your plans knowing you are going
to throw them away."

_______________________________________________________________


-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

mQGiBDt30nARBACUL2XadcLU9YS+3ndnWUUpnrhAD3Ab/TqJP1Obtz7H356+vigu
2m3exw2P9fky08SBdTbjOCV5OTmVcRW5F0SNMTIl8gXtGdIcQHCcZxi1GcRJtTBj
npfRCmZojHr8YgsKDdJ9xupNm+UYWcg1BpFX7xe65kXM4oVwVyuCcUT+VwCg6/Hd
6oVoRHm1eq1NdMbCfsc9x78D/2uGQSMJBKewQ/uSPpdEg+bkAQo5Y7MooEeYpLk9
Iroo4rqxp12k0+bWHWVMdfywI80tivorTIgBTKpCtjLaiKen9E9zECo+2OR4+XRQ
O6cBKXOm+V2lJOpFYOh500tJHPggshD21jksNPKJr9CM1byeZFw38fFJez+DyShZ
Gmc6A/9L21jqbVjxvAPwaqe09GEmpT1EYOWtcPYSh2cuo1v6XsGFXfi8tnRSizhH
xsyIfuOUuj0NKRIcXH60awrvABSa45fvQUefPw0hxSwdCnfc0yEqcD4x/03xTZJS
1hUK3T4oTlrXNugaCb59ezciEIh2y/d8pW1DR3RMcgXUSz39tLQhTWFya3VzIEdy
b3NzIDxtLnMuZ3Jvc3NAaHcuYWMudWs+iFcEExECABcFAjt30nAFCwcKAwQDFQMC
AxYCAQIXgAAKCRBTnmsr4a/0+e3tAJoDk08MfeqB83FXP6g8QlDyTlotSwCgiKwS
PPkVAbopX8mahpkrphEaunW5Ag0EO3fSghAIAOeLTMjdFkl/xBvjgD2N0n4DCK4N
EVfaXmkJK5dbTM3K6ZsJjILZ/TfC6R8oxYPsYKwkwLJJEe4E/h1TFg7ogDYbViW8
DihOyeN8zKSymLTbbikEJX8aNeCdBRuy87X0pBazDq6//+oTIatS2EsvMyKk0tEZ
Mh0vTmUJqoeorQeAKH66lPsH2A8QI6u0zv17t4S2aLNBXIbKgOJwsglZ1IwjwowR
KEGsLsHInNkP4bnlBjd51RMiJT979QMPmNgLGEId9Vi0oijtF/v6J91GtH++G6WE
HM08DsP/BKjRgO9ivWP7lOHSDCJpGYti5K6g38JAc1H2EoHGLD7eRgI/kR8AAwUH
/j6NVGLxnFGDHGcBXEb2Y4Wwhf/eUeE8Wit0qqRoNypBuzvWl96GXSPj/saQA/Ri
XcFWR1G3E4V/zdxYnU/xvvGADj9jKHpZjAs73fJJxLc85gzHLcx00tHcQVA07Ihr
hao8WY+BtKhkZorYEJEvSDJ9xufjL/o1t40eCsHQ3b9Jh/1lqQ2KT1lhDXAaDo1P
pE38E2uImC0umIW/UtuKJ1k52nXx2qulfcTjAHd35XidwGreH3aHy3fV+x/+wW1T
CQBHy27dG7JL3ESzPiKW0w2miWjtd4k5jqJTfgKI4wFUhAShxTjp8M051ArzopRJ
iX+s7RrHuEXHjh45V1zU5BWIRgQYEQIABgUCO3fSggAKCRBTnmsr4a/0+U/rAJ9o
lOhPGltVnRncYJPzyIk00H0JDQCfWrlnYDh5cRQmhnHilnrVO5Ykr24=i3eC
-----END PGP PUBLIC KEY BLOCK-----

-------------- next part --------------
include 'mpif.h'


INTEGER IERR,MYRANK,lop,test,request
character(len)  mpireqmax

integer status(MPI_STATUS_SIZE)

CALL getenv('MPI_REQUEST_MAX',mpireqmax)

write(6,*) 'MPI_REQUEST_MAX=',mpireqmax


CALL sleep(2)

CALL mpi_init(ierr)


CALL MPI_COMM_RANK(MPI_COMM_WORLD,MYRANK,IERR)

IF (MYRANK.EQ.0) THEN

  lop 
  DO WHILE(.TRUE.)

    lop 
    CALL MPI_Isend(test,1,MPI_INTEGER,1,0,MPI_COMM_WORLD,request,IERR)
!    CALL MPI_WAIT(request,status,ierr)
    
    write(6,fmt='(I6)',ADVANCE='NO') lop

    if(mod(lop,20).eq.0) write(6,*)

  END DO

END IF

IF (MYRANK.EQ.1) THEN

  lop 
  DO WHILE(.TRUE.)

    lop 
    CALL MPI_RECV(test,1,MPI_INTEGER,0,0,MPI_COMM_WORLD,status,IERR)

  END DO

END IF



CALL MPI_BARRIER(MPI_COMM_WORLD,ierr)
write(6,*) 'past barrier'
call flush(6)

CALL MPI_FINALIZE(MPI_COMM_WORLD,ierr)


END