[FLASH-USERS] Fatal error in PMPI_Waitall: See the MPI_ERROR field in MPI_Status for the error code

[Klaus Weide]

On Fri, 6 Dec 2019, guido wrote:

> Hello Flash users,
> 
> I’m having trouble running a simulation with FLASH 4.5. It compiled FLASH
> 4.5 using:
> 
> - hdf5-1.8.20 with parallel mode enabled
> 
> - mpich-3.2.1
> 
> - gcc-7.3.0
> 
> The simulations run fine till I increase the maximum level of refinement and
> the number of nodes used. For example, for simulation with lmax=9, it runs
> slow but safe with 8 nodes. However, when I increase the number of nodes to
> 10, the simulation crash trying to write the Coefficients in the Taylor
> expansion into file.

Guido,

>From your logfile it is obvious that you are not just using the solvers of 
FLASH 4.5, but at least one Hydro solver implementation that has never 
been part of a FLASH code release:

  physics/Hydro/HydroMain/split/Bouchut/Bouchut3

You are also using using the 'BHTree' Poisson Gravity solver, which has 
been released with FLASH, but I think not many members of this list have 
experience with using it.

It is not clear from your logfile that the error happens while "trying to 
write the Coefficients in the Taylor expansion into file." But if it is 
indeed related to those Taylor coefficients, then it is part of that 
Gravity solver.  Perhaps the Ewald-expansion part of that code only
works properly for some numbers of procs (e.g., powers of 2)?

Some further ideas:

You may want to try what happens if you use a combination of 
"outflow"/"isolated" boundary conditions. (I think the Taylor coefficient 
would not be used at all in that case.)

You may want to try a different Gravity solver.

You may want to get help from someone who knows the BHTree solver well.
It may help if you put something "BHTree solver" in the subject lines of 
related messages.

Klaus