[FLASH-USERS] Fatal error in PMPI_Waitall: See the MPI_ERROR field in MPI_Status for the error code
guido
g.granda at irya.unam.mx
Fri Dec 6 18:45:48 EST 2019
Hello Flash users,
I’m having trouble running a simulation with FLASH 4.5. It compiled
FLASH 4.5 using:
- hdf5-1.8.20 with parallel mode enabled
- mpich-3.2.1
- gcc-7.3.0
The simulations run fine till I increase the maximum level of refinement
and the number of nodes used. For example, for simulation with lmax=9,
it runs slow but safe with 8 nodes. However, when I increase the number
of nodes to 10, the simulation crash trying to write the Coefficients in
the Taylor expansion into file.
The log file is the following:
FLASH log file: 12-05-2019 12:01:48.640 Run number: 1
==============================================================================
Number of MPI tasks: 1
MPI version: 3
MPI subversion: 1
Dimensionality: 3
Max Number of Blocks/Proc: 3000
Number x zones: 8
Number y zones: 8
Number z zones: 8
Setup stamp: Wed Sep 18 16:44:51 2019
Build stamp: Wed Sep 18 16:45:20 2019
System info: Linux mouruka.crya.privado 2.6.32-504.16.2.el6.x86_64 #1
SMP Wed Apr 22 06:48:29
Version: FLASH 4.5_release
Build directory: /home/guido/FLASH4.5/obj_mc
Setup syntax: /home/guido/FLASH4.5/bin/setup.py magnetoHd/MC_evolution
-auto -3d -maxblocks=3000 -objdir=obj_mc -site=irya.guido
--without-unit=physics/Hydro/HydroMain/split/MHD_8Wave
f compiler flags:
/home/guido/libraries/compiled_with_gcc-7.3.0/mpich-3.2.1/bin/mpif90 -g
-c -O0 -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none
-Wuninitialized -g -c -fdefault-real-8 -fdefault-double-8
-ffree-line-length-none -Wuninitialized -DMAXBLOCKS=3000 -DNXB=8 -DNYB=8
-DNZB=8 -DN_DIM=3
c compiler flags:
/home/guido/libraries/compiled_with_gcc-7.3.0/mpich-3.2.1/bin/mpicc
-I/home/guido/libraries/compiled_with_gcc-7.3.0/hdf5-1.8.20/include
-DH5_USE_16_API -O0 -c -DMAXBLOCKS=3000 -DNXB=8 -DNYB=8 -DNZB=8 -DN_DIM=3
==============================================================================
Comment: MC evolution
==============================================================================
FLASH Units used:
Driver/DriverMain/Split
Driver/localAPI
Grid/GridBoundaryConditions
Grid/GridMain/paramesh/interpolation/Paramesh4/prolong
Grid/GridMain/paramesh/interpolation/prolong
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/headers
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/mpi_source
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/source
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/source
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/PM4_package/utilities/multigrid
Grid/GridMain/paramesh/paramesh4/Paramesh4dev/flash_avoid_orrery
Grid/GridSolvers/BHTree/Wunsch
Grid/localAPI
IO/IOMain/hdf5/serial/PM
IO/localAPI
PhysicalConstants/PhysicalConstantsMain
RuntimeParameters/RuntimeParametersMain
Simulation/SimulationMain/magnetoHD/MC_evolution
flashUtilities/contiguousConversion
flashUtilities/general
flashUtilities/interpolation/oneDim
flashUtilities/nameValueLL
flashUtilities/sorting/quicksort
flashUtilities/system/memoryUsage/legacy
monitors/Logfile/LogfileMain
monitors/Timers/TimersMain/MPINative
physics/Eos/EosMain/Gamma
physics/Eos/localAPI
physics/Gravity/GravityMain/Poisson/BHTree
physics/Hydro/HydroMain/split/Bouchut/Bouchut3
physics/Hydro/localAPI
physics/sourceTerms/Polytrope/PolytropeMain
==============================================================================
RuntimeParameters:
==============================================================================
bndpriorityone = 1
bndprioritythree = 3
bndprioritytwo = 2
checkpointfileintervalstep = 1000000 [CHANGED]
checkpointfilenumber = 0
dr_abortpause = 2
dr_dtminbelowaction = 1
dr_numposdefvars = 4
drift_break_inst = 0
drift_trunc_mantissa = 2
drift_verbose_inst = 0
eos_entrelescalechoice = 6
eos_loglevel = 700
fileformatversion = 9
forcedplotfilenumber = 0
gr_bhtwmaxqueuesize = 10000
gr_lrefmaxtimevalue_1 = -1
gr_lrefmaxtimevalue_10 = -1
gr_lrefmaxtimevalue_11 = -1
gr_lrefmaxtimevalue_12 = -1
gr_lrefmaxtimevalue_13 = -1
gr_lrefmaxtimevalue_14 = -1
gr_lrefmaxtimevalue_15 = -1
gr_lrefmaxtimevalue_16 = -1
gr_lrefmaxtimevalue_17 = -1
gr_lrefmaxtimevalue_18 = -1
gr_lrefmaxtimevalue_19 = -1
gr_lrefmaxtimevalue_2 = -1
gr_lrefmaxtimevalue_20 = -1
gr_lrefmaxtimevalue_3 = -1
gr_lrefmaxtimevalue_4 = -1
gr_lrefmaxtimevalue_5 = -1
gr_lrefmaxtimevalue_6 = -1
gr_lrefmaxtimevalue_7 = -1
gr_lrefmaxtimevalue_8 = -1
gr_lrefmaxtimevalue_9 = -1
gr_pmrpdivergencefree = 1
gr_pmrpifaceoff = 0
gr_pmrpl2p5d = 0
gr_pmrpmaxblocks = -1
gr_pmrpmflags = 1
gr_pmrpnboundaries = 6
gr_pmrpndim = 3
gr_pmrpnedgevar1 = -1
gr_pmrpnfacevar = -1
gr_pmrpnfielddivf = -1
gr_pmrpnfluxvar = -1
gr_pmrpnguard = -1
gr_pmrpnguardwork = -1
gr_pmrpnvar = -1
gr_pmrpnvarwork = 1
gr_pmrpnvarcorn = 0
gr_pmrpnvaredge = 0
gr_pmrpnxb = -1
gr_pmrpnyb = -1
gr_pmrpnzb = -1
gr_restrictallmethod = 3
gr_sanitizedatamode = 3 [CHANGED]
gr_sanitizeverbosity = 5
grv_bhewaldfieldnxv42 = 32
grv_bhewaldfieldnyv42 = 32
grv_bhewaldfieldnzv42 = 32
grv_bhewaldnper = 32
grv_bhewaldnrefv42 = -1
grv_bhewaldseriesn = 10
grv_bhmpdegree = 2
iprocs = 1
interpol_order = 2
irenorm = 0
jprocs = 1
kprocs = 1
lrefine_del = 0
lrefine_max = 9 [CHANGED]
lrefine_min = 3 [CHANGED]
lrefine_min_init = 1
max_particles_per_blk = 100
memory_stat_freq = 100000
meshcopycount = 1
min_particles_per_blk = 1
nbegin = 1
nblockx = 1
nblocky = 1
nblockz = 1
nend = 1000000 [CHANGED]
nrefs = 2
nsteptotalsts = 5
outputsplitnum = 1
plotfileintervalstep = 1000000 [CHANGED]
plotfilenumber = 0
refine_var_count = 4
rolling_checkpoint = 10000
sim_refine = 9 [CHANGED]
sweeporder = 123
wr_integrals_freq = 1
cfl = 0.800E+00
checkpointfileintervaltime = 0.316E+14 [CHANGED]
checkpointfileintervalz = 0.180+309
derefine_cutoff_1 = 0.300E+00 [CHANGED]
derefine_cutoff_2 = 0.200E+00
derefine_cutoff_3 = 0.200E+00
derefine_cutoff_4 = 0.200E+00
dr_dtmincontinue = 0.000E+00
dr_posdefdtfactor = 0.100E+01
dr_tstepslowstartfactor = 0.100E+00
dtinit = 0.316E+11 [CHANGED]
dtmax = 0.316E+13 [CHANGED]
dtmin = 0.316E+10 [CHANGED]
eintswitch = 0.100E-03 [CHANGED]
eos_singlespeciesa = 0.100E+01
eos_singlespeciesz = 0.100E+01
gamma = 0.167E+01
gr_bhtreelimangle = 0.500E+00
gr_bhtreemaxcellmass = 0.100+100
gr_bhtreemincellmass = 0.100E-98
gr_bhtreesafebox = 0.120E+01
gr_lrefinemaxredlogbase = 0.100E+02
gr_lrefinemaxredradiusfact = 0.000E+00
gr_lrefinemaxredtref = 0.000E+00
gr_lrefinemaxredtimescale = 0.100E+01
gr_lrefmaxtime_1 = -0.100E+01
gr_lrefmaxtime_10 = -0.100E+01
gr_lrefmaxtime_11 = -0.100E+01
gr_lrefmaxtime_12 = -0.100E+01
gr_lrefmaxtime_13 = -0.100E+01
gr_lrefmaxtime_14 = -0.100E+01
gr_lrefmaxtime_15 = -0.100E+01
gr_lrefmaxtime_16 = -0.100E+01
gr_lrefmaxtime_17 = -0.100E+01
gr_lrefmaxtime_18 = -0.100E+01
gr_lrefmaxtime_19 = -0.100E+01
gr_lrefmaxtime_2 = -0.100E+01
gr_lrefmaxtime_20 = -0.100E+01
gr_lrefmaxtime_3 = -0.100E+01
gr_lrefmaxtime_4 = -0.100E+01
gr_lrefmaxtime_5 = -0.100E+01
gr_lrefmaxtime_6 = -0.100E+01
gr_lrefmaxtime_7 = -0.100E+01
gr_lrefmaxtime_8 = -0.100E+01
gr_lrefmaxtime_9 = -0.100E+01
grv_bhaccerr = 0.100E+00
grv_bhextrnpotcenterx = 0.000E+00
grv_bhextrnpotcentery = 0.000E+00
grv_bhextrnpotcenterz = 0.000E+00
grv_bhnewton = -0.100E+01
hall_parameter = 0.000E+00
hyperresistivity = 0.000E+00
nusts = 0.100E+00
plotfileintervaltime = 0.316E+13 [CHANGED]
plotfileintervalz = 0.180+309
point_mass = 0.000E+00
point_mass_rsoft = 0.000E+00
polytropedens1 = 0.212E-24 [CHANGED]
polytropedens2 = 0.100+100
polytropedens3 = 0.100+100
polytropedens4 = 0.100+100
polytropedens5 = 0.100+100
polytropegamma1 = 0.100E+01
polytropegamma2 = 0.100E+01
polytropegamma3 = 0.100E+01
polytropegamma4 = 0.100E+01
polytropegamma5 = 0.100E+01
polytropekonst = 0.400E+09 [CHANGED]
refine_cutoff_1 = 0.700E+00 [CHANGED]
refine_cutoff_2 = 0.800E+00
refine_cutoff_3 = 0.800E+00
refine_cutoff_4 = 0.800E+00
refine_filter_1 = 0.100E-01
refine_filter_2 = 0.100E-01
refine_filter_3 = 0.100E-01
refine_filter_4 = 0.100E-01
rss_limit = -0.100E+01
sim_bx = 0.710E-05 [CHANGED]
sim_by = 0.000E+00
sim_bz = 0.000E+00
sim_cs = 0.200E+05 [CHANGED]
sim_delta = 0.463E+19 [CHANGED]
sim_mu_mol = 0.127E+01
sim_pert = 0.150E+01 [CHANGED]
sim_rho = 0.212E-21 [CHANGED]
sim_sigma = 0.250E+00 [CHANGED]
sim_vx = 0.000E+00
sim_vy = 0.000E+00
sim_vz = 0.000E+00
sim_x_cent = 0.000E+00
sim_y_cent = 0.000E+00
sim_z_cent = 0.000E+00
small = 0.100E-39 [CHANGED]
smalle = 0.100E-09
smallp = 0.100E-21 [CHANGED]
smallt = 0.100E+01 [CHANGED]
smallu = 0.100E-39 [CHANGED]
smallx = 0.100E-09
smlrho = 0.100E-39 [CHANGED]
tinitial = 0.000E+00
tmax = 0.252E+15 [CHANGED]
tstep_change_factor = 0.200E+01
wall_clock_checkpoint = 0.432E+05
wall_clock_time_limit = 0.605E+06
x_refine_center = 0.000E+00
xmax = 0.154E+20 [CHANGED]
xmin = -0.154E+20 [CHANGED]
y_refine_center = 0.000E+00
ymax = 0.154E+20 [CHANGED]
ymin = -0.154E+20 [CHANGED]
zfinal = 0.000E+00
zinitial = -0.100E+01
z_refine_center = 0.000E+00
zmax = 0.154E+20 [CHANGED]
zmin = -0.154E+20 [CHANGED]
unitsystem = CGS [CHANGED]
basenm = mc_evolution_ [CHANGED]
dr_posdefvar_1 = none
dr_posdefvar_2 = none
dr_posdefvar_3 = none
dr_posdefvar_4 = none
eosmode = dens_ie
eosmodeinit = dens_ie
geometry = cartesian
gr_pmrpoutputdir = ./
grav_boundary_type = periodic [CHANGED]
grav_boundary_type_x = periodic [CHANGED]
grav_boundary_type_y = periodic [CHANGED]
grav_boundary_type_z = periodic [CHANGED]
grv_bhewaldfname = ewald_coeffs
grv_bhewaldfnameaccv42 = ewald_field_acc
grv_bhewaldfnamepotv42 = ewald_field_pot
grv_bhextrnpotfile = external_potential.dat
grv_bhextrnpottype = planez
grv_bhmac = ApproxPartialErr
log_file = mc_evolution.log [CHANGED]
output_directory =
pc_unitsbase = CGS
plot_grid_var_1 = none
plot_grid_var_10 = none
plot_grid_var_11 = none
plot_grid_var_12 = none
plot_grid_var_2 = none
plot_grid_var_3 = none
plot_grid_var_4 = none
plot_grid_var_5 = none
plot_grid_var_6 = none
plot_grid_var_7 = none
plot_grid_var_8 = none
plot_grid_var_9 = none
plot_var_1 = dens [CHANGED]
plot_var_10 = magz [CHANGED]
plot_var_11 = none
plot_var_12 = none
plot_var_13 = none
plot_var_14 = none
plot_var_15 = none
plot_var_16 = none
plot_var_17 = none
plot_var_18 = none
plot_var_2 = pres [CHANGED]
plot_var_3 = none
plot_var_4 = eint [CHANGED]
plot_var_5 = velx [CHANGED]
plot_var_6 = vely [CHANGED]
plot_var_7 = velz [CHANGED]
plot_var_8 = magx [CHANGED]
plot_var_9 = magy [CHANGED]
prof_file = profile.dat
refine_var_1 = dens [CHANGED]
refine_var_2 = none
refine_var_3 = none
refine_var_4 = none
refine_var_thresh = dens [CHANGED]
run_comment = MC evolution [CHANGED]
run_number = 1
stats_file = mc.dat [CHANGED]
xl_boundary_type = periodic
xr_boundary_type = periodic
yl_boundary_type = periodic
yr_boundary_type = periodic
zl_boundary_type = periodic
zr_boundary_type = periodic
allowdtstsdominate = F
alwayscomputeuservars = T
alwaysrestrictcheckpoint = T
chkguardcellsinput = F
chkguardcellsoutput = F
converttoconsvdformeshcalls = F
converttoconsvdinmeshinterp = T
corners = F
dr_printtsteploc = T
dr_shortenlaststepbeforetmax = F
dr_useposdefcomputedt = F
drift_tuples = F
eachprocwritesownabortlog = F
eachprocwritessummary = F
earlyblockdistadjustment = T
enablemaskedgcfill = F
flux_correct = T
geometryoverride = F
gr_bcenableapplymixedgds = T
gr_bhphysmaccomm = F
gr_bhphysmactw = F
gr_bhuseunifiedtw = F [CHANGED]
gr_lrefinemaxbytime = F
gr_lrefinemaxreddobylogr = F
gr_lrefinemaxreddobytime = F
gr_pmrpadvancealllevels = F
gr_pmrpamrerrorchecking = F
gr_pmrpcartesianpm = F
gr_pmrpconserve = F
gr_pmrpconsvfluxdensities = T
gr_pmrpconsvfluxes = F
gr_pmrpcurvilinear = F
gr_pmrpcurvilinearconserve = F
gr_pmrpcylindricalpm = F
gr_pmrpdiagonals = T
gr_pmrpedgevalue = T
gr_pmrpedgevalueinteg = F
gr_pmrpemptycells = F
gr_pmrpforceconsistency = T
gr_pmrplsingularline = F
gr_pmrpnopermanentguardcells = F
gr_pmrppolarpm = F
gr_pmrppredcorr = F
gr_pmrpsphericalpm = F
gr_pmrptimingmpi = F
gr_pmrptimingmpix = F
gr_pmrpvardt = F
grav_temporal_extrp = F
grav_unjunkpden = T
grv_bhewaldalwaysgenerate = T
grv_bhlinearinterpolonlyv42 = T
grv_bhuserelaccerr = F
grv_useexternalpotential = F
grv_usepoissonpotential = T
ignoreforcedplot = F
io_writemscalarintegrals = F
killdivb = T [CHANGED]
plotfilegridquantitydp = F
plotfilemetadatadp = F
reducegcellfills = F
refine_on_particle_count = F
restart = F
summaryoutputonly = F
threadblocklistbuild = F
threaddriverblocklist = F
threaddriverwithinblock = F
threadeoswithinblock = F
threadhydroblocklist = F
threadhydrowithinblock = F
threadraytracebuild = F
threadwithinblockbuild = F
typematchedxfer = T
unbiased_geometry = F
updategravity = T
updatehydrofluxes = T
useburn = F
usecollectivehdf5 = T
useconductivity = F
usecool = F
usecosmology = F
usedeleptonize = F
usediffuse = F
usediffusecomputedtspecies = F
usediffusecomputedttherm = F
usediffusecomputedtvisc = F
usediffusecomputedtmagnetic = F
useenergydeposition = F
useflame = F
usegravity = T
useheat = F
useheatexchange = F
usehydro = T
useincompns = F
useionize = F
uselegacylabels = T
usemagneticresistivity = F
usemassdiffusivity = F
useopacity = F
useparticles = F
useplasmastate = F
usepolytrope = T [CHANGED]
useprimordialchemistry = F
useprotonemission = F
useprotonimaging = F
useradtrans = F
useraytrace = F
usests = F
usestsfordiffusion = F
usestir = F
usethomsonscattering = F
usetreeray = F
useturb = T
useviscosity = F
usexrayimaging = F
use_cma_advection = F
use_cma_flattening = F
use_flash_surr_blks_fill = T
use_reduced_orrery = T
use_steepening = T
writestatsummary = T
==============================================================================
WARNING: Ignored Parameters :
These parameters were found in the flash.par file, but they were
not declared in any Config file for the simulation!
energyFix
==============================================================================
Known units of measurement:
Unit CGS Value Base Unit
1 cm 1.0000 cm
2 s 1.0000 s
3 g 1.0000 g
4 K 1.0000 K
5 esu 1.0000 esu
6 mol 1.0000 mol
7 m 100.00 cm
8 km 1.00000E+05 cm
9 pc 3.08568E+18 cm
10 kpc 3.08568E+21 cm
11 Mpc 3.08568E+24 cm
12 Gpc 3.08568E+27 cm
13 Rsun 6.96000E+10 cm
14 AU 1.49598E+13 cm
15 yr 3.15569E+07 s
16 Myr 3.15569E+13 s
17 Gyr 3.15569E+16 s
18 kg 1000.0 g
19 Msun 1.98892E+33 g
20 amu 1.66054E-24 g
21 eV 11605. K
22 C 2.99792E+09 esu
23 LFLY 3.08568E+24 cm
24 TFLY 2.05759E+17 s
25 MFLY 9.88470E+45 g
26 clLength 3.08568E+24 cm
27 clTime 3.15569E+16 s
28 clMass 1.98892E+48 g
29 clTemp 1.16045E+07 K
-----------End of Units--------------------
Known physical constants:
Constant Name Constant Value cm s g K esu mol
1 Newton 6.67408E-08 3.0 -2.0 -1.0 0.0 0.0 0.0
2 speed of light 2.99792E+10 1.0 -1.0 0.0 0.0 0.0 0.0
3 Planck 6.62607E-27 2.0 -1.0 1.0 0.0 0.0 0.0
4 electron charge 4.80320E-10 0.0 0.0 0.0 0.0 1.0 0.0
5 electron mass 9.10938E-28 0.0 0.0 1.0 0.0 0.0 0.0
6 proton mass 1.67262E-24 0.0 0.0 1.0 0.0 0.0 0.0
7 fine-structure 7.29735E-03 0.0 0.0 0.0 0.0 0.0 0.0
8 Avogadro 6.02214E+23 0.0 0.0 0.0 0.0 0.0 -1.0
9 Boltzmann 1.38065E-16 2.0 -2.0 1.0 -1.0 0.0 0.0
10 ideal gas constant 8.31446E+07 2.0 -2.0 1.0 -1.0 0.0 -1.0
11 Wien 0.28978 1.0 0.0 0.0 1.0 0.0 0.0
12 Stefan-Boltzmann 5.67037E-05 0.0 -3.0 1.0 -4.0 0.0 0.0
13 Radiation Constant 7.56572E-15 -1.0 -2.0 1.0 -4.0 0.0 0.0
14 pi 3.1416 0.0 0.0 0.0 0.0 0.0 0.0
15 e 2.7183 0.0 0.0 0.0 0.0 0.0 0.0
16 Euler 0.57722 0.0 0.0 0.0 0.0 0.0 0.0
==============================================================================
Multifluid database: not configured in
==============================================================================
[ 12-05-2019 12:02:26.975 ] [gr_initGeometry] checking BCs for idir: 1
[ 12-05-2019 12:02:26.976 ] [gr_initGeometry] checking BCs for idir: 2
[ 12-05-2019 12:02:26.976 ] [gr_initGeometry] checking BCs for idir: 3
[ 12-05-2019 12:02:27.074 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:27.302 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 9
[GRID amr_refine_derefine] min blks 0 max blks 1 tot blks 9
[GRID amr_refine_derefine] min leaf blks 0 max leaf blks 1 tot leaf blks 8
[ 12-05-2019 12:02:27.543 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:27.968 ] [GRID gr_expandDomain]: iteration=1, create
level=3
INFO: Grid_fillGuardCells is ignoring masking.
[ 12-05-2019 12:02:28.078 ] [mpi_amr_comm_setup]: buffer_dim_send=74785,
buffer_dim_recv=1
[ 12-05-2019 12:02:28.157 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:29.011 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 73
[GRID amr_refine_derefine] min blks 0 max blks 1 tot blks 73
[GRID amr_refine_derefine] min leaf blks 0 max leaf blks 1 tot leaf blks 64
[ 12-05-2019 12:02:29.440 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:29.833 ] [GRID gr_expandDomain]: iteration=2, create
level=4
[ 12-05-2019 12:02:30.253 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:30.322 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 585
[GRID amr_refine_derefine] min blks 2 max blks 4 tot blks 585
[GRID amr_refine_derefine] min leaf blks 1 max leaf blks 3 tot leaf blks 512
[ 12-05-2019 12:02:32.046 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:32.564 ] [GRID gr_expandDomain]: iteration=3, create
level=5
[ 12-05-2019 12:02:32.684 ] [mpi_amr_comm_setup]:
buffer_dim_send=233185, buffer_dim_recv=135889
[ 12-05-2019 12:02:33.443 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:34.807 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 3033
[GRID amr_refine_derefine] min blks 11 max blks 14 tot blks 3033
[GRID amr_refine_derefine] min leaf blks 9 max leaf blks 11 tot leaf
blks 2654
[ 12-05-2019 12:02:35.561 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:35.977 ] [GRID gr_expandDomain]: iteration=4, create
level=6
[ 12-05-2019 12:02:36.173 ] [mpi_amr_comm_setup]:
buffer_dim_send=575965, buffer_dim_recv=351529
[ 12-05-2019 12:02:37.611 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:40.401 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 6297
[GRID amr_refine_derefine] min blks 24 max blks 26 tot blks 6297
[GRID amr_refine_derefine] min leaf blks 20 max leaf blks 23 tot leaf
blks 5510
[ 12-05-2019 12:02:41.154 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:41.544 ] [GRID gr_expandDomain]: iteration=5, create
level=7
[ 12-05-2019 12:02:41.815 ] [mpi_amr_comm_setup]:
buffer_dim_send=540997, buffer_dim_recv=411109
[ 12-05-2019 12:02:44.513 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:46.394 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 16185
[GRID amr_refine_derefine] min blks 62 max blks 65 tot blks 16185
[GRID amr_refine_derefine] min leaf blks 54 max leaf blks 57 tot leaf
blks 14162
[ 12-05-2019 12:02:47.277 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:47.696 ] [GRID gr_expandDomain]: iteration=6, create
level=8
[ 12-05-2019 12:02:48.106 ] [mpi_amr_comm_setup]:
buffer_dim_send=881329, buffer_dim_recv=699133
[ 12-05-2019 12:02:50.281 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:02:51.436 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 89497
[GRID amr_refine_derefine] min blks 348 max blks 351 tot blks 89497
[GRID amr_refine_derefine] min leaf blks 304 max leaf blks 308 tot leaf
blks 78310
[ 12-05-2019 12:02:53.153 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:02:53.443 ] [GRID gr_expandDomain]: iteration=7, create
level=9
[ 12-05-2019 12:02:56.118 ] [mpi_amr_comm_setup]:
buffer_dim_send=1952545, buffer_dim_recv=1773109
[ 12-05-2019 12:03:03.333 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:03:07.980 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 626345
[GRID amr_refine_derefine] min blks 2445 max blks 2449 tot blks 626345
[GRID amr_refine_derefine] min leaf blks 2139 max leaf blks 2142 tot
leaf blks 548052
[ 12-05-2019 12:03:13.985 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:03:14.055 ] [GRID gr_expandDomain]: iteration=8, create
level=9
[ 12-05-2019 12:03:25.399 ] [mpi_amr_comm_setup]:
buffer_dim_send=6183577, buffer_dim_recv=5623945
[ 12-05-2019 12:03:55.420 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:03:58.377 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 97801
[GRID amr_refine_derefine] min blks 381 max blks 384 tot blks 97801
[GRID amr_refine_derefine] min leaf blks 333 max leaf blks 336 tot leaf
blks 85576
[ 12-05-2019 12:04:01.822 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:04:01.849 ] [GRID gr_expandDomain]: iteration=9, create
level=9
[ 12-05-2019 12:04:03.709 ] [mpi_amr_comm_setup]:
buffer_dim_send=2129845, buffer_dim_recv=1958629
[ 12-05-2019 12:04:11.340 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:04:15.464 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 626345
[GRID amr_refine_derefine] min blks 2445 max blks 2449 tot blks 626345
[GRID amr_refine_derefine] min leaf blks 2139 max leaf blks 2142 tot
leaf blks 548052
[ 12-05-2019 12:04:24.519 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:04:24.588 ] [GRID gr_expandDomain]: iteration=10, create
level=9
[ 12-05-2019 12:04:35.959 ] [mpi_amr_comm_setup]:
buffer_dim_send=6183577, buffer_dim_recv=5623945
[ 12-05-2019 12:05:06.370 ] [GRID amr_refine_derefine]: initiating
refinement
[ 12-05-2019 12:05:09.335 ] [GRID amr_refine_derefine]: redist. phase.
tot blks requested: 97801
[GRID amr_refine_derefine] min blks 381 max blks 384 tot blks 97801
[GRID amr_refine_derefine] min leaf blks 333 max leaf blks 336 tot leaf
blks 85576
[ 12-05-2019 12:05:12.763 ] [GRID amr_refine_derefine]: refinement complete
[ 12-05-2019 12:05:12.790 ] [GRID gr_expandDomain]: iteration=11, create
level=9
[ 12-05-2019 12:05:14.289 ] [gr_ensureValidNeighborInfo] found
mpi_pattern_id: -10
[ 12-05-2019 12:05:14.317 ] [mpi_amr_comm_setup]: buffer_dim_send=73525,
buffer_dim_recv=62437
[ 12-05-2019 12:05:14.806 ] memory: /proc vsize (MiB): 2918.54 (min)
3030.76 (max) 2946.03 (avg)
[ 12-05-2019 12:05:14.807 ] memory: /proc rss (MiB): 1798.14 (min)
1883.50 (max) 1830.61 (avg)
[ 12-05-2019 12:05:14.808 ] memory: /proc vsize (MiB): 2918.54 (min)
3030.76 (max) 2946.03 (avg)
[ 12-05-2019 12:05:14.809 ] memory: /proc rss (MiB): 1798.18 (min)
1883.53 (max) 1830.67 (avg)
[ 12-05-2019 12:05:17.553 ] [BHTree]: Coefficients in the Taylor
expansion written into file
The error file says:
Fatal error in PMPI_Waitall: See the MPI_ERROR field in MPI_Status for
the error code
The out file says:
RuntimeParameters_read: ignoring unknown parameter "energyFix"...
Grid_init: resolution based on runtime params:
lrefine dx dy dz
3 9.643E+17 9.643E+17 9.643E+17
4 4.821E+17 4.821E+17 4.821E+17
5 2.411E+17 2.411E+17 2.411E+17
6 1.205E+17 1.205E+17 1.205E+17
7 6.027E+16 6.027E+16 6.027E+16
8 3.013E+16 3.013E+16 3.013E+16
9 1.507E+16 1.507E+16 1.507E+16
MaterialProperties initialized
Cosmology initialized
Initializing Polytropic Equation of State
Source terms initialized
iteration, no. not moved = 0 0
refined: total leaf blocks = 8
refined: total blocks = 9
[amr_morton_process]: Initializing surr_blks using standard orrery
implementation
INFO: Grid_fillGuardCells is ignoring masking.
iteration, no. not moved = 0 6
iteration, no. not moved = 1 0
refined: total leaf blocks = 64
refined: total blocks = 73
iteration, no. not moved = 0 46
iteration, no. not moved = 1 2
iteration, no. not moved = 2 0
refined: total leaf blocks = 512
refined: total blocks = 585
iteration, no. not moved = 0 459
iteration, no. not moved = 1 58
iteration, no. not moved = 2 0
refined: total leaf blocks = 2654
refined: total blocks = 3033
iteration, no. not moved = 0 2675
iteration, no. not moved = 1 765
iteration, no. not moved = 2 0
refined: total leaf blocks = 5510
refined: total blocks = 6297
iteration, no. not moved = 0 5642
iteration, no. not moved = 1 1146
iteration, no. not moved = 2 0
refined: total leaf blocks = 14162
refined: total blocks = 16185
iteration, no. not moved = 0 15209
iteration, no. not moved = 1 1503
iteration, no. not moved = 2 0
refined: total leaf blocks = 78310
refined: total blocks = 89497
iteration, no. not moved = 0 79298
iteration, no. not moved = 1 4927
iteration, no. not moved = 2 0
refined: total leaf blocks = 548052
refined: total blocks = 626345
iteration, no. not moved = 0 91484
iteration, no. not moved = 1 36351
iteration, no. not moved = 2 0
refined: total leaf blocks = 85576
refined: total blocks = 97801
iteration, no. not moved = 0 89804
iteration, no. not moved = 1 6879
iteration, no. not moved = 2 0
refined: total leaf blocks = 548052
refined: total blocks = 626345
iteration, no. not moved = 0 91447
iteration, no. not moved = 1 36080
iteration, no. not moved = 2 0
refined: total leaf blocks = 85576
refined: total blocks = 97801
NSIZE = 4 1
Tree solver initialized
Finished with Grid_initDomain, no restart
Ready to call Hydro_init
Hydro initialized
Gravity initialized
Initial dt verified
I’m not sure what is the reason for this kind of crash, have you ever
experienced something similar?
My guess is that it is a communication problem due to the use of
mpich-3.2.1 but i’m not sure.
Cheers,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20191206/239ce6fc/attachment.htm>
More information about the flash-users
mailing list