<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii">
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head><body bgcolor="#FFFFFF" text="#000000" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<font style="font-size: 11px;" face="Menlo"><tt>Hello all,<br>
<br>
First time poster, hope I given enough info etc, let me know if not.</tt></font><div><font face="monospace"><span style="font-size: 11px;"><br></span></font><div><font style="font-size: 11px;" face="Menlo"><tt>I'm doing some
2D, </tt></font><span style="font-size: 11px; font-family: Menlo; ">axisymmetric hydro simulations of core collapse</span></div><div><span style="font-size: 11px; font-family: Menlo; ">with Sean Couch. </span><span style="font-size: 11px; font-family: Menlo; ">We
are implementing particles and I am running into</span></div><div><span style="font-size: 11px; font-family: Menlo; ">MPI errors and </span><tt style="font-size: 11px; ">segfaults after running for some time (these have never</tt></div><div><tt style="font-size: 11px; ">occurred when I don't include particles). </tt><span style="font-size: 11px; font-family: Menlo; ">For example, I'm </span><span style="font-size: 11px; font-family: Menlo; ">running
~30</span></div><div><span style="font-size: 11px; font-family: Menlo; ">models, each with 10000 particles and 16 MPI processes (2 nodes), within</span></div><div><span style="font-size: 11px; font-family: Menlo; ">the </span><span style="font-size: 11px; font-family: Menlo; ">last
24 hours </span><span style="font-size: 11px; font-family: Menlo; ">about </span><span style="font-family: Menlo; font-size: 11px; ">70% have failed with particle related errors at</span></div><div><span style="font-family: Menlo; font-size: 11px; ">various times. I have </span><span style="font-family: Menlo; font-size: 11px; ">tried updating to the </span><span style="font-size: 11px; font-family: Menlo; ">most </span><span style="font-size: 11px; font-family: Menlo; ">recent </span><span style="font-size: 11px; font-family: Menlo; ">openmpi version</span></div><div><span style="font-family: Menlo; font-size: 11px; ">(openmpi/1.6.5, with </span><span style="font-family: Menlo; font-size: 11px; ">gcc/4.8.2) and the errors </span><span style="font-family: Menlo; font-size: 11px; ">persist (I was using</span></div><div><span style="font-size: 11px; font-family: Menlo; ">openmpi/1.6.1 and gcc/4.7.2). </span><span style="font-size: 11px; font-family: Menlo; ">I did </span><span style="font-size: 11px; font-family: Menlo; ">implement my own</span></div><div><span style="font-family: Menlo; font-size: 11px; ">ParticleInitialization </span><span style="font-size: 11px; font-family: Menlo; ">method, but I doubt this </span><span style="font-size: 11px; font-family: Menlo; ">is </span><span style="font-size: 11px; font-family: Menlo; ">the issue as there</span></div><div><span style="font-size: 11px; font-family: Menlo; ">are </span><span style="font-size: 11px; font-family: Menlo; ">initialized and evolve </span><span style="font-size: 11px; font-family: Menlo; ">early on fine. But I </span><span style="font-size: 11px; font-family: Menlo; ">figured </span><span style="font-size: 11px; font-family: Menlo; ">I would mention</span></div><div><span style="font-size: 11px; font-family: Menlo; ">it just </span><span style="font-family: Menlo; font-size: 11px; ">in case.</span></div><div><div><div><font style="font-size: 11px;" face="Menlo"><tt><br>At least some of these errors seem robust, i.e. they are reproducible</tt></font></div><div><font style="font-size: 11px;" face="Menlo"><tt>with checkpoints, and to some extent reproducible even when I change</tt></font></div><div><font style="font-size: 11px;" face="Menlo"><tt>the number of processes (checkpointing from a 16 MPI process, 2 node</tt></font></div><div><font style="font-size: 11px;" face="Menlo"><tt>run to a </tt></font><span style="font-size: 11px; font-family: Menlo; ">8 MPI process, 1 node run gives a termination at the same time</span></div><div><span style="font-size: 11px; font-family: Menlo; ">step, however </span><span style="font-size: 11px; font-family: Menlo; ">a different error, 4 MPI process, on a 8-core node doesn't</span></div><div><span style="font-size: 11px; font-family: Menlo; ">crash on a checkpoint restart). </span></div><div><span style="font-size: 11px; font-family: Menlo; "><br></span></div><div><span style="font-size: 11px; font-family: Menlo; ">I'm not sure not to go about debugging </span><span style="font-size: 11px; font-family: Menlo; ">this, </span><span style="font-size: 11px; font-family: Menlo; ">so I </span><span style="font-size: 11px; font-family: Menlo; ">figured I would </span><span style="font-size: 11px; font-family: Menlo; ">start</span></div><div><span style="font-size: 11px; font-family: Menlo; ">with the users list to solicit advice </span><span style="font-size: 11px; font-family: Menlo; ">from </span><font face="Menlo" style="font-size: 11px; "><tt>Particle experts for any </tt></font><font face="Menlo" style="font-size: 11px; "><tt>easy</tt></font></div><div><font face="Menlo" style="font-size: 11px; "><tt>solutions </tt></font><span style="font-size: 11px; font-family: Menlo; ">and/or tips on the best place </span><span style="font-size: 11px; font-family: Menlo; ">to </span><span style="font-size: 11px; font-family: Menlo; ">start. The errors seem </span><span style="font-size: 11px; font-family: Menlo; ">to be</span></div><div><span style="font-size: 11px; font-family: Menlo; ">of the </span><font face="Menlo"><span style="font-size: 11px;">following
type (see below). I've included some infer about our setup</span></font></div><div><font face="Menlo"><span style="font-size: 11px;">and how I </span></font><span style="font-size: 11px; font-family: Menlo; ">add the particles in my configuration at the end of the email</span></div><div><span style="font-size: 11px; font-family: Menlo; ">(perhaps </span><span style="font-size: 11px; font-family: Menlo; ">I am missing something there).</span></div><div><font style="font-size: 11px;" face="Menlo">
</font></div><div><tt><font face="Menlo"><br></font></tt></div><div><tt><font face="Menlo">Thanks for any help, Happy Holidays,</font></tt></div><div><tt><font face="Menlo">Evan O'Connor </font></tt></div><div><br></div><div><tt><font face="Menlo">Typical Crash Type 1: invalid rank:</font></tt></div><div><tt><font face="Menlo"><br></font></tt></div><div><tt><font face="Menlo"><div>[tpb218:3617] *** An error occurred in MPI_Send</div><div>[tpb218:3617] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0</div><div>[tpb218:3617] *** MPI_ERR_RANK: invalid rank</div><div>[tpb218:3617] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort</div><div>--------------------------------------------------------------------------</div><div>mpirun has exited due to process rank 2 with PID 3617 on</div><div>node tpb218 exiting improperly. There are two reasons this could occur:</div><div><br></div><div>1. this process did not call "init" before exiting, but others in</div><div>the job did. This can cause a job to hang indefinitely while it waits</div><div>for all processes to call "init". By rule, if one process calls "init",</div><div>then ALL processes must call "init" prior to termination.</div><div><br></div><div>2. this process called "init", but exited without calling "finalize".</div><div>By rule, all processes that call "init" MUST call "finalize" prior to</div><div>exiting or it will be considered an "abnormal termination"</div><div><br></div><div>This may have caused other processes in the application to be</div><div>terminated by signals sent by mpirun (as reported here).</div><div>--------------------------------------------------------------------------</div><div><br></div><div><br></div></font></tt></div><div><tt><font face="Menlo">Typical Crash Type 2: segfault, invalid memory, either occurs in</font></tt></div><div><tt><font face="Menlo">grid_moveparticles or io_writeparticles</font></tt></div><div><tt><font face="Menlo"><br></font></tt></div><div><tt><font face="Menlo">a) grid_moveparticles:</font></tt></div><div><tt><font face="Menlo"><div>Program received signal SIGSEGV: Segmentation fault - invalid memory reference.</div><div><br></div><div>Backtrace for this error:</div><div>#0 0x2B588B0082D7</div><div>#1 0x2B588B0088DE</div><div>#2 0x2B588C25091F</div><div>#3 0x2B588C2A7131</div><div>#4 0x621E7E in ut_sortonprocs_</div><div>#5 0x50C6C4 in gr_ptmovepttopt_</div><div>#6 0x448B17 in grid_moveparticles_</div><div>#7 0x461EF6 in particles_advance_</div><div>#8 0x431750 in driver_evolveflash_</div><div>[tpb205][[47673,1],13][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)</div><div>--------------------------------------------------------------------------</div><div>mpirun noticed that process rank 5 with PID 8352 on node tpb206 exited on signal 11 (Segmentation fault).</div><div>--------------------------------------------------------------------------</div></font></tt></div><div><tt><br></tt></div></div><div><tt>b) io_writeparticles:</tt></div><div><tt><div>Program received signal SIGSEGV: Segmentation fault - invalid memory reference.</div><div><br></div><div>Backtrace for this error:</div><div>#0 0x2B677C23C2D7</div><div>#1 0x2B677C23C8DE</div><div>#2 0x2B677D48491F</div><div>#3 0x5EB058 in __namevaluell_data_MOD_namevaluell_checkreal</div><div>#4 0x5ED9E4 in namevaluell_setreal_</div><div>#5 0x434329 in driver_sendoutputdata_</div><div>#6 0x45560D in io_updatescalars_</div><div>#7 0x457E4C in io_writeparticles_</div><div>#8 0x4548B1 in io_output_</div><div>#9 0x431868 in driver_evolveflash_</div><div>[tpb203][[2990,1],13][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)</div><div>--------------------------------------------------------------------------</div><div>mpirun noticed that process rank 5 with PID 10525 on node tpb204 exited on signal 11 (Segmentation fault).</div><div>--------------------------------------------------------------------------</div></tt></div><div><tt><br></tt></div></div><div><tt><br></tt></div><div><tt>lines added to configuration file: Shell is implemented in a similar way as LATTICE</tt></div><div><tt><br></tt></div><div><tt><div>PARTICLETYPE passive INITMETHOD shell MAPMETHOD quadratic ADVMETHOD rungekutta</div><div>REQUIRES Particles/ParticlesMain</div><div>REQUESTS Particles/ParticlesMain/passive/RungeKutta</div><div>REQUESTS Particles/ParticlesMapping/Quadratic</div><div>REQUESTS Particles/ParticlesInitialization/Shell</div><div>REQUIRES IO/IOMain</div><div>REQUIRES IO/IOParticles</div><div>REQUIRES Grid/GridParticles</div><div><br></div><div>PARTICLEPROP dens REAL</div><div>PARTICLEPROP temp REAL</div><div>PARTICLEPROP ye REAL</div><div>PARTICLEPROP velx REAL</div><div>PARTICLEPROP vely REAL</div></tt></div><div><tt><br></tt></div><div><tt>The setup line is:</tt></div><div><tt><br></tt></div><div><tt>./setup CoreCollapse/leakage -auto -2d +cylindrical -nxb=16 -nyb=16 -objdir ccsn2dLeak threadBlockList=False +pm4dev threadWithinBlock=False +newMpole +uhdopt</tt></div><div><tt><br></tt></div><div><tt><br></tt></div></div></body></html>