[FLASH-USERS] Full message: FLASH 4.7: MPI-count sensitivity despite fixed-layout repeatability in a 2D radiation-hydrodynamics test

[Yun Hu]

Dear FLASH users and developers,

Apologies for the incomplete message sent earlier; only the greeting was included. The full message follows below.

We seek advice on MPI-count sensitivity in a locally configured FLASH 4.7 laser-driven CH-shell application. Our immediate goal is reliable relative X-ray predictions under a frozen configuration.

Within each MPI-count or repeatability comparison, we kept the executable and physical inputs unchanged. We compared native cells without interpolation or peak alignment. The XBL identifiers below label local audit records, not public references.

Recorded baseline configuration

| Item | Configuration | Audit record |
| Model / Hydro | 2D axisymmetric R-Z; three-temperature unsplit Hydro; HLLC and minmod | XBL-260911-838 |
| Mesh | PARAMESH; specialized refinement enabled; moving refinement off; lref=6, lrefine_max=6, nrefs=2 | XBL-260911-838 |
| Implicit diffusion | 24-group radiation diffusion; PCG with AMG; gr_hypreRelTol=1e-8 | XBL-260911-838 |
| Build metadata | mpiifort: -O2 -r8 -i4 -real_size 64; mpiicc: -O2 | XBL-260911-838 |
| Libraries | Static HYPRE 2.11.2; parallel HDF5 1.8.13; Intel MPI 2021.5 | XBL-260906-732; XBL-260910-818 |
| Execution | OpenMP/MKL threads fixed to one; measured core binding checked | XBL-260911-838 |

The exact underlying compiler version and complete local patch inventory remain unverified in this record. The recorded flags do not specify an explicit floating-point model.

Controlled comparisons

| Test | Observations | Audit record |
| MPI count: 96 versus 64 ranks, sequentially on the same two allocated nodes (48 versus 32 per node) | Initial arrays identical; matching native geometry at 141.7647 ps, but 56/62 checked fields differ | XBL-260907-744 |
| Repeatability: two independent cold starts, 64 ranks on one allocated node, same measured binding map | Initial and final arrays identical across all 62 checked fields; endpoint 141.7647 ps; also match the earlier same-configuration reference | XBL-260911-835 |

For the MPI-count test, density differences were 76.7375% by normalized maximum absolute difference and 0.90832% by axisymmetric-volume-weighted relative L1. Each norm uses the larger corresponding field norm as denominator. The logged total-energy difference was 0.00307799%. (XBL-260907-744)

These are between-run differences, not errors against a known solution or measured X-ray yield. Fixed-layout repeatability covers the checked early-time window only; it is not a full-pulse validation.

Porting the executable and major libraries did not eliminate differences. CPU topology, system libraries, communication provider and MPI tuning remained different. (XBL-260910-818)

Separate tolerance tests used modified settings, not the unchanged baseline above. Tightening tolerance reduced differences for one captured radiation system with identical matrix, right-hand side and initial guess. (XBL-260908-794) It did not generally remove field and timestep divergence during subsequent evolution. (XBL-260908-801) We have not established the root cause or demonstrated a source-code defect.

Could you advise on:

Are there known decomposition-sensitive issues or recommended fixes for this solver combination?
Which single diagnostic or standard FLASH test would best distinguish linear-solver sensitivity from mesh, coupling or time-integration effects?

We can prepare sanitized configuration excerpts or a distributable reproducer if helpful; the latter has not yet been constructed or validated. We are not attaching large checkpoints, source code or material tables to this initial message.

Thank you for your guidance.

Best regards,

harry
Yulin University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20260911/509498ad/attachment.htm>