1.1 What's New in FLASH4

This Guide describes the release version 4.8 of FLASH4. FLASH4 includes all the well tested capabilities of FLASH3. There were a few modules in the official releases of FLASH2 which were added and tested by local users, but did not have standardized setups that could be used to test them after the migration to FLASH3. Those modules are not included in the official releases of FLASH3 or FLASH4, however, they are being made available to download ”as is” from the Flash Center's website. We have ensured that they have been imported into FLASH4 to the extent that they conform to the architecture and compile. We cannot guarantee that they work correctly; they are meant to be useful starting points for users who need their functionality. We also welcome setups contributed by the users that can meaningfully test these units. If such setups become available to us, the units will be released in future.

In terms of the code architecture, FLASH4 closely follows FLASH3. The major changes from FLASH3 are several new capabilities in both physics solvers and infrastructure. Major effort went into the design of the FLASH3 architecture to ensure that the code can be easily modified and extended by internal as well as external developers. Each code unit in FLASH4, like in FLASH3 has a well defined interface and follows the rules for inheritance and encapsulation defined in FLASH3. One of the largest achievements of FLASH3 was the separation of the discretized `grid' architecture from the actual physics. This untangling required changes in the deepest levels of the code, but has demonstrated its worth by allowing us to import a new AMR package Chombo into the code.

Because of the increasing importance of software verification and validation, the Flash code group has developed a test-suite application for FLASH3. The application is called FlashTest and can be used to setup, compile, execute, and test a series of FLASH code simulations on a regular basis. FlashTest is available without a license and can be downloaded from the Code Support Web Page. There is also a more general open-source version of FlashTest which can be used to test any software in which an application is configured and then executed under a variety of different conditions. The results of the tests can then be visualized in a browser with FlashTestView, a companion to FlashTest that is also open-source.

Many but not all parts of FLASH4 are backwards compatible with FLASH2, and they are all compatible with FLASH3. The Flash code group has written extensive documentation detailing how to make the transition from FLASH2 to FLASH3 as smooth as possible. The user should follow the "Name changes from FLASH2 to FLASH3” link on the Code Support Web Page for help on transitioning to FLASH4 from FLASH2. The transition from FLASH3 to FLASH4 does not require much effort from the users except in any custom implementation they may have.

The new capabilities in FLASH4 that were not included in FLASH3 include

• 3T capabilities in the split and unsplit Hydro solvers. There is support for non-cartesian geometry and the unsplit solver also supports stationary rigid body.
• Upwind biased constrained transport (CT) scheme in the unsplit staggered mesh MHD solver
• Full corner transport upwind (CTU) algorithm in the unsplit hydro/MHD solver
• Cylindrical geometry support in the unsplit staggered mesh MHD solver on UG and AMR. A couple of MHD simulation setups using cylindrical geometry.
• Units for radiation diffusion, conduction, and heat exchange.
• Equation-of state unit includes table based multi-material multi-temperature implementation.
• The Opacities unit with the ability to use hot and cold opacities.
• The laser drive with threading for performance
• Ability to replicate mesh for multigroup diffusion or other similar applications.
• Several important solvers have been threaded at both coarse-grain (one block per thread) and fine-grain (threads within a block) levels.
• Several new HEDP simulation setups.
• A new multipole solver
• Ability to add particles during evolution

The enhancements and bug fixes to the existing capabilities since FLASH4-beta release are :

• The HLLD Riemann solver has been improved to handle MHD degeneracy.
• PARAMESH's handling for face-centered variables in order to ensure divergence-free magnetic fields evolution on AMR now uses gr_pmrpDivergenceFree=.true. and gr_pmrpForceConsistency=.true. by default.
• The HEDP capabilities of the code have been exercised and are therefore more robust.
• Laser 3D in 2D ray tracing has been added. The code traces rays in a real 3D cylindrical domain using a computational 2D cylindrical domain and is based on a polygon approximation to the angular part.
• In non-fixedblocksize mode, restart with particles did not work when starting with a different processor count. This bug has now been fixed.
• All I/O implementations now support reading/writing 0 blocks and 0 particles.
• There is support for particles and face variables in PnetCDF
• Initializaton of of the computation domain has been optimized by eliminating unnecessary invocations of PARAMESH's “digital orrery” algorithm at simulation startup. It is possible to run the orrery in a reduced communicator in order to speed up FLASH initialization.
• The custom region code and corresponding Grid API routines have been removed.
• PARAMESH4DEV is now the default PARAMESH implementation.

The new capabilities in FLASH4.2 ... FLASH4.2.2 since FLASH4.0.1 include:

• New Core-Collapse Super Nova (CCSN) physics, with complete nuclear EOS routines, local neutrino heating/cooling and multispecies neutrino leakage.
• New unsplit Hydro and MHD implementations, highly optimized for performance. These implementations are now the default option. We have retained the old implementations as an unsplit_old alternative for compatibility reasons.
• New support for 3T magnetohydrodynamics, designed for HEDP problems.
• A new magnetic resistivity implementation, SpitzerHighZ, for HEDP problems. We have also extended the support for resistivity in cylindrical geometry in the unsplit solver.
• New threading capabilities for unsplit MHD, compatible with all threading strategies followed by the code.
• New, improved multipole Poisson solver, implementing the algorithmic refinements described in http://dx.doi.org/10.1088/0004-637X/778/2/181 and http://arxiv.org/abs/1307.3135.
• Reorganization of the EnergyDeposition unit. A new feature has been included that allows EnergyDeposition to be called once every $n$ time steps.

The new capabilities in FLASH4.3 since FLASH4.2.2 include:

• The sink particles implementation now has support for particles to remain active when leaving the grid domain (in case of outflow boundary conditions).
• New Proton Imaging unit: The new unit is a simulated diagnostic of the Proton Radiography used in HEDP experiments.
• Flux-limited-diffusion for radiation (implemented in RadTransMain/MGD) is now available for astrophysical problem setups:
  • MatRad3 (matter+rad [2T] stored in three components) implementations for several Eos types: Gamma, Multigamma, and (experimentally) Helmholtz/SpeciesBased.
  • Implemented additional terms in FLD Rad-Hydro equations to handle streaming and transition-to-streaming regimes better - including radiation pressure. This is currently available as a variant of the unsplit Hydro solver code, under HydroMain/unsplit_rad . We call this RADFLAH - Radiation Flux-Limiter Aware Hydro. Setup with shortcut +uhd3tR instead of +uhd3t . This has had limited testing, mostly in 1D spherical geometry.
  • New test setups under Simulation/SimulationMain/radflaHD: BondiAccretion, RadBlastWave
  • Various fixes in Eos implementations.
  • New "outstream" diffusion solver boundary condition for streaming limit. (currently 1D spherical only)
  • Added Levermore-Pomraning flux limiter.
  • More flexible setup combinations are now easily possible - can combine, e.g., species declared on setup command line with SPECIES in Config files and initialized with Simulation_initSpecies, by setup with ManualSpeciesDirectives=True.
  • Created an "Immediate" HeatExchange implementation.
  • EXPERIMENTAL: ExpRelax variant of RadTrans diffusion solver, implements the algorithm described in Gittings et al (2008) for the RAGE code, good for handling strong matter-radiation coupling; for one group (grey) only.
  • EXPERIMENTAL: Unified variant of RadTrans diffusion solver, for handling several coupled scalar equations with HYPRE.
  • EXPERIMENTAL: More accurate implementation of flux limiting (and evaluation of diffusion coeffs): apply limiter to face values, not cell centered values.
• Gravity can now be used in 3T simulations.
• Laser Energy Deposition: New ray tracing options added based on cubic interpolation techniques. Two variants: 1) Piecewise Parabolic Ray Tracing (PPRT) and 2) Runge Kutta (RK) ray tracing.
• Introduction of new numerical tool units: 1) Interpolate: currently contains the routines to set up and perform cubic interpolations on rectangular 1D,2D,3D grids, 2) Roots: (will) contain all routines that solve $f(x)=0$ (currently contains quadratic, cubic and quartic polynomial root solvers, 3) Runge Kutta: sets up and performs Runge Kutta integration of arbitrary functions (passed as arguments).
• Unsplit Hydro/MHD: Local CFL factor using CFL_VAR. (Declare a "VARIABLE cfl" and initialize it appropriately.)
• Unsplit Hydro/MHD: Significant reorganization.
  • reorganized definition and use of scratch data. Memory savings.
  • use hy_memAllocScratch and friends.
  • hy_fullRiemannStateArrays (instead of FLASH_UHD_NEED_SCRATCHVARS)
  • New runtime parameter hy_fullSpecMsFluxHandling, default TRUE. resulting in flux-corrected handling for species and mass scalars, including USM.
  • Use shockLowerCFL instead of shockDetect runtime parameter.
  • Revived EOSforRiemann option.
  • More accurate handling of geometric effects close to the origin in 1D spherical geometry.

Important changes in FLASH4.4 since FLASH4.3 include:

• The default Hydro implementation has changed from split PPM to unsplit Hydro. A new shortcut +splitHydro can be used to request a split Hydro implementation.
• Updated values of many physical constants to 2014 CODATA values. This may cause differences from previously obtained results. The previous values of constants provided by the PhysicalConstants unit can be restored by replacing the file PhysicalCosntants_init.F90 with an older version; the version from FLASH4.3 is included as PhysicalConstants_init.F90.flash43. This should only be done to reproduce previous simulation results to bit accuracy.
• An improved Newton-Raphson search in the 3T Multi-type Eos implemention (MTMMMT, including Eos based on IONMIX tables) can prevent some cases of convergence failure by bounding the search. This implementation follows original improvements made to the Helmholtz Eos implementation by Dean Townsley.
• Added new Poisson solvers (Martin-Cartwright Geometric Multigrid and BiPCGStab, which uses multigrid aspreconditioner). Combinations of homogeneous Dirichlet, Neumann, and periodic boundary conditions are supported (although not yet “isolated” boundaries for self-gravity).
• Added the IncompNS physics unit, which provides a solver for incompressible flow problems on rectangular domains. Multistep and Runge-Kutta explicit projection schemes are used for time integration. Implementations on staggered grid arrangement for both uniform grid (UG) and adaptive mesh refinement (AMR) are provided. The new Poisson solvers are employed for AMR cases, whereas the homogeneous trigonometric solver + PFFT can be used in UG. Typical velocity boundary conditions for this problem are implemented.
• The ProtonImaging diagnostics code has been improved. Time resolved proton imaging is now possible, where protons are traced through the domain during several time steps. The original version (tracing of protons during one time step with fixed domain) is still available.
• The code for Radiation-Fluxlimiter-Aware Hydro has been updated. Smoothing of the flux-limiter function within the enhanced Hydro implementation has been implemented and has been shown effective in increasing stability in 1D simulations.
• New Opacity implementations: BremsstrahlungAndThomson and OPAL. These are for gray opacities.
• In addition to the FLASH4.4 release, the publicly available Python module opacplot2 has received significant development (credit to JT Laune). It can assist in handling EoS/opacity tables, and includes command line tools to convert various table formats to IONMIX and to compare between different tables. More information can be found in the Flash Center's GitHub repository at https://github.com/flash-center/opacplot2.

New additions in FLASH4.5 since FLASH4.4 include:

• New implementation for synthetic Thomson scattering simulated diagnostic.

• A new 1D simulation of a 2-temperature Supernova evolution using the flux-limiter-aware hydro treatment of radiation, and related code changes.

• A new timestep limiter, which ensures that a given list of variables remains positive-definite during hydrodynamic advection. This can be used as an alternative to lowering the Hydro unit's CFL runtime parameter, and has been shown useful for 3T simulations in particular, where non-conservative equations are considered. This will obviate negative temperature errors when using 3T Hydro and MHD. See Sec:dr_posdef in the Driver chapter for more information.

• Expanded the Sedov simulation to be initialized from a quasi-analytical profile, and added diagnostic variables to keep track of various simulation error measures.

• New flux limiter "vanLeeer1.5" for unsplit Hydro and MHD.

• Fixed an issue in which PARAMESH block distribution failed to settle down because of message delay, reported for some machines. This is done by inserting sleep(1.) calls in extreme cases.

• Implement saving of laser irradiation (to "lase" variable) for 2DCyl3D ray tracing, too. Previously was 3D only.

• We have included detailed instructions that outline how to easily install FLASH on Windows, Mac OS, and Linux distributions, using Docker. See Sec:qs-docker in the "Quick Start" chapter.

• FLASH Interface is a new graphical interface (GUI) for managing parameter files. and is now available for download. The code was developed by Christopher Walker (cnwalker@uchicago.edu). It currently supports:
  • Configuring and editing existing flash.par files;
  • Generating new flash.par files by choosing among the full set of possible parameters;
  • Filtering the full set of possible FLASH parameters (found in setup_params) via categories;
  • Searching for FLASH params by name, and viewing full descriptions of each parameter's role and function.
  The source code, along with download and build instructions can be found at https://github.com/cnwalker/FLASH-Interface. Compiled executables for a number of operating systems can be found at http://flash.rochester.edu/site/flashcode/FLASH-Interface

• The PRaLine code for Proton Radiography Linear reconstruction is now included with the FLASH code tarball and is available for download at https://github.com/flash-center/PRaLine This standalone Python code was originally developed for the magnetic field reconstruction method using proton radiography, discussed in Graziani et al. 2017 (in press), https://arxiv.org/abs/1603.08617 Alemayehu Solomon Bogale (alemsolobog@rochester.edu) restructured the code and made it suitable for public release. For more information see the README files in tools/protonRad and tools/protonRad/PRaLine .

New additions in FLASH4.6 since FLASH4.5 include:

• Implementation of new nonideal MHD effects: Hall term (Cartesian 1D/2D/3D and cylindrical 2D), and Biermann battery (Cartesian 2D/3D and cylindrical 2D) as a source term and a flux-based formulation for 3T.

• Corrections on SpitzerHighZ magnetic resistivity. Now using the NRL plasma formulary equation.

• New high-order reconstruction methods in unsplit Hydro / MHD solvers:
  • PCM: Piecewise Cubic Method, see Lee et al, JCP 341, 230, 2017
  • GP: Gaussian Process, see Reyes et al., JCP 381, 189, 2019; Reyes et al., JSC 76, 443, 2018

• Added support for anisotropic thermal conduction. Anisotropic conductivities depend on the direction of a magnetic field, so anisotropic conduction is meant to be used together with MHD. Currently only available with the uniform Grid (UG) implementation.

• New implementation for simulated Thomson scattering experimental diagnostic using ray tracing. (source/diagnostics/ThomsonScattering)

• Implementation of X-ray imaging simulated experimental diagnostic using ray tracing.
  (source/diagnostics/XrayImaging)

• Implementation of proton emission simulated experimental diagnostic using ray tracing.
  (source/diagnostics/ProtonEmission)

• Enabled support for 1D cylindrical laser energy deposition.

• Applied all known patches for working with HDF5 versions 1.10.x.

• Compatibility updates to the setup utility so it works with Python2 and Python3 versions.

• Updated various Config files for Python3 compatibility.

These are the changes from FLASH4.6 to FLASH4.6.1:

• Corrected Ohmic heating term for nonideal MHD in hy_uhd_unsplitUpdate: erroneous multiplication by density

• New runtime parameter for optionally specifying laser beam intensities in 1D and 2D Cartesian simulations in a more natural way by determining the beam power profile via beam power, not power per length or area unit.

• Enabled and updated XrayImaging unit, included new unit tests.

• Fixed occasional generation of invalid refinement pattern by PARAMESH in 2D and 3D spherical coordinates.

• Disabled PM_OPTIMIZE_MORTONBND_FETCHLIST to avoid rare runtime error in guard cell communication.

• Added modifications to make setup work better with Python3, including removing 8bit characters and TAB character consistency. We have also added some extra features in the setup command.

• Improved detection of case-insensitive file system by setup to avoid unnecessary recompilation by make in some Docker environments.

• Disabled use of 1D profile in default configuration of Sedov example.

These are the changes from FLASH4.6.1 to FLASH4.6.2:

• For unsplit Hydro with curvilinear coordinates we have improved the computation of velocity divergence used in determining the strength of artificial viscosity (when use_avisc is TRUE). The effects of artificial viscosity can be expected to change, particularly in cells close to the coordinate axis in 2D cylindrical coordinates.

• Corrected wrong order of operations in an MHD code path when using flux correction: energy corrections (proper accounting of magnetic energy in cells) could be applied too late in interior cells. The problem only occurred in 1D MHD configurations (with the MHD_StaggeredMesh code), which could then lead to errors in results of 3T simulations.

• Corrected wrong Eos mode in Eos_wrapped calls, as well as energy adjustment, in the Biermann battery source term code.

• LaserRayTrace and ThomsonScatteringMain/WithRayTracing: Improved ray tracing in situations where rays enter the computational domain in locations that have significant electron density by adding ray speed adjustment at domain entry.

• Fixed some problems in Multitype EOS that showed up in tests with non-fixed-blocksize uniform grid mode. Convergence of root search should now be improved in some EOS modes.

• Slightly improved time step computation for anisotropic diffusion (Uniform Grid).

• FLASH now requires being compiled against MPI version 2 or higher. Some setups continue to work when compiled against an MPI-1 library, but this will not be true in general.

These are the changes from FLASH4.6.2 to FLASH4.7:

• LaserRayTrace and ThomsonScatteringMain/WithRayTracing: 3D-cylindrical ray tracing; other ray tracing changes. (experimental)

• Boundary condition aware LaserRaysCreate routines that allow rays to enter from periodic/reflecting boundaries with appropriate symmetry transformations.

• Support for 3D cylindrical geometry: split and unsplit Hydro, (experimental)

• Reorganize flux correction for the staggered mesh MHD implementation so that MagneticResistivity can be used with conserveAngField disabled in cylindrical coordinates.

• Enhancments in implicit solves:
  • Anisotropic thermal conduction now supports; AMR grids with PARAMESH, as well as slope limiting to avoid unphysical temperature changes.
  • Periodic boundary conditions are now supported in implicit HYPRE solvers.

  • We have added an operator split HYPRE solver for implicit magnetic diffusion on uniform grids.

• We have add the full Braginskii extended-MHD terms in the generalized OHM's law for the unsplit staggered mesh hydro solver, including:
  • Full Anisotropic Magnetic Resistivity
    • UG +AMR
    • $\eta_{\parallel}$
    • $\eta_\perp$
    • $\eta_\wedge$
  • Hall
  • Nernst
  • Seebeck
  • cross-field term
  • Biermann

• The Biermann solver now has a symmetrized use of SHOK_VAR (previously left-biased)

• New materialProperties implementations of transport coefficents, including:
  • Ji & Held - thermal conduction
  • Davies & Wen - thermal electric & resistivity
  • Spitzer Viscosity
  • New multi-species API for calculating $Z^2$
  • Fixed bug in calculation of $\tau$ for Epperlein & Haines
  • Consolidate older implementations to use $Z_{\text{eff}}$

• We have implemented a Current Drive Unit that implements a magnetic field boundary condition for modeling Z-pinch experiments using a variety of models for the current drive.
  • This includes a circuit model that can model systems like the Z-machine and CESZAR.

• We have added native readers in FLASH for tabulated data, including the LANL SESAME format for EoS and the LANL TOPS format for opacities.

• The unsplit hydrodynamics solver has been updated with the option for a volume-of-fluid interface capturing scheme for maintaining sharp material interfaces. Available for 1D/2D uniform grids in Cartesian geometry.

• The I/O routines have been updated for compatibility with Open-MPI 3.

• The python scripts, including setup, have been updated to be fully compatible with Python3.

These are the changes from FLASH4.7 to FLASH4.7.1:

• We have added the missing FL_NONE flux limiter for time step calculations in extended MHD that use the Nernst term.

• We have removed the deprecated local log-lambda subroutine from Spitzer magnetic resistivity unit that could cause some compile issues.

• We have corrected calls to PlasmaState_tau in hy_uhd_computeDtExtMHD.

• We have modified the ion viscous heating in hy_uhd_ragelike to now use the correct scratch variables in hydro & MHD.

• We have fixed improper memory deallocations in eos_idealgamma3T when set up with multitype EoS in non-fixed block size uniform grids.

• We have fixed a bug in implicit magnetic diffusion HYPRE solver that only allowed diffusion of MAGX.

• We have updated the SESAME EoS table reader to use geometric spacing in the interpolated density/temperature grids as well as a species runtime parameter to specify the double format of the table (defaults to E22.15).

• We have made consistent all the output of the Circuit unit (Circuit.dat) so all data uses the same time centering.

• We have fixed a bug in Thomson Scattering diagnostic that occurred when compiled in debug mode.

• We have modified the fast magnetosonic speed calculation in timestep computation to avoid negative numbers.

These are the changes from FLASH4.7.1 to FLASH4.8

• Added python bindings to a subset of flash units using the pybind11 library
• notable routines that may be written in python and coupled to a FLASH simulation include:
• Simulation initialization & adjustEvolution
• user-defined boundary conditions
• python function as a runtime parameter file
• Examples are provided in the source/Simulation/SimulationMain/python directory
• Added support for CMake builds with the "-cmake" setup argument. Site configurations may be provided with a "site.cmake" in the corresponding directory in "sites"
• Introducing a new unified interface to the HYPRE library that may be used solve generic parabolic PDEs in FLASH in "source/Grid/GridSolvers/HYPRE/Unified". The interface is now fully compatible with AMR and only requires users to specify coefficient tensors and an UNK index for transport coefficients.
• Introducing a new fully implicit operator split viscosity solver built on the Unified HYPRE interface in both UG & AMR grids
• The implicit magnetic resistivity solver has been moved to the Unified HYPRE interface and now supports AMR grids.
• vector diffusion requires HYPRE versions >= 2.24
• The solver evolves the cell-centered fields and there are two update modes for coupling to the staggered face fields
• In the staggered update, a full divergence preserving constrained-transport update is performed
• In the other, the cell-centered fields are averaged to the faces
• The *_fullState interfaces for MagneticResistivity & Conductivity are now the only interface
• Those that used (xdens, xtemp) as input have been removed
• The PlasmaState unit now calculates the ion-electron equilibration time. There are no longer separate calculations in the Leemore/spitzer heat exchange implementations, and the same calculation can now be used in other material property units.
• The circuit unit has been reorganized.
• Added an interface for setting Dirichlet boundary conditions in the RadTrans MGD solver to be more easily specified from a fixed radiation temperature. See the "RadSlab" provided test problem. This enables radiative ablation problems.
• BiermannSource + 3D now will abort, since the source term not compatible with the staggered mesh update
• 3T MHD simulations using the flux-based Biermann battery solver (biermannSource=.false.) no longer require useBiermann3T so that Biermann may be used without turning on Hall MHD. This will allow larger timesteps in certain problems.
• Bug fixes:
• Multi-material property TOPS opacity tables are now supported
• A bug in the usage of TOPS tables with the op_useLogTables switch has been fixed
• The SESAME reader has been made more robust to edge cases in the file format

The following features are provided on an EXPERIMENTAL basis. They may only work in limited circumstances and/or have not yet been tested to our satisfaction.

• New Laser - Async communication (experimental).
• Electron-Entropy Advection in Hydro for non-ideal Eos.
• New cubic and quartic equation solvers have been added and are ready to be used. They return only real cubic and quartic roots. The routines can be found in the flashUtilites/general section
• An alternative setup tool “setup_alt” intended to be a compatible replacement with a cleaner structure.