8.4 Boundary Conditions

Much of the FLASH3 code within the Grid unit that deals with implementing boundary conditions has been organized into a separate subunit, GridBoundaryConditions. Note that the following aspects are still handled elsewhere:

• Recognition of bounday condition names as strings (in runtime parameters) and constants (in the source code); these are defined in RuntimeParameters_mapStrToInt and in constants.h, respectively.
• Handling of periodic boundary conditions; this is done within the underlying GridMain implementation. When using PARAMESH, the subroutine gr_createDomain is responsible for setting the neighbors of top-level blocks (to either other top-level blocks or to external boundary conditions) at initialization, after Nblockx $\times$ Nblocky $\times$ Nblockz root blocks have been created. periodic (wrap-around) boundary conditions are initially configured in this routine as well. If periodic boundary conditions are set in the $x$-direction, for instance, the first blocks in the $x$-direction are set to have as their left-most neighbor the blocks that are last in the $x$-direction, and vice versa. Thus, when the guard cell filling is performed, the periodic boundary conditions are automatically maintained.
• Handling of user-defined boundary conditions; this should be implemented by code under the
  Simulation directory.
• Low-level implementation and interfacing, such as are part of the PARAMESH code.
• Behavior of particles at a domain boundary. This is based on the boundary types described below, but their handling is implemented in GridParticles.

Although the GridBoundaryConditions subunit is included in a setup by default, it can be excluded (if no Config file “REQUIRES” it) by specifying -without-unit=Grid/GridBoundaryConditions. This will generally only make sense if all domain boundaries are to be treated as periodic. (All relevant runtime parameters xl_boundary_type etc. need to be set to "periodic" in that case.)

8.4.1 Boundary Condition Types

Boundary conditions are determined by the physical problem. Within FLASH, the parallel structure of blocks means that each processor works independently. If a block is on a physical boundary, the guard cells are filled by calculation since there are no neighboring blocks from which to copy values. Boundaries are selected by setting runtime parameters such as xl_boundary_type (for the `left' $X$-boundary) to one of the supported boundary types (Table 8.4.1) in flash.par. Even though the runtime parameters for specifying boundary condition types are strings, the Grid unit understands them as defined integer constants defined in the file constants.h, which contains all global constants for the code. The translation from the string specified in “flash.par” to the constant understood by the Grid unit is done by the routine RuntimeParameters_mapStrToInt.

Table 8.1: Hydrodynamical boundary conditions supported by FLASH. Boundary type ab may be replaced with $a$={x,y,z} for direction and $b$={l,r} for left/right edge. All boundary types listed except the last (user) have an implementation in GridBoundaryConditions.

ab_boundary_type	Description
periodic	Periodic (`wrap-around')
reflect,reflecting	Non-penetrating boundaries; plane symmetry, the normal vector components change sign
outflow	Zero-gradient boundary conditions; allows shocks to leave the domain
diode	like outflow, but fluid velocities are never allowed to let matter flow into the domain: normal velocity components are forced to zero in guard cells if necessary
axisymmetric	like reflect, but both normal and toroidal vector components change sign. Typically used with cylindrical geometry (R-Z) for the Z symmetry axis.
eqtsymmetric	like reflect for velocities but the magnetic field components, poloidal and toroidal, change sign. The sign of the normal magnetic field component remains the same. Typically used with cylindrical geometry (R-Z) for the R axis to emulate equatorial symmetry.
hydrostatic-f2	Hydrostatic boundary handling as in FLASH2. See remark in text.
hydrostatic-f2+nvrefl, hydrostatic-f2+nvout, hydrostatic-f2+nvdiode	Variants of hydrostatic-f2, where the normal velocity is handled specially in various ways, analogous to reflect, outflow, and diode boundary conditions, respectively. See remark in text.
user-defined or user	The user must implement the desired boundary behavior; see text.

To use any of the hydrostatic-f2* boundary conditions, the setup must include Grid/GridBoundaryConditions/Flash2HSE. This must usually be explicitly requested, for example with a line  

REQUIRES Grid/GridBoundaryConditions/Flash2HSE

in the simulation directory's Config file.

Note that the grav_boundary_type runtime parameter is used by some implementations of the Gravity unit to define the type of boundary for solving a self-gravity (Poisson) problem; see Gravity_init. This runtime parameter is separate from the ab_boundary_type ones interpreted by GridBoundaryConditions, and its recognized values are not the same (although there is some overlap).

Table 8.2: Additional boundary condition types recognized by FLASH. Boundary type ab may be replaced with a={x,y,z} for direction and b={l,r} for left/right edge. These boundary types are either reserved for implementation by users and/or future FLASH versions for a specific purpose (as indicate by the remarks), or are for special uses within the Grid unit.

ab_boundary_type	Constant	Remark
isolated	--	used by Gravity only for grav_boundary_type
--	DIRICHLET	used for multigrid solver
--	GRIDBC_MG_EXTRAPOLATE	for use by multigrid solver
--	PNEUMANN	(for use by multigrid solver)
hydrostatic	HYDROSTATIC	Hydrostatic, other implementation than FLASH2
hydrostatic+nvrefl	HYDROSTATIC_NVREFL	Hydrostatic variant, other impl.  than FLASH2
hydrostatic+nvout	HYDROSTATIC_NVOUT	Hydrostatic variant, other impl.  than FLASH2
hydrostatic+nvdiode	HYDROSTATIC_NVDIODE	Hydrostatic variant, other impl.  than FLASH2

8.4.2 Boundary Conditions at Obstacles

The initial coarse grid of root blocks can be modified by removing certain blocks. This is done by providing a non-trivial implementation of Simulation_defineDomain. Effectively this creates additional domain boundaries at the interface between blocks removed and regions still included. All boundary conditions other than periodic are possible at these additional boundaries, and are handled there in the same way as on external domain boundaries. This feature is only available with PARAMESH. See the documentation and example in Simulation_defineDomain for more details and some caveats.

8.4.3 Implementing Boundary Conditions

Users may need to implement boundary conditions beyond those provided with FLASH3, and the GridBoundaryConditions subunit provides several ways to achieve this. Users can provide an implementation for the user boundary type; or can provide or override an implementation for one of the other recognized types.

The simple boundary condition types reflect, outflow, diode are implemented in the
Grid_bcApplyToRegion.F90 file in Grid/GridBoundaryConditions. A users can add or modify the handling of some boundary condition types in a version of this file in the simulation directory, which overrides the regular version. There is, however, also the interface Grid_bcApplyToRegionSpecialized which by default is only provided as a stub and is explicitly intended to be implemented by users.
A Grid_bcApplyToRegionSpecialized implementation gets called before Grid_bcApplyToRegion and can decide to either handle a specific combination of boundary condition type, direction, grid data structure, etc., or leave it to Grid_bcApplyToRegion. These calls operate on a region of one block's cells at a time. FLASH will pass additional information beyond that needed for handling simple boundary conditions to Grid_bcApplyToRegionSpecialized, in particular a block handle through which an implementation can retrieve coordinate information and access other information associated with a block and its cells.

The GridBoundaryConditions subunit also provides a simpler kind of interface if one includes Grid/GridBoundaryConditions/OneRow in the setup. When using this style of interface, users can implement guard cell filling one row at a time. FLASH passes to the implementation one row of cells at a time, some of which are interior cells while the others represent guard cells outside the boundary that are to be modified in the call. A row here means a contiguous set of cells along a line perpendicular to the boundary surface. There are two versions of this interface: Grid_applyBCEdge is given only one fluid variable at a time, but can also handle data structures other than unk; whereas Grid_applyBCEdgeAllUnkVars handles all variables of unk along a row in one call. Cell coordinate information is included in the call arguments. FLASH invokes these functions through an implementation of Grid_bcApplyToRegionSpecialized in Grid/GridBoundaryConditions/OneRow which acts as a wrapper. GridBoundaryConditions/OneRow also provides a default implementation of Grid_applyBCEdge (which implements the simple boundary conditions) and Grid_applyBCEdgeAllUnkVars (which calls Grid_applyBCEdge) each. Another implementation of Grid_applyBCEdgeAllUnkVars can be found in GridBoundaryConditions/OneRow/Flash2HSE, this one calls Grid_applyBCEdge or, for FLASH2-type hydrostatic boundaries, the code for handling them. These can be used as templates for overriding implementations under Simulation. It is not recommended to try to mix both Grid_bcApplyToRegion*-style and Grid_applyBCEdge*-style overriding implementations in a simulation directory, since this could become confusing.

Note that in all of these cases, i.e., whether boundary guard cell filling for a boundary type is implemented in Grid_bcApplyToRegion, Grid_bcApplyToRegionSpecialized, Grid_applyBCEdge, or
Grid_applyBCEdgeAllUnkVars, the implementation does not fill guard cells in permanent data storage (the unk array and similar data structures) directly, but operates on buffers. FLASH3 fills some parts of the buffers with current values for interior cells before the call, and copies updated guardcell data from some (other) parts of the buffers back into unk (or similar) storage after the handling routine returns.

All calls to handlers for boundary conditions are for one face in a given dimension at a time. Thus for each of the IAXIS, JAXIS, and KAXIS dimensions there can be up to two series of calls, once for the left, i.e., “LOW,” and once for the right, i.e., “HIGH,” face. PARAMESH 4 makes additional calls for filling guard cells in edge and corner regions of blocks, these calls result in additional Grid_bcApplyToRegion* invocations for those cells that lie in diagonal directions from the block interior.

The boundary condition handling interfaces described so far can be implemented (and will be used!) independent of the Grid implementation chosen. At a lower level, the various implementations of GridMain have different ways of requesting that boundary guard cells be filled. The GridBoundaryConditions subunit collaborates with GridMain implementations to provide to user code uniform interfaces that are agnostic of lower-level details. However, it is also possible -- but not recommended -- for users to replace a routine that is located deeper in the Grid unit. For PARAMESH 4, the most relevant routine is amr_1blk_bcset.F90, for PARAMESH 2 it is tot_bnd.F90, and for uniform grid UG it is gr_bcApplyToAllBlks.F90.

8.4.3.1 Additional Concerns with PARAMESH 4

Boundary condition handling has become significantly more complex in FLASH3. In part this is so because PARAMESH 4 imposes requirements on guard cell filling code that do not exist in the other GridMain implementations:

1. In other Grid implementations, filling of domain boundary guard cells is under control of the “user” (in this context, the user of the grid implementation, i.e., FLASH): These cells can be filled for all blocks at a time that is predictable to the user code, as a standard part of handling Grid_fillGuardCells, only. With PARAMESH 4, the user-provided amr_1blk_bcset routine can be called from within the depths of PARAMESH on individual blocks (and cell regions, see below) during guard cell filling and at other times when the user has called a PARAMESH routine. It is not easy to predict when and in which sequence this will happen.
2. PARAMESH 4 does not want all boundary guard cells filled in one call, but requests individual regions in various calls.
3. PARAMESH 4 does not let the user routine amr_1blk_bcset operate on permanent storage (unk etc.) directly, but on (regions of) one-block buffers.
4. PARAMESH 4 occasionally invokes amr_1blk_bcset to operate on regions of a block that belongs to a remote processor (and for which data has been cached locally). Such block data is not associated with a valid local blockID, making it more complicated for user code to retrieve metadata that may be needed to implement the desired boundary handling.

Some consequences of this for FLASH3 users:

• User code that implements boundary conditions for the grid inherits the requirement that it must be ready to be called at various times (when certain Grid routines are called).
• User code that implements boundary conditions for the grid inherits the requirement that it must operate on a region of the cells of a block, where the region is specified by the caller.
• Such user code must not assume that it operates on permanent data (in unk etc.). Rather, it must be prepared to fill guardcells for a block-shaped buffer that may or may not end up being copied back to permanent storage.

  User code also is not allowed to make certain PARAMESH 4 calls while a call to amr_1blk_bcset is active, namely those that would modify the same one-block buffers that the current call is working on.

• The user code must not assume that the block data it is acting on belongs to a local block. The data may not have a valid blockID. The code will be passed a “block hande” which can be used in some ways, but not all, like a valid blockID.
  Caveat Block Handles: See the README file in Grid/GridBoundaryConditions for more information on how a block handle can be used.