8.7 Chombo

We have included an experimental Grid implementation which makes use of Chombo library in FLASH4. Since this is very much a work in progress it should not be considered as production grade yet. We expect new developments and improvement to happen at a rapid pace, so please contact us if you would like to use the latest snapshots of our Chombo integration in FLASH before our next major code release.

Most flash.par parameters relevant to the Grid unit will continue to work; the differences are described later in this section. There are some important restrictions to be aware of when using this Grid implementation.

• Only Cartesian geometries are handled.
• Only cell-centered solution variables are supported, i.e. those variables defined with keywords VARIABLE and SCRATCHCENTERVAR in Config files.

FLASH does not include the source code for Chombo library and so Chombo must be independently downloaded and then built using the instructions given in Section 5.9. Please also see the restrictions in Hydro Section 15.1.6 and I/O Section 9.12.

Our integration of FLASH with Chombo is slightly unusual because we do not use the higher-level Chombo classes for solving time-dependent problems on an AMR mesh. The reason is that these higher level classes also "drive" the simulation, that is, control the initialization and evolution of simulations. These are tasks that already happen in a FLASH application in the Driver unit. Our approach is therefore to use the lower-level Chombo classes so that Chombo library is only responsible for Grid related tasks.

The actual patch-based mesh is allocated and managed in Chombo library. Each individual patch is stored in contiguous memory locations with the slowest varying dimension being the solution variable. This has allowed us to create a Grid_getBlkPtr that builds a 4D Fortran pointer object from the memory address of the first element of a specified patch. The patch is treated as a standard FLASH block in the physics units of FLASH. The Grid unit obtains the memory address and other metadata such as patch size, coordinates, index limits, cell spacing and refinement level through non-decorated C++ function calls which query Chombo objects.

8.7.1 Using Chombo in a UG configuration

This is a Grid implementation that operates in the same way as the standard FLASH uniform grid mode in which there is one block per MPI process. The parameters accepted are identical to those described for a non fixed blocksize uniform grid in Section 8.5.2. The only difference is that the checkpoint file is written in Chombo file layout rather than FLASH file layout. Setup a simulation using:

 

./setup Sedov -auto +chombo_ug -parfile=test_chombo_ug_2d.par

8.7.2 Using Chombo in an AMR configuration

The real motivation for integrating Chombo into FLASH is for its adaptive mesh which is patch-based rather than an oct-tree like Paramesh. It has several desirable features including:

• refinement can happen cell-by-cell rather than block-by-block.
• blocks (or patches in Chombo terminology) are not limited to a fixed number of cells.
• refinement jumps of greater than 2 can happen between neighboring blocks.

A major difference in the generated meshes is that there is no longer the concept of “LEAF” blocks which cover the global domain. This was useful in Paramesh because a union of the space covered by the children of a block cover is exactly equal to the space covered by the parent block. It is therefore possible to evolve solution only on leaf blocks. Such complete overlap is not guaranteed in Chombo, in fact it rarely happens. Therefore, to be compatible with Chombo, physics units must evolve a solution on all blocks. For reference, the LEAF definition in Flash.h for FLASH-Chombo application now means all blocks.

A FLASH application with Chombo uses the refinement criteria similar to the one described in Section 8.6.3. The slight modification is that the $i_Xi_Yi_Z$-th cell is tagged when $E_{i_Xi_Yi_Z}$ exceeds a threshold value. Recall that in the Paramesh, the entire block is tagged for refinement when the error estimator of any of cells within a block exceeds a given threshold. As a further extension to this cell tagging procedure we also tag all cells within a small radius of a given tagged cell. The tagging radius is set using the runtime parameter tagradius; a value of 1 means that 1 cell to the left and right of a tagged cell in each dimension is also tagged. Values below the default value of 2 may cause problems because of introduction of fine-coarse interfaces in regions of the domain where there are steep gradients. For example, a Sod problem does not converge in Riemann solver when the value is 0.

The runtime parameters are described below:

• lrefine_max Used in an identical way to Paramesh. It is still a unit-based parameter and not zero-based. Defaults to 1.

• igridsize,jgridsize,kgridsize The global number of cells on the coarsest level of the adaptive mesh. Varying igridsize gives the same level of control as the unused Paramesh parameters lrefine_min and nblockx. Defaults to 16.

• maxBlockSize The maximum number of cells along a single dimension of a block. Defaults to 16.

• BRMeshRefineBlockFactor The minimum number of cells along a single dimension of a block. See BRMeshRefine class in Chombo user-guide. Defaults to 8.

• BRMeshRefineFillRatio Overall grid efficiency to be generated. If this number is set low, the grids will tend to be larger and less filled with tags. If this number is set high, the grids will tend to be smaller and more filled with tags. See BRMeshRefine class in Chombo user-guide. Defaults to 0.75.

• BRMeshRefineBufferSize Proper nesting buffer size. This will be the minimum number of level l cells between any level l+1 cell and a level l-1 cell. See BRMeshRefine class in Chombo user-guide. Defaults to 1.

• verbosity The verbosity of debugging output written to local log files - higher values mean progressively more debugging output. As a rough guide 0 means no debugging output, 2 means debugging output from FLASH objects, and values above 2 add debugging output from Chombo objects. Defaults to 0.

• refRatio The refinement jump between fine-coarse levels, typically 2, 4 or 8. Defaults to 2.

• restrictBeforeGhostExchange Invokes a full mesh restriction before guard cells are exchanged. This is to make the guard cell exchange behave the same way as Paramesh guard cell exchange. We do not know yet if it is necessary. Defaults to True.

• tagRadius The radius of cells around a tagged cell that should also be tagged. Defaults to 2.

Set convertToConsvdForMeshCalls runtime parameter to .true. to interpolate mass specific solution variables (see Section 8.6.2.1). We have not yet added code for convertToConsvdInMeshInterp runtime parameter and so it should not be used.

Setup a simulation using:

 

./setup Sedov -auto +chombo_amr -parfile=test_chombo_amr_2d.par

The GridParticles and GridSolvers sub-unit do not interoperate with Chombo yet.