8.2 GridMain Data Structures

The Grid unit is the most extensive infrastructure unit in the FLASH code, and it owns data that most other units wish to fetch and modify. Since the data layout in this unit has implications on the manageability and performance of the code, we describe it in some detail here.

FLASH can be run with a grid discretization that assumes cell-centered data, face-centered data, or a combination of the two. Paramesh and Uniform Grid store physical data in multidimensional F90 arrays; cell-centered variables in unk, short for “unknowns”, and face-centered variables in arrays called facevarx, facevary, and facevarz, which contain the face-centered data along the $ x$, $ y$, and $ z$ dimensions, respectively. The cell-centered array unk is dimensioned as array(NUNK_VARS,nxb,nyb,nzb,blocks), where nxb, nyb, nzb are the spatial dimensions of a single block, and blocks is the number of blocks per processor (MAXBLOCKS for PARAMESH and 1 for UG). The face-centered arrays have one extra data point along the dimension they are representing, for example facevarx is dimensioned as array(NFACE_VARS,nxb+1,nyb,nzb,blocks). Some or all of the actual values dimensioning these arrays are determined at application setup time. The number of variables and the value of MAXBLOCKS are always determined at setup time. The spatial dimensions nxb,nyb,nzb can either be fixed at setup time, or they may be determined at runtime. These two modes are referred to as FIXEDBLOCKSIZE and NONFIXEDBLOCKSIZE. Chombo, which is written in C++, maintains its internal data very differently. However, through the wrapper layer provided by Grid, the Chombo data structures mimic the data layout understood by the FLASH solvers. Details of Grid implementation with Chombo are described in section Sec:chombo

All values determined at setup time are defined as constants in a file Flash.h generated by the setup tool. This file contains all application-specific global constants such as the number and naming of physical variables, number and naming of fluxes and species, etc.; it is described in detail in Chp:Flash.h.

For cell-centered variables, the Grid unit also stores a variable type that can be retrieved using the Simulation_getVarnameType routine; see Sec:ConfigFileSyntax for the syntax and meaning of the optional TYPE attribute that can be specified as part of a VARIABLE definition read by the setup tool.

In addition to the primary physical variables, the Grid unit has another set of data structures for storing auxiliary fluid variables. This set of data structures provides a mechanism for storing such variables whose spatial scope is the entire physical domain, but who do not need to maintain their guard cells updated at all times. The data structures in this set include: SCRATCHCENTERVAR, which has the same shape as the cell centered variables data structure; and SCRATCHFACEXVAR, SCRATCHFACEYVAR and SCRATCHFACEZVAR, which have the same shape as the corresponding face centered variables data structures. Early releases of FLASH3 had SCRATCHVAR, dimensioned array(NSCRATCH_GRID_VARS,nxb+1,nyb+1,nzb+1,blocks), as the only grid scope scratch data structure. For reasons of backward compatibility, and to maximize reusability of space, SCRATCHVAR continues to exist as a supported data structure, though its use is deprecated. The datastructures for face variables, though supported, are not used anywhere in the released code base. The unsplit MHD solver StaggeredMesh discussed in Sec:usm_algorithm gives an example of the use of some of these data structures. It is important to note that there is no guardcell filling for the scratch variables, and the values in the scratch variables become invalid after a grid refinement step. While users can define scratch variables to be written to the plotfiles, they are not by default written to checkpoint files. The Grid unit also stores the metadata necessary for work distribution, load balancing, and other housekeeping activities. These activities are further discussed in Sec:Grid UG and Sec:Grid AMR, which describe individual implementations of the Grid unit.