8.8 GridMain Usage

The Grid unit has the largest API of all units, since it is the custodian of the bulk of the simulation data, and is responsible for most of the code housekeeping. The Grid_init routine, like all other Unit_init routines, collects the runtime parameters needed by the unit and stores values in the data module. If using UG, the Grid_init also creates the computational domain and the housekeeping data structures and initializes them. If using AMR, the computational domain is created by the Grid_initDomain routine, which also makes a call to mesh package's own initialization routine. The physical variables are all owned by the Grid unit, and it initializes them by calling the Simulation_initBlock routine which applies the specified initial conditions to the domain. If using an adaptive grid, the initialization routine also goes through a few refinement iterations to bring the grid to desired initial resolution, and then applies the Eos function to bring all simulation variables to thermodynamic equilibrium. Even though the mesh-based variables are under Grid's control, all the physics units can operate on and modify them.

A suite of get/put accessor/mutator functions allows the calling unit to fetch or send data by the block. One option is to get a pointer Grid_getBlkPtr, which gives unrestricted access to the whole block and the client unit can modify the data as needed. The more conservative but slower option is to get some portion of the block data, make a local copy, operate on and modify the local copy and then send the data back through the “put” functions. The Grid interface allows the client units to fetch the whole block (Grid_getBlkData), a partial or full plane from a block (Grid_getPlaneData), a partial or full row (Grid_getRowData), or a single point (Grid_getPointData). Corresponding “put” functions allow the data to be sent back to the Grid unit after the calling routine has operated on it. Various getData functions can also be used to fetch some derived quantities such as the cell volume or face area of individual cells or groups of cells. There are several other accessor functions available to query the housekeeping information from the grid. For example Grid_getListOfBlocks returns a list of blocks that meet the specified criterion such as being “LEAF” blocks in PARAMESH, or residing on the physical boundary.

FLASH3 Transition: The Grid_getBlkData and Grid_putBlkData functions replace the dBaseGetData and dBasePutData functions in FLASH2. The bulk of the dBase functionality from FLASH2 is now handled by the Grid unit. For example, the global mesh data structures “unk” and “tree” now belong to Grid, and any information about them is queried from it. However, dBase and Grid are not identical. dBase was a central storage for all data, whereas in FLASH4 some of the data, such as simulation parameters like dt and simulation time are owned by the Driver unit instead of the Grid unit.

FLASH3 Transition: In FLASH2, variables $nxb$, $nyb$ and $nzb$ traditionally described blocksize. From FLASH3 on, the symbols NXB, NYB, and NYB are intended not to be used directly except in the Grid unit itself. The variables like iHi_gc and iHi etc.  that marked the endpoints of the blocks were replaced in FLASH3 with in their new incarnation (GRID_IHI_GC etc. ) Again, these new variables are used only in sizing arrays in declaration headers. Even when they are used for array sizing, they are enclosed in preprocessor blocks that can be eliminated at compile time. This separation was done to compartmentalize the FLASH3 code. The grid layout is not required to be known in any other unit (with the exception of IO). FLASH3 provides an API function Grid_getBlkIndexLimits that fetches the block endpoints from the Grid unit for each individual block. The fetched values are then used as do loop end points in all $\left< i,j,k \right>$ loops in all the client units. When working in NONFIXEDBLOCKSIZE mode, the same fetched values are also used to size and allocate arrays. We have retained GRID_IHI_GC etc.  for array sizing as compile time option for performance reasons. Statically allocated arrays allow better compiler optimization.

In addition to the functions to access the data, the Grid unit also provides a collection of routines that drive some housekeeping functions of the grid without explicitly fetching any data. A good example of such routines is Grid_fillGuardCells. Here no data transaction takes place between Grid and the calling unit. The calling unit simply instructs the Grid unit that it is ready for the guard cells to be updated, and doesn't concern itself with the details. The Grid_fillGuardCells routine makes sure that all the blocks get the right data in their guardcells from their neighbors, whether they are at the same, lower or higher resolution, and if instructed by the calling routine, also ensures that EOS is applied to them.

In large-scale, highly parallel FLASH simulations with AMR, the processing of Grid_fillGuardCells calls may take up a significant part of available resource like CPU time, communication bandwidth, and buffer space. It can therefore be important to optimize these calls in particular. From FLASH3.1, Grid_fillGuardCells provides ways to

• operate on only a subset of the variables in unk (and facevarx, facevary, and facevarz), by masking out other variables;
• fill only some the nguard layers of guard cells that surround the interior of a block (while possibly excepting a “sweep” direction);
• combine guard cell filling with EOS calls (which often follow guard cell exchanges in the normal flow of execution of a simulation in order to ensure thermodynamical consistency in all cells, and which may also be very expensive), by letting Grid_fillGuardCells make the calls on cells where necessary;
• automatically determine masks and whether to call EOS, based on the set of variables that the calling code actually needs updated. by masking out other variables.

These options are controlled by OPTIONAL arguments, see Grid_fillGuardCells for documentation. When these optional arguments are absent or when a Grid implementation does not support them, FLASH falls back to safe default behavior which may, however, be needlessly expensive.

Another routine that may change the global state of the grid is Grid_updateRefinement. This function is called when the client unit wishes to update the grid's resolution. again, the calling unit does not need to know any of the details of the refinement process.

FLASH3 Transition: As mentioned in Chp:Architecture, FLASH allows every unit to identify scalar variables for checkpointing. In the Grid unit, the function that takes care of consolidating user specified checkpoint variable is Grid_sendOutputData. Users can select their own variables to checkpoint by including an implementation of this function specific to their requirements in their Simulation setup directory.