4.2 Unit Architecture

A FLASH unit defines its own Application Programming Interface (API), which is a collection of routines the unit exposes to other units in the code. A unit API is usually a mix of accessor functions and routines which modify the state of the simulation.

A good example to examine is the Grid unit API. Some of the accessor functions in this unit are Grid_getCellCoords, Grid_getBlkData, and Grid_putBlkData, while Grid_fillGuardCells and
Grid_updateRefinement are examples of API routines which modify data in the Grid unit.

A unit can have more than one implementation of its API. The Grid Unit, for example, has both an Adaptive Grid and a Uniform Grid implementation. Although the implementations are different, they both conform to the Grid API, and therefore appear the same to the outside units. This feature allows users to easily swap various unit implementations in and out of a simulation without affecting they way other units communicate. Code does not have to be rewritten if the users decides to implement the uniform grid instead of the adaptive grid.

4.2.1 Stub Implementations

Since routines can have multiple implementations, the setup script must select the appropriate implementation for an application. The selection follows a few simple rules described in Architecture Tips. The top directory of every unit contains a stub or null implementation of each routine in the Unit's API. The stub functions essentially do nothing. They are coded with just the declarations to provide the same interface to callers as a corresponding “real” implementation. They act as function prototypes for the unit. Unlike true prototypes, however, the stub functions assign default values to the output-only arguments, while leaving the other arguments unaltered. The following snippet shows a simplified example of a stub implementation for the routine Grid_getListOfBlocks.

 

subroutine Grid_getListOfBlocks(blockType, listOfBlocks, count)


implicit none


integer, intent(in) :: blockType
  integer,dimension(*),intent(out) :: listOfBlocks
  integer, intent(out) :: count


count=0
  listOfBlocks(1)=0


return
end subroutine Grid_getListOfBlocks

While a set of null implementation routines at the top level of a unit may seem like an unnecessary added layer, this arrangement allows FLASH to include or exclude units without the need to modify any existing code. If a unit is not included in a simulation, the application will be built with its stub functions. Similarly, if a specific implementation of the unit finds some of the API functions irrelevant, it need not provide any implementations for them. In those situations, the applications include stubs for the unimplemented functions, and full implementations of all the other ones. Since the stub functions do return valid values when called, unexpected crashes from un-initialized output arguments are avoided.

The Grid_updateRefinement routine is a good example of how stub functions can be useful. In the case of a simulation using an adaptive grid, such as PARAMESH, the routine Driver_evolveFlash calls Grid_updateRefinement to update the grid's spacing. The Uniform Grid however, needs no such routine because its grid is fixed. There is no error, however, when Driver_evolveFlash calls Grid_updateRefinement during a Uniform Grid simulation, because the stub routine steps in and simply returns without doing anything. Thus the stub layer allows the same Driver_evolveFlash routine to work with both the Adaptive Grid and Uniform Grid implementations.

FLASH3 Transition: While the concept of “null” or “stub” functions existed in FLASH2, FLASH3 formalized it by requiring all units to publish their API (the complete Public Interface) at the top level of a unit's directory. Similarly, the inheritance through Unix directory structure in FLASH4 is essentially the same as that of FLASH2, but the introduction of a formal naming convention has clarified it and made it easier to follow. The complete API can be found online at
http://flash.rochester.edu/site/flashcode/user_support/.

4.2.2 Subunits

One or more subunits sit under the top level of a unit. Among them the unit's complete API is implemented. The subunits are considered peers to one another. Each subunit must implement at least one API function, and no two subunits can implement the same API function. The division of a unit into subunits is based upon identifying self-contained subsets of its API. In some instances, a subunit may be completely excluded from a simulation, thereby saving computational resources. For example, the Grid unit API includes a few functions that are specific to Lagrangian tracer particles, and are therefore unnecessary to simulations that do not utilize particles. By placing these routines in the GridParticles subunit, it is possible to easily exclude them from a simulation. The subunits have composite names; the first part is the unit name, and the second part represents the functionality that the subunit implements. The primary subunit is named UnitMain, which every unit must have. For example, the main subunit of Hydro unit is HydroMain and that of the Eos unit is EosMain.

In addition to the subunits, the top level unit directory may contain a subdirectory called localAPI. This subdirectory allows a subunit to create a public interface to other subunits within its own unit; all stub implementations of the subunit public interfaces are placed in localAPI. External units should not call routines listed in the localAPI; for this reason these local interfaces are not shown in the general source API tree.

A subunit can have a hierarchy of its own. It may have more than one unit implementation directories with alternative implementations of some of its functions while other functions may be common between them. FLASH exploits the inheritance rules described in Architecture Tips. For example, the Grid unit has three implementations for GridMain: the Uniform Grid (UG), PARAMESH 2, and PARAMESH 4. The procedures to apply boundary conditions are common to all three implementations, and are therefore placed directly in GridMain. In addition, GridMain contains two subdirectories. One is UG, which has all the remaining implementations of the API specific to the Uniform Grid. The other directory is organized as paramesh, which in turn contains two directories for the package of PARAMESH 2 and another organizational directory paramesh4. Finally, paramesh4 has two subdirectories with alternative implementations of the PARAMESH 4 package. The directory paramesh also contains all the function implementations that are common between PARAMESH 2 and PARAMESH 4. Following the naming convention described in Architecture Tips, paramesh is all lowercase, since it has child directories that have some API implementation. The namespace directories Paramesh2, Paramesh4.0 andParamesh4dev contain functions unique to each implementation. An example of a unit hierarchy is shown in Figure 4.1. The kernels are described below in Sec:privateRoutines.

Figure 4.1: The unit hierarchy and inheritance.

4.2.3 Unit Data Modules, _init, and _finalize routines

Each unit must have a F90 data module to store its unit-scope local data and an Unit_init file to initialize it. The Unit_init routines are called by the Driver unit once by the routine Driver_initFlash at the start of a simulation. They get unit specific runtime parameters from the RuntimeParameters unit and store them in the unit data module.

Every unit implementation directory of UnitMain, must either inherit a Unit_data module, or have its own. There is no restriction on additional unit scope data modules, and individual Units determine how best to manage their data. Other subunits and the underlying computational kernels can have their own data modules, but the developers are encouraged to keep these data modules local to their subunits and kernels for clarity and maintainability of the code. It is strongly recommended that only the data modules in the Main subunit be accessible everywhere in the unit. However, no data module of a unit may be known to any other unit. This restriction is imposed to keep the units encapsulated and their data private. If another part of the code needs access to any of the unit data, it must do so through accessor functions.

Additionally, when routines use data from the unit's data module the convention is to indicate what particular data is being used with the ONLY keyword, as in use Unit_data, ONLY : un_someData. See the snippet of code below for the correct convention for using data from a unit's FORTRAN Data Module.  

subroutine Driver_evolveFlash()


use Driver_data, ONLY: dr_myPE, dr_numProcs, dr_nbegin, &
       dr_nend, dr_dt, dr_wallClockTimeLimit, &
       dr_tmax, dr_simTime, dr_redshift, &
       dr_nstep, dr_dtOld, dr_dtNew, dr_restart, dr_elapsedWCTime


implicit none


integer   :: localNumBlocks

Each unit must also have a Unit_finalize routine to clean up the unit at the termination of a FLASH run. The finalization routines might deallocate space or write out completion messages.

4.2.4 Private Routines: kernels and helpers

All routines in a unit that do not implement the API are classified as private routines. They are divided into two broad categories: the kernel is the collection of routines that implement the unit's core functionality and solvers, and helper routines are supplemental to the unit's API and sometimes act as a conduit to its kernel. A helper function is allowed to know the other unit's APIs but is itself known only locally within the unit. The concept of helper functions allows minimization of the unit APIs, which assists in code maintenance. The helper functions follow the convention of starting with an “un_” in their name, where “un” is in some way derived from the unit name. For example, the helper functions of the Grid unit start with gr_, and those of Hydro unit start with hy_. The helper functions have access to the unit's data module, and they are also allowed to query other units for the information needed by the kernel, by using their accessor functions. If the kernel has very specific data structures, the helper functions can also populate them with the collected information. An example of a helper function is gr_expandDomain, which refines an AMR block. After refinement, equations of state usually need to be called, so the routine accesses the EOS routines via Eos_wrapped.

The concept of kernels, on the other hand, facilitates easy import of third party solvers and software into FLASH. The kernels are not required to follow either the naming convention or the inheritance rules of the FLASH architecture. They can have their own hierarchy and data modules, and the top level of the kernel typically resides at leaf level of the FLASH unit hierarchy. This arrangement allows FLASH to import a solver without having to modify its internal code, since API and helper functions hide the higher level details from it, and hide its details from other units. However, developers are encouraged to follow the helper function naming convention in the kernel where possible to ease code maintenance.

The Grid unit and the Hydro unit both provide very good examples of private routines that are clearly distinguishable between helper functions and kernel. The AMR version of the Grid unit imports the PARAMESH version 2 library as a vendor supplied branch in our repository. It sits under the lowest namespace directory Paramesh2 in Grid hierarchy and maintains the library's original structure. All other private functions in the paramesh branch of Grid are helper functions and their names start with gr_. In the Hydro unit the entire hydrodynamic solver resides under the directory PPM, which was imported from the PROMETHEUS code (see Sec:PPM algorithm). PPM is a directional solver and requires that data be passed to it in vector form. Routines like hy_sweep and hy_block are helper functions that collect data from the Grid unit, and put it in the format required by PPM. These routines also make sure that data stay in thermodynamic equilibrium through calls to the Eos unit. Neither PARAMESH 2, nor PPM has any knowledge of units outside their own.