9.2 Output Files

The IO unit can output 4 different types of files: checkpoint files, plotfiles, particle files and flash.dat, a text file holding the integrated grid quantities. FLASH also outputs a logfile, but this file is controlled by the Logfile Unit; see Chp:Logfile Unit for a description of that format.

There are a number of runtime parameters that are used to control the output and frequency of IO files. A list of all the runtime parameters and their descriptions for the IO unit can be found online all of them. Additional description is located in Table 9.2 for checkpoint parameters, Table 9.3 for plotfile parameters, Table 9.4 for particle file parameters, Table 9.5 for flash.dat parameters, and Table 9.6 for genereal IO parameters.

9.2.1 Checkpoint files - Restarting a Simulation

Checkpoint files are used to restart a simulation. In a typical production run, a simulation can be interrupted for a number of reasons-- e.g., if the machine crashes, the present queue window closes, the machine runs out of disk space, or perhaps (gasp) there is a bug in FLASH. Once the problem is fixed, a simulation can be restarted from the last checkpoint file rather than the beginning of the run. A checkpoint file contains all the information needed to restart the simulation. The data is stored at full precision of the code (8-byte reals) and includes all of the variables, species, grid reconstruction data, scalar values, as well as meta-data about the run.

The API routine for writing a checkpoint file is IO_writeCheckpoint. Users usually will not need to call this routine directly because the FLASH IO unit calls IO_writeCheckpoint from the routine IO_output which checks the runtime parameters to see if it is appropriate to write a checkpoint file at this time. There are a number of ways to get FLASH to produce a checkpoint file for restarting. Within the flash.par, runtime parameters can be set to dump output. A checkpoint file can be dumped based on elapsed simulation time, elapsed wall clock time or the number of timesteps advanced. A checkpoint file is also produced when the simulation ends, when the max simulation time tmax, the minimum cosmological redshift, or the total number of steps nend has been reached. A user can force a dump to a checkpoint file at another time by creating a file named .dump_checkpoint in the output directory of the master processor. This manual action causes FLASH to write a checkpoint in the next timestep. Checkpoint files will continue to be dumped after every timestep as long as the code finds a .dump_checkpoint file in the output directory, so the user must remember to remove the file once all the desired checkpoint files have been dumped. Creating a file named .dump_restart in the output directory will cause FLASH to output a checkpoint file and then stop the simulation. This technique is useful for producing one last checkpoint file to save time evolution since the last checkpoint, if the machine is going down or a queue window is about to end. These different methods can be combined without problems. Each counter (number of timesteps between last checkpoint, amount of simulation time single last checkpoint, the change in cosmological redshift, and the amount of wall clock time elapsed since the last checkpoint) is independent of the others, and are not influenced by the use of a .dump_checkpoint or .dump_restart.

Runtime Parameters used to control checkpoint file output include:

Table 9.2: Checkpoint IO parameters.

Parameter	Type	Default value	Description
checkpointFileNumber	INTEGER	0	The number of the initial checkpoint file. This number is appended to the end of the filename and incremented at each subsequent output. When restarting a simulation, this indicates which checkpoint file to use.
checkpointFileIntervalStep	INTEGER	0	The number of timesteps desired between subsequent checkpoint files.
checkpointFileIntervalTime	REAL	1.	The amount of simulation time desired between subsequent checkpoint files.
checkpointFileIntervalZ	REAL	HUGE(1.)	The amount of cosmological redshift change that is desired between subsequent checkpoint files.
rolling_checkpoint	INTEGER	10000	The number of checkpoint files to keep available at any point in the simulation. If a checkpoint number is greater than rolling_checkpoint, then the checkpoint number is reset to 0. There will be at most rolling_checkpoint checkpoint files kept. This parameter is intended to be used when disk space is at a premium.
wall_clock_checkpoint	REAL	43200.	The maximum amount of wall clock time (seconds) to elapse between checkpoints. When the simulation is started, the current time is stored. If wall_clock_checkpoint seconds elapse over the course of the simulation, a checkpoint file is stored. This is useful for ensuring that a checkpoint file is produced before a queue closes.
restart	BOOLEAN	.false.	A logical variable indicating whether the simulation is restarting from a checkpoint file (.true.) or starting from scratch (.false.).
appendParRestart	BOOLEAN	.false.	A logical variable indicating whether the parameter file is automatically appended with the appropriate values of restart, checkpointFileNumber and plotFileNumber after a checkpoint file is written (except for the first checkpoint file).

FLASH is capable of restarting from any of the checkpoint files it produces. The user should make sure that the checkpoint file is valid (e.g., the code did not stop while outputting). To tell FLASH to restart, set the restart runtime parameter to .true. in the flash.par. Also, set checkpointFileNumber to the number of the file from which you wish to restart. If plotfiles or particle files are being produced set plotfileNumber and particleFileNumber to the number of the next plotfile and particle file you want FLASH to output. In FLASH4 plotfiles and particle file outputs are forced whenever a checkpoint file is written. Sometimes several plotfiles may be produced after the last valid checkpoint file. Resetting plotfileNumber to the first plotfile produced after the checkpoint from which you are restarting will ensure that there are no gaps in the output. See Sec:Plotfiles for more details on plotfiles.

9.2.2 Plotfiles

A plotfile contains all the information needed to interpret the grid data maintained by FLASH. The data in plotfiles, including the grid metadata such as coordinates and block sizes, are stored at single precision to preserve space. This can, however, be overridden by setting the runtime parameters plotfileMetadataDP and/or plotfileGridQuantityDP to true to set the grid metadata and the quantities stored on the grid (dens, pres, temp, etc.) to use double precision, respectively. Users must choose which variables to output with the runtime parameters plot_var_1, plot_var_2, etc., by setting them in the flash.par file. For example:  

plot_var_1 = "dens"
plot_var_2 = "pres"

Currently, we support a number of plotvars named plot_var_n up to the number of UNKVARS in a given simulation. Similarly, scratch variables may be output to plot files lbl:OutputScratchVariables. At this time, the plotting of face centered quantities is not supported.

FLASH3 Transition: In FLASH2 a few variables like density and pressure were output to the plotfiles by default. Because FLASH4 supports a wider range of simulations, it makes no assumptions that density or pressure variables are even included in the simulation. In FLASH4 a user must define plotfile variables in the flash.par file, otherwise the plotfiles will not contain any variables.

The interface for writing a plotfile is the routine IO_writePlotfile. As with checkpoint files, the user will not need to call this routine directly because it is invoked indirectly through calling IO_output when, based on runtime parameters, FLASH4 needs to write a plotfile. FLASH can produce plotfiles in much the same manner as it does with checkpoint files. They can be dumped based on elapsed simulation time, on steps since the last plotfile dump or by forcing a plotfile to be written by hand by creating a.dump_plotfile in the output directory. A plotfile will also be written at the termination of a simulation as well.

If plotfiles are being kept at particular intervals (such as time intervals) for purposes such as visualization or analysis, it is also possible to have FLASH denote a plotfile as “forced". This designation places the word forced between the basename and the file format type identifier (or the split number if splitting is used). These files are numbered separately from normal plotfiles. By default, plotfiles are considered forced if output for any reason other than the change in simulation time, change in cosmological redshift, change in step number, or the termination of a simulation from reaching nend , zFinal, or tmax. This option can be disabled by setting ignoreForcedPlot to true in a simulations flash.par file. The following runtime parameters pertain to controlling plotfiles:

Table 9.3: Plotfile IO parameters.

Parameter	Type	Default value	Description
plotFileNumber	INTEGER	0	The number of the starting (or restarting) plotfile. This number is appended to the filename.
plotFileIntervalTime	REAL	1.	The amount of simulation time desired between subsequent plotfiles.
plotFileIntervalStep	INTEGER	0	The number of timesteps desired between subsequent plotfiles.
plotFileIntervalZ	INTEGER	HUGE(1.)	The change in cosmological redshift desired between subsequent plotfiles.
plot_var_1, ..., plot_var_n	STRING	"none"	Name of the variables to store in a plotfile. Up to 12 variables can be selected for storage, and the standard 4-character variable name can be used to select them.
ignoreForcedPlot	BOOLEAN	.false.	A logical variable indicating whether or not to denote certain plotfiles as forced.
forcedPlotfileNumber	INTEGER	0	An integer that sets the starting number for a forced plotfile.
plotfileMetadataDP	BOOLEAN	.false.	A logical variable indicating whether or or not to output the normally single-precision grid metadata fields as double precision in plotfiles. This specifically affects coordinates, block size, and bounding box.
plotfileGridQuantityDP	BOOLEAN	.false.	A logical variable that sets whether or not quantities stored on the grid, such as those stored in unk, are output in single precision or double precision in plotfiles.

9.2.3 Particle files

When Lagrangian particles are included in a simulation, the ParticleIO subunit controls input and output of the particle information. The particle files are stored in double precision. Particle data is written to the checkpoint file in order to restart the simulation, but is not written to plotfiles. Hence analysis and metadata about particles is also written to the particle files. The particle files are intended for more frequent dumps. The interface for writing the particle file is IO_writeParticles. Again the user will not usually call this function directly because the routine IO_output controls particle output based on the runtime parameters controlling particle files. They are controlled in much of the same way as the plotfiles or checkpoint files and can be dumped based on elapsed simulation time, on steps since the last particle dump or by forcing a particle file to be written by hand by creating a .dump_particle_file in the output directory. The following runtime parameters pertain to controlling particle files:

Table 9.4: Particle File IO runtime parameters.

Parameter	Type	Default value	Description
particleFileNumber	INTEGER	0	The number of the starting (or restarting) particle file. This number is appended to the end of the filename.
particleFileIntervalTime	REAL	1.	The amount of simulation time desired between subsequent particle file dumps.
particleFileIntervalStep	INTEGER	0	The number of timesteps desired between subsequent particle file dumps.
particleFileIntervalZ	REAL	HUGE(1.)	The change in cosmological redshift desired between subsequent particle file dumps.

FLASH3 Transition: From FLASH3 on each particle dump is written to a separate file. In FLASH2 the particles data structure was broken up into real and integer parts, where as in FLASH3 all particle properties are real values. See Sec:Output formats and Chp:Particles for more information about the particles data structure in FLASH4. Additionally, filtered particles are not implemented in FLASH4.

All the code necessary to output particle data is contained in the IO subunit called IOParticles. Whenever the Particles unit is included in a simulation the correct IOParticles subunit will also be included. For example as setup:  

./setup IsentropicVortex -2d -auto -unit=Particles +ug

will include the IO unit IO/IOMain/hdf5/serial/UG and the correct IOParticles subunit
IO/IOParticles/hdf5/serial/UG. The shortcuts +parallelio, +pnetcdf, +ug will also cause the setup script to pick up the correct IOParticles subunit as long as a Particles unit is included in the simulation.

9.2.4 Integrated Grid Quantities - flash.dat

At each simulation time step, values which represent the overall state (e.g., total energy and momentum) are computed by calculating over all cells in the computations domain. These integral quantities are written to the ASCI file flash.dat. A default routine IO_writeIntegralQuantities is provided to output standard measures for hydrodynamic simulations. The user should copy and modify the routine IO_writeIntegralQuantities into a given simulation directory to store any quantities other than the default values. Two runtime parameters pertaining to the flash.dat file are listed in the table below.

Table 9.5: flash.dat runtime parameters.

Parameter	Type	Default value	Description
stats_file	STRING	"flash.dat"	Name of the file to which the integral quantities are written.
wr_integrals_freq	INTEGER	1	The number of timesteps to elapse between outputs to the scalar/integral data file (flash.dat)

9.2.5 General Runtime Parameters

There are several runtime parameters that pertain to the general IO unit or multiple output files rather than one particular output file. They are listed in the table below.

Table 9.6: General IO runtime parameters.

Parameter	Type	Default value	Description
basenm	STRING	"flash_"	The main part of the output filenames. The full filename consists of the base name, a series of three-character abbreviations indicating whether it is a plotfile, particle file or checkpoint file, the file format, and a 4-digit file number. See Sec:Output file names for a description of how FLASH output files are named.
output_directory	STRING	""	Output directory for plotfiles, particle files and checkpoint files. The default is the directory in which the executable sits. output_directory can be an absolute or relative path.
memory_stat_freq	INTEGER	100000	The number of timesteps to elapse between memory statistic dumps to the log file (flash.log).
useCollectiveHDF5	BOOLEAN	.true.	When using the parallel HDF5 implementation of IO, will enable collective mode for HDF5.
summaryOutputOnly	BOOLEAN	.false.	When set to .true. write an integrated grid quantities file only. Checkpoint, plot and particle files are not written unless the user creates a .dump_plotfile, .dump_checkpoint, .dump_restart or .dump_particle file.