37.1 Introduction

Drift is a debugging tool added to FLASH to help catch programming mistakes that occur while refactoring code in a way that should not change numerical behavior. Historically, simulation checkpoints have been used to verify that results obtained after a code modification have not affected the numerics. But if changes are observed, then the best a developer can do to narrow the bug hunting search space is to look at pairs of checkpoint files from the two different code bases sequentially. The first pair to compare unequal will tell you that somewhere between that checkpoint and its immediate predecessor something in the code changed the numerics. Therefor, the search space can only be narrowed to the limit allow by the checkpointing interval, which in FLASH, without clever calls to IO sprinkled about, is at best once per time cycle.

Drift aims to refine that granularity considerably by allowing comparisons to be made upon every modification to a block's contents. To achieve this, drift intercepts calls to Grid_releaseBlkPtr, and inserts into them a step to checksum each of the variables stored on the block. Any checksums that do not match with respect to the last checksums recorded for that block are logged to a text file along with the source file and line number. The developer can then compare two drift logs generated by the different runs using diff to find the first log entry that generates unequal checksums, thus telling the developer which call to Grid_releaseBlkPtr first witnessed divergent values.

The following are example excerpts from two drift logs. Notice the checksum value has changed for variable dens on block 18. This should clue the developer in that the cause of divergent behavior lies somewhere between Eos_wrapped.F90:249 and hy_ppm_sweep.F90:533.

inst=2036 step=1 src=Eos_wrapped.F90:249 blk=57 dens E8366F6E49DD1B44 eint 89D635E5F46E4CE4 ener C6ED4F02E60C9E8F pres 6434628E2D2E24E1 temp DB675D5AFF7D48B8 velx 42546C82E30F08B3 inst=2100 step=1 src=hy_ppm_sweep.F90:533 blk=18 dens A462F49FFC3112DE eint 9CD79B2E504C7C7E ener 4A3E03520C3536B9 velx 8193E8C2691A0725 vely 86C5305CB7DE275E

inst=2036 step=1 src=Eos_wrapped.F90:249 blk=57 dens E8366F6E49DD1B44 eint 89D635E5F46E4CE4 ener C6ED4F02E60C9E8F pres 6434628E2D2E24E1 temp DB675D5AFF7D48B8 velx 42546C82E30F08B3 inst=2100 step=1 src=hy_ppm_sweep.F90:533 blk=18 dens 5E52D67C5E93FFF1 eint 9CD79B2E504C7C7E ener 4A3E03520C3536B9 velx 8193E8C2691A0725 vely 86C5305CB7DE275E