[FLASH-USERS] How to generate big checkpoint files in FLASH

[Christoph Federrath]

Hi Zheng,

the Sedov test is not really a good test to check file compression. As pointed out earlier, there are large patches that have the same values of density, velocity, etc., so that will compress well. In contrast, a fully-developed turbulent field of density, velocity, pressure, etc will be very hard to compress. I recommend to use the turbulence stirring unit to create FLASH output files with fully-developed turbulence and try your compression algorithm on that. You can simply look into Section 16.3 in the user guide and specifically into 16.3.2-16.3.4, which contains a unit test setup that you can modify to create high-resolution (big) output files. I’m also happy to make some of my own existing checkpoint files available—the biggest files that I’m currently producing with this setup are 19 TB from a simulation with 10000^3 grid cells.

Kind regards
Christoph

—
Christoph Federrath
http://www.mso.anu.edu.au/~chfeder <http://www.mso.anu.edu.au/~chfeder>

> On 3 Mar 2016, at 05:14, Zheng Yuan <zhengyuan2014 at u.northwestern.edu <mailto:zhengyuan2014 at u.northwestern.edu>> wrote:
> 
> yes... I run FLASH using 32*32 dim for 200 iterations. The compression ratios of the first 20 iterations are extreme high. How much time should I simulate in order to get the 'fully developed turbulence field'? (My major is computer science. Sorry for this naive question..)
> 
> Best
> Zheng
> 
> On 3/2/16 11:56 AM, Dean Townsley wrote:
>> Zheng,
>> 
>> Also, if you are testing compression algorithms, you should consider the physical state of the data you are compressing.
>> 
>> The initial conditions or state after only a few time steps of something like the Sod problem or Sedov will be very easily compressible because there are large regions on the grid that are in a spatially uniform state.  Conversely, a fully developed turbulence field (e.g. StirTurb run for a decent fraction of a turnover time) is likely to be harder to compress.
>> 
>> Dean
>> 
>> On 03/02/2016 11:47 AM, Zheng Yuan wrote:
>>> Thanks Klaus! I will use 3D instead of 2D. Someone also suggested that larger values of lrefine_min & lrefine_max results in larger checkpoint file sizes. I will use 3D & larger Irefine_min&max, while keeping nxb*nyb*nzb = 8*8*8.
>>> 
>>> Best
>>> Zheng
>>> 
>>> On 3/2/16 11:23 AM, Klaus Weide wrote:
>>>> On Tue, 1 Mar 2016, Zheng Yuan wrote:
>>>> 
>>>>> Dear all,
>>>>> 
>>>>> My goal is to run FLASH using 1K MPI processes and get large checkpoint files
>>>>> (500 GB each. These large checkpoint files will be used as the test data of a
>>>>> parallel data compression algorithm proposed by our group.). Does anybody know
>>>>> how to set the parameters to run FLASH in large scale and get large checkpoint
>>>>> files?
>>>> Hello,
>>>> 
>>>> If you really "just" want large checkpoint files, you should start with a 3D
>>>> setup!
>>>> 
>>>> 
>>>>> Currently, I am trying to run FLASH/Sedov to generate large checkpoint files.
>>>>> However, FLASH reports an error when I increase the dimension of the grid.
>>>>> 
>>>>> I configured FLASH using:
>>>>> 
>>>>> ./setup Sedov -auto +pnetcdf -objdir=sedov_2d -parfile=sedov_io_69b_2d.par -2d
>>>>> -nxb=64 -nyb=64 -maxblocks=200
>>>>> 
>>>>> To run more iterations, I changed the 'nend' parameter to 10 in
>>>>> sedov_2d/flash.par and make sedov_2d
>>>>> 
>>>>> I run flash using command:
>>>>> 
>>>>> mpirun -n 4 ./flash4
>>>>> 
>>>>> 
>>>>> FLASH terminate at the third iteration. The error is:
>>>>  *** Wrote plotfile to sedov_2d_6lev_ncmpi_plt_cnt_0001 ****
>>>>        n          t         dt  (         x, y,          z) |  dt_hydro
>>>>        1 2.0000E-10 2.5000E-05  ( 4.993E-01,  4.875E-01, 0.000E+00) |  1.265E-05
>>>>  *** Wrote checkpoint file to sedov_2d_6lev_ncmpi_chk_0001 ****
>>>>  *** Wrote plotfile to sedov_2d_6lev_ncmpi_plt_cnt_0002 ****
>>>>        2 5.0000E-05 2.5000E-05  ( 5.134E-01,  5.017E-01, 0.000E+00) |  8.673E-06
>>>>  *** Wrote checkpoint file to sedov_2d_6lev_ncmpi_chk_0002 ****
>>>>  *** Wrote plotfile to sedov_2d_6lev_ncmpi_plt_cnt_0003 ****
>>>>        3 1.0000E-04 2.5000E-05  ( 5.139E-01,  4.988E-01, 0.000E+00) |  6.407E-07
>>>>  *** Wrote checkpoint file to sedov_2d_6lev_ncmpi_chk_0003 ****
>>>>     Nonconvergence in subroutine rieman
>>>>  .....
>>>> 
>>>> 
>>>> By increasing NXB and NYB, you have increased the resolution. This
>>>> requires a smaller timestep for stability, see "dt_hydro" column.  But
>>>> your flash.par fixes dtmin and dtmax at 2.5e-5, the direct effect of
>>>> this can be seen in the "dt" column. So you are forcing FLASH to do
>>>> Hydro advances with a time step that is too large for stability.
>>>> So it is not surprising that the simulation fails after a few steps.
>>>> 
>>>> Basically, aking your runtime parameter "dtmin" much smaller should
>>>> get you a simulation that runs.
>>>> 
>>>> Btw, a more common way to increase resolution (thus file sizes) would
>>>> be to increase lrefine_in and/or lrefine_max and/or Nblock{X,Y,Z},
>>>> rather than keep increasing the block size.
>>>> 
>>>> Klaus
>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20160303/d2ce8cff/attachment.htm>