[FLASH-USERS] Insufficient memory during IO

[Yi-Hao Chen]

Hi all,

I am having a memory problem during the IO process. The error message
is "forrtl:
severe (41): insufficient virtual memory." Since there is no virtual memory
on the machine that I am using (stampede2), this means FLASH uses too much
memory.

The memory problem usually occurs during the writing of particle files. The
last few lines of the log file look like:

 [ 08-29-2018  04:56:37.695 ] [IO_writePlotfile] open: type=plotfile
name=Sim_hdf5_plt_cnt_0585
 [ 08-29-2018  04:56:48.700 ] [IO_writePlotfile] close: type=plotfile
name=Sim_hdf5_plt_cnt_0585
 [ 08-29-2018  04:56:48.707 ] [IO_writeParticles] open: type=particles
name=Sim_hdf5_part_0585
 [ 08-29-2018  04:56:48.708 ] [IO_writeParticles]: done called
Particles_updateAttributes()

I am running MHD AMR simulation and do have a large number of passive
particles. However, if I log into the computing node during the simulation,
the memory usage is usually less than 60% of the available memory. But it
seems the memory usage suddenly jumps and exceeds the limit during the
writing of output files. Does FLASH use much more memory during IO? I am
using FLASH4.5 with Intel MPI compiler and serial HDF5.

The problem is not reproducible and does not happen very often. If I
restart the simulation from the last available checkpoint file, it will not
crash at the same time.

I would appreciate if you have any thoughts on how to identify the problem
or solve it. Thank you!

Sincerely,
Yi-Hao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20180829/81ef31ef/attachment.htm>