[FLASH-USERS] Understanding FLASH memory requirements

[Klaus Weide]

On Fri, 26 Apr 2019, Ryan Farber wrote:

> I've seen quite a few of my simulations crash here and the typical "fix"
> would be requesting an extra node (more memory) on Stampede2. However, the
> machine I'm working on now doesn't have nearly as much memory per node as
> Stampede2's skylake nodes. So, I'd like to better understand FLASH's memory
> requirements.
> 
> My estimate for the memory required:
> NPROP_VARS = 26
> NSPECIES = 2
> NMASS_SCALARS = 1
> NUNK_VARS = 29
> 
> NFACE_VARS = 2
> 
> NPROP_FLUX = 10
> NSPECIES_FLUX = 2
> NMASS_SCALARS_FLUX = 1
> NFLUXES = 13
> 
> NVARS_TOTAL = 29 + (2*3) + 13 = 48

You shouldn't need to add the NFLUXES, at least not in full - they 
don't required full blocks' worth of memory.

> NSCRATCH_GRID_VARS = 3
> NSCRATCH_CENTER_VARS = 71
> 
> total_grid_vars = 48+3+71 = 122
> 
> NDIM = 3
> nxb = 8
> NGUARD = 4
> MAXBLOCKS = 500
> procs = 360
> bytes_per_GB = 1e9
> grid_space_gb = total_grid_vars*MAXBLOCKS*procs*(nxb+2*NGUARD)**NDIM /
> bytes_per_GB

It seems you have forgotten a factor of 8, for size of a double precision 
real in bytes.


> So, I'm confused about why the two runs report using such different amounts
> of memory. I'm also confused about why my estimate is so far off the value
> of rss*procs ~ 300 GB for the 360 procs, 30 nodes, maxblocks=500 case.

It is not clear that rss is the most relevant number.

Anyway, you should also look at the memory numbers after the 
first time step,, in the

!   ...(happily proceeds)

part.