[FLASH-USERS] Understanding FLASH memory requirements

[Ryan Farber]

Hi Klaus,

Thanks for the info!

I eventually found that my simulation fits onto 19 processors for
maxblocks=1000, 180 processors (vs. 30 nodes for maxblocks=500, 360
processors).
I'm thinking now that the reason it doesn't scale perfectly (i.e., from 30
nodes down to 15 nodes for 2x higher maxblocks and 2x fewer processors) is
because the processor assignment is depends on several factors (Morton
z-filling curve, data locality, etc.)
So, I think it is reasonable.

Thanks for the help!

Best,
--------
Ryan


On Fri, May 3, 2019 at 8:24 AM Klaus Weide <klaus at flash.uchicago.edu> wrote:

> On Fri, 26 Apr 2019, Ryan Farber wrote:
>
> > Hi Klaus,
> >
> > Thanks for your response! I think I've got a handle on things now, but I
> > typed out my thought process below in case it helps other people in the
> > future.
> >
> > > grid_space_gb = total_grid_vars*MAXBLOCKS*procs*(nxb+2*NGUARD)**NDIM /
> > > > bytes_per_GB
> > >
> > > It seems you have forgotten a factor of 8, for size of a double
> precision
> > > real in bytes.
> > >
> > > Good point! So I actually get ~730 GB memory required, though I'm
> > overestimating since you mentioned that NFLUXES doesn't require a full
> > block's worth of memory (why is that?).
>
> The flux arrays that are used directly by the Hydro unit are only
> declared / allocated for one block [they don't have something like
> ',...,MAXBLOCKS)' in the dimensions], so they don't count here.
>
> There are various arrays, both inside the PARAMESH code and the
> higher-level Grid unit code, to store fluxes in connection with flux
> correction (flux conservation). Those do scale with MAXBLOCKS. But they
> are not declared for all cells, but only (more or less) for cell faces at
> block faces. See the following files for the gory details (search for
> NFLUXES or nfluxes):
>
>   amr_initialize.F90
>   Grid/GridMain/paramesh/Grid_data.F90
>   Grid/GridMain/paramesh/Grid_init.F90
>
>
>
> (There is a line
>
>   #define NVARS_TOTAL (NUNK_VARS + (NFACE_VARS * 3) + NFLUXES)
>
> in Flash.h but that doesn't seem to be very useful - I don't see
> NVARS_TOTAL actually being used for anything.)
>
>
> Klaus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20190515/450182b4/attachment.htm>