[FLASH-USERS] Sedov scaling test

[mateuszr at umich.edu]

Hi all,

thanks very much for the help with the Sedov scaling test.
Switching to mpich2 fixed the problem entirely and the scaling was 
practically linear.

  thanks again,

      Mateusz


Quoting Anthony Chan <chan at mcs.anl.gov>:

>
>
> On Mon, 11 Feb 2008, mateuszr at umich.edu wrote:
>
>>
>> Hi all,
>>
>> I am testing a very small Dell cluster (8 cpus total). The machine 
>> is a dual quad core Xeon with 2 GB per core. The OS is Rocks 4.3, 
>> and the compilers used is the stock gcc 3.4.6 with mpich 1.2.7, 
>> mpif90 (Makefile.h enclosed below). FLASH version is 2.5 and I am 
>> using a Sedov test setup.
>>
>> The setup does not seem to scale with the number of processors at all.
>> If anything, the execution time increases very slightly as I double 
>> the number of processors. I also noticed that all timesteps are 
>> printed out to the screen n times, where n is the number of 
>> processors. It seems to me that for some reason, the code just runs 
>> the same calculation n times instead of distributing the load
>> evenly among all processors.
>
> It is posssible that you have n MPI_COMM_WORLD where each has only 1 
> process (instead of 1 MPI_COMM_WORLD with n processes).   You should
> use MPICH2 which is more robust than MPICH1 especially in process management.
>
> A.Chan
>
>>
>> It seems to me that this problem has nothing to do with the 
>> processor communication and, quite likely, is (embarrasingly?) 
>> trivial to solve.
>>
>> Has anybody seen this behavior before ?
>>
>> thanks,
>> Mateusz
>>
>>
>> P.S. I am enclosing Makefile.h below:
>>
>> # FLASH makefile definitions for ix86 Linux (Portland Group compiler)
>>
>> # note, in order to get pgprof to work properly, it was necessary to 
>> # download a new
>> version from ftp://ftp.pgroup.com/ that plays nicely
>> # with GNOME.
>>
>> #----------------------------------------------------------------------------
>> # Set the HDF/HDF5 library paths -- these need to be updated for your system
>> #----------------------------------------------------------------------------
>> #HDF4_PATH = /usr/local/hdf4
>> #HDF5_PATH = /opt/HDF5-1.4.2-patch1-serial/
>> HDF5_PATH = /opt/packages/hdf5-1.6.6/
>>
>> ZLIB_PATH  =
>>
>> PAPI_PATH  =
>> PAPI_FLAGS =
>>
>> NCMPI_PATH =
>> MPE_PATH   =
>>
>> #----------------------------------------------------------------------------
>> # Compiler and linker commands
>> #
>> #   Use the MPICH wrappers around the compilers -- these will automatically
>> #   load the proper libraries and include files.  Version of MPICH prior
>> #   to 1.2.2 (?) do not recognize .F90 as a valid Fortran file extension.
>> #   You need to edit mpif90 and add .F90 to the test of filename extensions,
>> #   or upgrade your MPICH.
>> #----------------------------------------------------------------------------
>> FCOMP   = /opt/mpich/intel/bin/mpif90
>> CCOMP   = /opt/mpich/intel/bin/mpicc
>> CPPCOMP = /opt/mpich/intel/bin/mpicc
>> LINK    = /opt/mpich/intel/bin/mpif90
>>
>> # pre-processor flag
>> PP      = -D
>>
>> #----------------------------------------------------------------------------
>> # Compilation flags
>> #
>> #  Three sets of compilation/linking flags are defined: one for optimized
>> #  code, one for testing, and one for debugging.  The default is to 
>> use the # _OPT
>> version.  Specifying -debug to setup will pick the _DEBUG version,
>> #  these should enable bounds checking.  Specifying _TEST is used 
>> for # flash_test, and
>> is set for quick code generation, and (sometimes) #  profiling.  The 
>> Makefile generated
>> by setup will assign the generic token #  (ex. FFLAGS) to the proper 
>> set of flags (ex.
>> FFLAGS_OPT).
>> #----------------------------------------------------------------------------
>> #FFLAGS_OPT = -c -r8 -i4 -CB -traceback -I/opt/packages/hdf5-1.6.6/include
>> -I/opt/mpich/intel/include/ -L/opt/mpich/intel/lib 
>> -L/opt/packages/hdf5-1.6.6/lib
>> -I/opt/packages/fftw-2.1.5/include -L/opt/packages/fftw-2.1.5/lib 
>> -lhdf5 -lrfftw_mpi
>> -lfftw_mpi -lrfftw
>> FFLAGS_OPT = -c -r8 -i4 -CB -traceback -I/opt/packages/hdf5-1.6.6/include
>> -I/opt/mpich/intel/include/ -L/opt/mpich/intel/lib 
>> -L/opt/packages/hdf5-1.6.6/lib
>> -I/opt/packages/fftw-2.1.5/include -L/opt/packages/fftw-2.1.5/lib 
>> -ldrfftw_mpi
>> -ldfftw_mpi -ldrfftw -ldfftw
>> # FFLAGS_OPT = -c -r8 -i4 -CB -traceback -I/usr/include/openmpi
>> -I/usr/include/openmpi/64/ -L/usr/lib/openmpi
>> #FFLAGS_OPT = -c -r8 -i4 -CB -traceback -I/opt/mpich/gnu/include 
>> -L/opt/mpich/gnu/lib
>> FFLAGS_DEBUG = -g -c -Mbounds -r8 -i4 FFLAGS_TEST = -c -r8 -i4 -Mprof=lines
>>
>> F90FLAGS =
>>
>> #CFLAGS_OPT = -O2  -c -I/opt/packages/hdf5-1.6.6/include 
>> -I/usr/include/openmpi
>> -I/usr/include/openmpi/64/ -L/usr/lib/openmpi
>> #CFLAGS_OPT = -O2  -c -I/opt/packages/hdf5-1.6.6/include 
>> -I/opt/mpich/gnu/include
>> -L/opt/mpich/gnu/lib
>> #CFLAGS_OPT = -O2  -c -I/opt/packages/hdf5-1.6.6/include 
>> -I/opt/intel/impi/3.1/include64/
>> -L/opt/intel/impi/3.1/lib64/ -L/opt/packages/hdf5-1.6.6/lib
>> -I/opt/packages/fftw-2.1.5/include -L/opt/packages/fftw-2.1.5/lib 
>> -lhdf5 -lrfftw_mpi
>> -lfftw_mpi -lrttfw
>> CFLAGS_OPT = -O2  -c -I/opt/packages/hdf5-1.6.6/include 
>> -I/opt/intel/impi/3.1/include64/
>> -L/opt/intel/impi/3.1/lib64/ -L/opt/packages/hdf5-1.6.6/lib
>> -I/opt/packages/fftw-2.1.5/include -L/opt/packages/fftw-2.1.5/lib 
>> -lhdf5 -ldrfftw_mpi
>> -ldfftw_mpi -ldrfftw -ldfftw
>>
>> CFLAGS_DEBUG = -g -c
>> CFLAGS_TEST = -c
>>
>> # if we are using HDF5, we need to specify the path to the include files
>> CFLAGS_HDF5 = -I $(HDF5_PATH)/include
>>
>> CFLAGS_NCMPI =
>>
>> #----------------------------------------------------------------------------
>> # Linker flags
>> #
>> #  There is a seperate version of the linker flags for each of the 
>> _OPT, # _DEBUG, and
>> _TEST cases.
>> #----------------------------------------------------------------------------
>>
>> LFLAGS_OPT   = -o
>> LFLAGS_DEBUG = -g -o
>> LFLAGS_TEST  = -Mprof=lines -o
>>
>>
>> #----------------------------------------------------------------------------
>> # Library specific linking
>> #
>> #  If a FLASH module has a 'LIBRARY xxx' line in its Config file, we need to
>> #  create a macro in this Makefile.h for LIB_xxx, which will be added to the
>> #  link line when FLASH is built.  This allows us to switch between 
>> different
>> #  (incompatible) libraries.  We also create a _OPT, _DEBUG, and _TEST
>> #  library macro to add any performance-minded libraries (like fast math),
>> #  depending on how FLASH was setup.
>> #----------------------------------------------------------------------------
>>
>> LIB_OPT   = LIB_DEBUG =
>> LIB_TEST  =
>>
>>
>> #LIB_MATH  = -L/u/mateusz/fftw-2.1.5/lib -ldrfftw_mpi -ldfftw_mpi 
>> -ldrfftw -ldfftw
>> #LIB_MATH = -L/opt/packages/fftw-2.1.5/lib -ldrfftw_mpi -ldfftw_mpi 
>> -ldrfftw -ldfftw
>> LIB_MATH = -L/opt/packages/fftw-2.1.5/lib  
>> -L/opt/packages/hdf5-1.6.6/lib -lrfftw_mpi
>> -lfftw_mpi -lrfftw -lfftw -lhdf5 #LIB_HDF4  = -L $(HDF4_PATH)/lib 
>> -lmfhdf -ldf -ljpeg -lz
>> LIB_HDF5 = -L $(HDF5_PATH)/lib $(LIB_MATH) -lmpi -lz -lsz
>> #LIB_HDF5  = -lhdf5 $(LIB_MATH) -lmpi
>>
>> LIB_PAPI  =
>>
>> LIB_NCMPI =
>> LIB_MPE   =
>>
>> #----------------------------------------------------------------------------
>> # Additional machine-dependent object files
>> #
>> #  Add any machine specific files here -- they will be compiled and linked
>> #  when FLASH is built.
>> #----------------------------------------------------------------------------
>>
>> MACHOBJ =
>>
>>
>> #----------------------------------------------------------------------------
>> # Additional commands
>> #----------------------------------------------------------------------------
>>
>> MV = mv -f
>> AR = ar -r
>> RM = rm -f
>> CD = cd
>> RL = ranlib
>> ECHO = echo
>>
>>
>>
>
>
>