Next: 5.10 Setup a FLASH Up: 5. The FLASH configuration Previous: 5.8 Setup a hybrid Contents Index
In FLASH4 we have introduced a new grid implementation that makes use of Chombo library (Sec:chombo). This allows us to create FLASH applications that use an adaptive patch-based mesh for the first time.
You can create a FLASH application using Chombo mesh with the standard FLASH setup script and the designated shortcuts +chombo_ug or +chombo_amr. These shortcuts instruct the setup script to:
The shortcuts are used in FLASH setup lines as follows:
./setup Sedov -auto +chombo_ug -parfile=test_chombo_ug_2d.par ./setup Sod -auto +chombo_amr -parfile=test_chombo_amr_2d.par
The setup lines for all the problems that we have ever tested can be
found in the file
sites/code.rochester.edu/flash_test/test.info.
A FLASH application that makes use of Chombo library has the following dependencies:
The mixed-language Fortran/C interoperability features are critical for this project. Not all Fortran compilers have working C interoperability as this is a new feature that was introduced in the Fortran 2003 standard. Before spending time building Chombo and customizing FLASH Makefile headers, you should use our standalone unit test to determine whether your compilers have the basic mixed-language functionality needed by this project. The unit test is at source/Grid/GridMain/Chombo/wrapper/unit_tests/1 and is independent of FLASH. It is a standalone Fortran application that uses the iso_c_binding function c_f_pointer to construct a Fortran pointer object given a raw C memory address and a shape argument. To build, enter this directory and execute make "compiler name", where "compiler name" is gnu, intel, ibm, sun.
Old compilers, such as gfortran version 4.1.2, will give a compile-time error because iso_c_binding module is not available, and at least one version of gfortran, namely gfortran version 4.4.0, has an array indexing run-time bug that is detected by this unit test. The array indexing bug is documented at GCC bugzilla 40962 and gives the following incorrect answers shown in Figure 5.3.
We expect to pick up the complete set of integer values from 0 to 23 in contiguous memory locations Fortran array element: 1 1 1 1. Expected: 0, actual: 0. Fortran array element: 1 2 1 1. Expected: 1, actual: 1. Fortran array element: 1 1 2 1. Expected: 2, actual: 2. Fortran array element: 1 2 2 1. Expected: 3, actual: 3. Fortran array element: 1 1 3 1. Expected: 4, actual: 4. Fortran array element: 1 2 3 1. Expected: 5, actual: 5. Fortran array element: 1 1 1 2. Expected: 6, actual: 3. Fortran array element: 1 2 1 2. Expected: 7, actual: 4. Fortran array element: 1 1 2 2. Expected: 8, actual: 5. Fortran array element: 1 2 2 2. Expected: 9, actual: 6. Fortran array element: 1 1 3 2. Expected: 10, actual: 7. Fortran array element: 1 2 3 2. Expected: 11, actual: 8. Fortran array element: 1 1 1 3. Expected: 12, actual: 6. Fortran array element: 1 2 1 3. Expected: 13, actual: 7. Fortran array element: 1 1 2 3. Expected: 14, actual: 8. Fortran array element: 1 2 2 3. Expected: 15, actual: 9. Fortran array element: 1 1 3 3. Expected: 16, actual: 10. Fortran array element: 1 2 3 3. Expected: 17, actual: 11. Fortran array element: 1 1 1 4. Expected: 18, actual: 9. Fortran array element: 1 2 1 4. Expected: 19, actual: 10. Fortran array element: 1 1 2 4. Expected: 20, actual: 11. Fortran array element: 1 2 2 4. Expected: 21, actual: 12. Fortran array element: 1 1 3 4. Expected: 22, actual: 13. Fortran array element: 1 2 3 4. Expected: 23, actual: 14. FAILURE! Picked up unexpected C++ assigned values |
If this test fails at compile-time or run-time then FLASH-Chombo applications will not work! You can save a lot of time and frustration by running this unit test before continuing with the build process.
In contrast to Paramesh, the Chombo source code is not included in the
FLASH source tree. This means that you, the user, must manually
download and then build Chombo before building FLASH applications that
make use of Chombo. You should download version 3.0 or 3.1 of Chombo
from
https://seesar.lbl.gov/anag/chombo/ as these are
the only versions of Chombo that have been tested with FLASH. Please
refer to their user guide which is available from the same download
page for help building Chombo. It contains a very clear and detailed
build procedure along with a Chombo support email address that you can
contact if you experience problems building Chombo.
You should build 1D,2D and 3D versions of Chombo library in a MPI configuration (MPI = TRUE) with parallel HDF5 (USE_HDF = TRUE). A parallel HDF5 implementation means a HDF5 installation that is configured with -enable-parallel. You can check whether you have a parallel HDF5 build using the following command line:
grep "Parallel HDF5" /path/to/hdf5/lib/libhdf5.settings
You should also explicitly turn off Chombo memory tracking (USE_MT = FALSE) because it is a feature that does not work with FLASH's usage of Chombo. Note that it is not always sufficient to set USE_MT = FALSE in your Make.defs.local; for Intel compiler we also had to remove USE_MT = TRUE from the default compiler settings in ./lib/mk/compiler/Make.defs.Intel.
Only after you have successfully built and run the Chombo unit tests described in their user guide should you have any expectation that FLASH will work with Chombo library. For your reference we show the Make.defs.local macro definitions that we have used to build Chombo on a x86_64 machine in a FLASH-compatible way in Figure 5.4 .
CXX = icpc FC = ifort MPI = TRUE MPICXX = /home/cdaley/software/mpich2/1.3.1/intel/bin/mpicxx USE_64 = TRUE USE_HDF = TRUE HDFINCFLAGS = -I/home/cdaley/software/hdf5/1.8.5-patch1/intel/include -DH5_USE_16_API HDFLIBFLAGS = -L/home/cdaley/software/hdf5/1.8.5-patch1/intel/lib -lhdf5 HDFMPIINCFLAGS = -I/home/cdaley/software/hdf5/1.8.5-patch1/intel/include -DH5_USE_16_API HDFMPILIBFLAGS = -L/home/cdaley/software/hdf5/1.8.5-patch1/intel/lib -lhdf5 USE_MT = FALSE syslibflags = -lstdc++ -lz |
#elif defined(_POWER) || defined(_POWERPC) || defined(__POWERPC__)
#elif defined(_POWER) || defined(_POWERPC) || defined(__POWERPC__) || defined(__powerpc__)
At this stage you can optionally build and run the unit test source/Grid/GridMain/Chombo/wrapper/unit_tests/2. This is a standalone Fortran MPI application that uses Chombo library to create a distributed mesh over 4 MPI processes. It can be built by entering this directory and then executing the command make all. We expect that this program will be helpful for debugging mixed-language programming issues between FLASH and Chombo (especially on untested architectures / compiler combinations) independently of a full-blown FLASH application. Note that you will need to edit the Makefile appropriately for your installation of Chombo.
In order to build a FLASH application with Chombo you must create a custom FLASH Makefile.h named Makefile.h.chombo. We use a new file named Makefile.h.chombo because it includes convenience shell functions (described later) that will fail if a Chombo package is not available. At minimum the FLASH Makefile.h.chombo must include definitions for the new macros CFLAGS_CHOMBO and LIB_CHOMBO. These can be manually defined, but this is not recommended because it does not guarantee that FLASH's C++ wrapper classes are built the same way as the Chombo library. Example macro definitions for a FLASH 2D simulations are shown below:
CFLAGS_CHOMBO: -I/home/cdaley/flash/chomboForFlash/current/lib/include -DCH_SPACEDIM=2 -DCH_Linux -DCH_IFC -DCH_MPI -DMPICH_SKIP_MPICXX -ULAM_WANT_MPI2CPP -DMPI_NO_CPPBIND -DCH_USE_SETVAL -DCH_USE_COMPLEX -DCH_USE_64 -DCH_USE_DOUBLE -DCH_USE_HDF5 -I/home/cdaley/software/hdf5/1.8.5-patch1/intel/include -DH5_USE_16_API -DCH_FORT_UNDERSCORE -I/home/cdaley/flash/chomboForFlash/current/lib/src/BaseTools -DCH_LANG_CC LIB_CHOMBO: -L/home/cdaley/flash/chomboForFlash/current/lib -lamrtimedependent2d.Linux.64.mpicxx.ifort.DEBUG.MPI -lamrtools2d.Linux.64.mpicxx.ifort.DEBUG.MPI -lboxtools2d.Linux.64.mpicxx.ifort.DEBUG.MPI -lbasetools2d.Linux.64.mpicxx.ifort.DEBUG.MPI -lstdc++
This is obviously hugely inconvenient because the macros must be kept in sync with the current Chombo build and also the Chombo library names are dimension dependent. Our mechanism in FLASH4 to hide this build complexity is to add shell functions in Makefile.h.chombo that extract:
The shell functions appear in the Makefile.h.chombo in sites/code.rochester.edu, so this file should be used as your reference. A subset of this file is shown in Figure 5.5.
CHOMBO_PATH = /home/cdaley/flash/chomboForFlash/current #-------------------------------------- # Extract dimensionality from Flash.h. # The code in this section should not need to be modified. #-------------------------------------- _DIM := $(shell grep "define NDIM" Flash.h | cut -d " " -f 3) #-------------------------------------- # Extract Chombo build information from the Makefile at CHOMBO_PATH. # The code in this section should not need to be modified. #-------------------------------------- _MPI := $(shell make vars DIM=$_DIM -C $CHOMBO_PATH/lib | awk -F 'MPICXX=' '/^MPICXX/print $$2') ifeq ($(strip $(_MPI)),) $(error "Chombo MPICXX variable is empty") endif _CPP := $(shell make vars DIM=$_DIM -C $CHOMBO_PATH/lib | awk -F 'CPPFLAGS=' '/^CPPFLAGS/print $$2') _LIB := $(shell make vars DIM=$_DIM -C $CHOMBO_PATH/lib | awk -F 'config=' '/^config/print $$2') _PHDF_INC := $(shell make vars DIM=$_DIM -C $CHOMBO_PATH/lib | awk -F 'HDFMPIINCFLAGS=' '/^HDFMPIINCFLAGS/print $$2') _PHDF_LIB := $(shell make vars DIM=$_DIM -C $CHOMBO_PATH/lib | awk -F 'HDFMPILIBFLAGS=' '/^HDFMPILIBFLAGS/print $$2') #-------------------------------------- # Use Chombo build information to get consistent macro values for the FLASH build. #-------------------------------------- # Use two passes of dirname to strip the bin/mpicxx MPI_PATH := $(shell dirname $(shell dirname $(shell which $(_MPI)))) FCOMP = $MPI_PATH/bin/mpif90 CCOMP = $MPI_PATH/bin/mpicc CPPCOMP = $MPI_PATH/bin/mpicxx LINK = $MPI_PATH/bin/mpif90 CFLAGS_CHOMBO = -I$CHOMBO_PATH/lib/include $_CPP -DCH_LANG_CC CFLAGS_HDF5 = $(_PHDF_INC) LIB_CHOMBO = -L$(CHOMBO_PATH)/lib -lamrtimedependent$_LIB -lamrtools$_LIB -lboxtools$_LIB -lbasetools$_LIB -lstdc++ LIB_HDF5 = $(_PHDF_LIB) -lz |
You should now be ready to setup and build FLASH applications with Chombo mesh package.