[FLASH-USERS] Big Problem??

[Seyit Hocuk]

Hi,

It has been some time now, but to continue the discussion, I have tested 
Jeans standard problem and Sedov standard problem including changing the 
refinement levels. The result is exactly same for all three different 
computers. Unfortunately my own setup shows differences and it seems the 
end result changes by the coice of computer/environment/compiler 
whatever. My own setup has a lot more physics than the simpe test problems.

One thing I am thinking about is that I adjusted the refinement criteria 
to use a reference density instead of looking at the second order 
derivative. I don't know if this could be the problem, but dt_hydro 
should not depend on the refinement criteria so I guess there is more to it.

Unfortunately it is not easy to change compilers and some other stuff 
(like cpu/kernel etc.) on the different computers because I have limited 
rights on those machines. I don't want to find out I have to choose one 
machine and stick with it. How do I know which machine is the best. 
Anyway, I am going to try at least to get the libraries the same (hdf5 / 
mpi) for all machines and see if that helps. Btw the latest hdf5 (1.8.1) 
doesn't seem to work with Flash2.5.

I will also play with compiler options. Try to make them exactly the 
same and avoid aggressive optimizations. I hope this is where the 
solution lies.

Thanks,
Seyit




Tomasz Plewa wrote:
> Although I prefer a low cost approach to debugging which avoids 
> dealing with setup/system/compiler information at first and attempts 
> to eliminate 99% of the source code, here are  quick comments on your 
> systems.
>
> (1) pgi 6.0 is a quite antique compiler. It's so old that now I do not 
> even remember what problems we may have had with it. I suspect quite a 
> few and none of them might be relevant...
>
> (2) As you might noticed, intel has a newer compiler version. Your 
> CentOS5 system needs upgrades on both mpich2 and compiler sides.
>
> (3) Your Ubuntu system seems relatively up-to-date although HDF5 1.8.x 
> is out there and seems working fine.
>
> (4) Compilers are known to do strange things to perfect codes! So if 
> you run into a trouble on a new system, one of the first things to try 
> is to lower compiler optimization level. Avoiding -fast or -ip or 
> -inline is usually a good idea. Simple -O0 may suffice for a couple of 
> steps. As a bonus, the code compiles fast. Then the trick is to get it 
> crashing quickly...
>
> Again, I suggest to simplify your application, isolate the problem, 
> fix it, then find the best compiler, mpich, options, etc., to run your 
> production code.
>
> Enjoy your weekend!
>
> Tomek
> -- 
> Seyit Hocuk wrote:
>> Hi Nathan,
>>
>> Thats a good Idea, I'll start simple and test those first thing 
>> monday. It's too late for me now.
>>
>> To answer your questions:
>>
>> The problem I am running is my own home made, actually it's a 
>> modified version of Jeans. Basically including heat, cool, eos gamma, 
>> compositions, adjusted refinement criteria similar to one of the 
>> other problems (with delta_ref and delta_deref) and a scala of other 
>> small things.
>>
>> 1)
>> Centos5
>> Linux celsius 2.6.18-53.1.19.el5 #1 SMP Wed May 7 08:20:19 EDT 2008 
>> i686 i686 i386 GNU/Linux
>> ifort intel compiler version 8.0
>> hdf5 1.6.6 (serial)
>> mpich2 Version:  1.0.3
>> gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
>> FFLAGS_OPT   =  -c -r8 -i4 -fast -ipo -ipo_obj  -I $(MPI_PATH)/include
>>
>> 2)
>> Debian (??)
>> Linux hpcibm1 2.6.8-11-amd64-k8-smp #1 SMP Sun Oct 2 20:03:22 UTC 
>> 2005 x86_64 GNU/Linux
>> pgf90 6.0-5 32-bit target on x86-64 Linux
>> hdf5 1.4.5 (serial)
>> mpich2 version 1.0.5
>> gcc version 3.3.5 (Debian 1:3.3.5-13)
>> FFLAGS_OPT = -c -fast -r8 -i4
>>
>> 3)
>> Ubuntu
>> Linux si01 2.6.24.3 #1 SMP Fri Jul 4 11:02:09 CEST 2008 x86_64 GNU/Linux
>> ifort intel compiler version 10.1.012
>> hdf5 1.6.7 (serial)
>> mpich2 Version: 1.0.6
>> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
>> FFLAGS_OPT   = -c -r8 -i4  -xT -O3 -no-prec-div -static -I 
>> $(MPI_PATH)/include
>> or (only -ipo difference, result seemingly doesn't change)
>> FFLAGS_OPT   = -c -r8 -i4  -fast -I $(MPI_PATH)/include
>>
>>
>>
>>
>> Nathan Hearn wrote:
>>> Hi Seyit,
>>>
>>>     Are you seeing these differences in the standard test problems
>>> included in Flash (e.g., Noh, Sod, Sedov), or are they specific to the
>>> problem that you are running.  (Really, this follows from Tomek's
>>> advice to look at the simplest configurations where the differences
>>> are seen.)
>>>
>>>     Also, it would be good to include some more information in your
>>> messages, for instance: the problem that you are running, the specific
>>> architectures and compilers, the optimization flags, and the library
>>> versions for HDF5 and MPI that you are using.  Details like these can
>>> be really helpful in evaluating and fixing the issues you are seeing.
>>>
>>>
>>> - Nathan
>>>
>>>
>>> On Fri, Jul 18, 2008 at 11:32 AM, Seyit Hocuk <seyit at astro.rug.nl> 
>>> wrote:
>>>  
>>>> Oh man,
>>>>
>>>> I am using 3 different computers (with different architectures, 
>>>> compilers,
>>>> optimizations, environments). All three start differently 
>>>> immediately from
>>>> the beginning (difference in dt_hydro and refinement).
>>>> Should I worry very very very much?
>>>> I'm now doing a complete simulation and am going to check if my end 
>>>> results
>>>> also change.
>>>>
>>>> I'm trembling by the thought.....
>>>>     
>>
>>
>