[FLASH-USERS] 2D Spherical and Cartesian Geometry

[Klaus Weide]

On Tue, 27 Sep 2016, Michiel Bustraan wrote:

> Klaus,
> 
> I'm sorry for being unclear earlier.
> 
> When I made the changes to the unsplit hydro you suggested, it did not 
> seem to make a difference. Changing from 0-180 to 0-90 just caused those 
> issues to happen around 90 degrees. I think the issue lay more in the 
> structure of the grid or the use of hydro solver than the actual 
> boundaries.
> 
> I ran a simulation overnight that seemed to be working when I left, with 
> split hydro. It produced a few plotfiles that show reasonable results. 
> However, when coming in about an hour ago, I saw that it crashed overnight 
> with the same error message about missing neighbors.
> 
> I'm attaching a logfile from an identical simulation I just ran, with no 
> changes apart from increasing the number of AMR levels and changing the y 
> boundaries from 0-90 to 0-180. The AMR levels I changed to make it crash 
> sooner so I could provide a clean logfile as soon as possible, and the 
> boundaries because I first thought that might help. The original 
> simulation, which ran at least a couple of hours before crashing, had only 
> 3 refine levels.

Michiel,

Based on the log file, I notices some unusual things about your simulation.


During Grid domain initialization, your simulation failed to converge
to a stable configuration.  Normally, I would expect to see something
like the following group of lines in a log file:

 [ 09-27-2016  06:01:27.760 ] [GRID amr_refine_derefine]: refinement complete
 [ 09-27-2016  06:01:27.761 ] [GRID gr_expandDomain]: iteration=5, create level=5
 [ 09-27-2016  06:01:27.863 ] [mpi_amr_comm_setup]: buffer_dim_send=5779, buffer_dim_recv=4677
 [ 09-27-2016  06:01:27.886 ] [GRID gr_expandDomain]: iteration=6, create level=5
 [ 09-27-2016  06:01:27.991 ] [GRID gr_expandDomain]: iteration=7, create level=5

indicating two or three "GRID gr_expandDomain" iterations where the
configuration (i.e., the refinement pattern) does not change any more.

So it seems something unusual is going on in either your refinement
criteria (Does your simulation provide modified refinement routines?)
or in your Simulation_initBlock.

It seems your simulation is derived from the provided RadBlastWave
example, by throwing out A LOT of stuff; but maybe you still did not
throw out enough?  In particular, the Simulation_initBlock of RadBlastWave
has lots of superfluous code, and was probably only tested for the 2D
spherical case. Make sure you try a minimal version, which should be
really simple for initializing the whole domain to a constant state.

The way you define your mesh is unusual, for an AMR Grid in FLASH.
Normally, one would choose smaller values of nblockX and nblockY, and
make up for that with larger values of lrefine_max. For example,

 nblockX = 50
 nblockY = 50
 lrefine_min = 1
 lrefine_max = 5

and

 nblockX = 25
 nblockY = 25
 lrefine_min = 2
 lrefine_max = 6

should give exactly the same grid configuration, etc.

Note also that your Simulation is maybe doing something else than you
think, in terms of equation of state data.  You are using the 1T
"Gamma" Eos implementation. So the relevant data determining the
properties of your material will be the runtime parameters "gamma",
"eos_singleSpeciesA", and (maybe) "eos_singleSpecieZ".  You have also
configured in the Multispecies code unit and define an "H1" species,
but those properties (and the initializations in Simulations_initSpecies)
will be just ignored by the Eos implementation.

I am not sure how all this would lead to the errors you are observing;
still thinking about it.


> Missing neighbors on PE    0, lb= 1445,surrblks=(1430,   0, 2)(1435,   0, 2)(1462,   0, 1)(1440,   0, 2)(1445,   0, 2)(1464,   0, 1)(  -1,  -1,-1)(  -1,  -1,-1)(  -1,  -1,-1)
>  *** Wrote plotfile to rad_blast_wave_castro_hires2_forced_hdf5_plt_cnt_0000 ****

Note that FLASH has given you an extra plotfile, that could be useful in 
checking the sanity of the mesh. If you send it to me (NOT the list), I 
will have a quick look whether there's something wrong in the neighbor 
relationships of blocks.

> Regarding the EOS, what I meant was that I was running previous 
> simulations (in 1D spherical and then in 2D cartesian) by defining 
> eosMode, eosModeInit, and hy_eosmodeafter with some variation of dens_temp 
> (some of which is still there in the comments of flash.par). When I ran 
> simulation with the split hydro in 2D spherical, the code remarked that 
> that wasn't an appropriate choice of EOS, and would only run if I selected 
> dens_ei. I was curious what might have caused this.

Those runtime parameters should probably all be left as "dens_ie" (aka 
"dens_ei") [or some dens_ie_* variant, in the case of 3T setups], except 
for eosModeInit (which has to be chosen depending on the set of variables 
that are explicitly initialized in Simulation_initBlock).





Klaus