[FLASH-USERS] Negative/zero densities and internal energies after filling gc

[Anshu Dubey]

Seyit and Ross

FLASH evolves the solution only on the leaf blocks, and as such all
the provided solvers are guaranteed to be applied to only the leaf
blocks. If parents or ancestors have any valid data, it is because of
restriction from their children (with some very minor exceptions). If
you are interested in evolving at all levels, you will have make sure
that other solvers also operate on all blocks, by changing "LEAF" to
"ALLBLOCKS" in calls to Grid_getListOfBlocks. And you will have to
enable the "advance_all_levels" mode in Paramesh. However, you should
be aware that some of the solvers cannot be trivially changed in this
way, for example active particles
and self gravity.

There will be some other changes necessary, since different parts of
FLASH operate under the assumption that only leaf blocks are
propagated. At the least, various calls to Grid_restrictAllLevels in
IO_* subroutines and a call to amr_restrict in
Grid_updateRefinement.F90 should be removed or disabled.

If it isn't really necessary to evolve the solution on all levels
at the same time, we recommend that you stick with evolving only on the leaf
blocks - especially if this is the first time using FLASH3

We were able to reproduce negative values in density in the Jeans
problem in some of the guardcells of parent blocks after changing the
refinement criterion. Those guardcell values are coming from the
fine-coarse boundaries where same level neighbor of a "parent" block
is an "ancestor" block. Ancestor blocks do not always  have valid
values. These guardcells have no impact on solution as long as you are
advancing on LEAF blocks only, so they can be  safely ignored.   We
verified by filling the ancestor blocks with zeros, and in the parents
blocks in question, the zeroes showed up in the exact same pattern
that we had seen with the -ve values.

We don't yet understand why these warnings are triggered in the case
of Jeans but not in many other problems.  To reduce or suppress the
annoying messages, you could to either of two things:
1) Remove or comment out the line '#define DEBUG_CONSCONV'
  in gr_sanitizeDataAfterInterp.F90.  You should then see only
  messages with one line per block / variable on standard output, no
  dumping of the blocks contents.
2) Change gr_sanitizeDataAfterInterp so it only checks leaf blocks, not
  leaf and parent blocks. That is, you might suppressing the checking
  or the output if (nodetype(block)==2) .

Although we didn't see any negative density values in the interior of
parent blocks, or anywhere in the LEAF blocks, if you are getting
them, then there is cause for concern. I wouldn't simply take absolute
values, because something else may be going wrong with your
simulation. You cannot rely on those results.

We hope that the above explanation will be helpful in solving your
problems.  If you are seeing  some behavior that is different from
what I described above, we'd like to know more about it.

The flash code group.

On Thu, Apr 9, 2009 at 11:49 AM, Seyit Hocuk <seyit at astro.rug.nl> wrote:
> Hi,
>
> Ross, I tried but your trick didn't work unfortunately. It even made it more
> instable as it seemed. Tomek might be right.
>
> To everybody,
>
> I've reduced the problem a bit. Jeans' specific refinement is bad. I think
> it doesn't work properly, so I commented some of the refinement criteria in
> Grid_markRefineDerefine and used Grid_markRefineSpecialized for the
> threshold criteria instead, which works fine. So that's solved.
>
> I also set convertToConsvdForMeshCalls to .True. in runtime parameters
> instead of using convertToConsvdInMeshInterp, which should have worked
> better because it is supposed to avoid spurius things in paramesh3+. However
> convertToConsvdInMeshInterp calls at some point gr_sanitizeDataAfterInterp,
> which creates all the weirdness. It is my humble opinion that something goes
> wrong with conservation there.
>
> Lastly, I did a very bad thing to solve the negative densities. In
> Eos_wrapped, where the densities become negative after getting the density
> from solnData (it seems the value is ok, just the sign changed), I use the
> absolute value to get passed that problem
> (ASB(solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)))). I know
> this isn't the way, but everything looks ok for now. Until a better solution
> is found, I will continue like this.
>
> With my limited knowledge and understanding of the code, I could only come
> up with this trick.
>
> Can anybody come up with a better solution or has anybody else implemented
> cooling successfully?
>
> Thanks,
> Seyit
>
>
>
>
> Tomasz Plewa wrote:
>>
>> This would imply there is an inconsistency in applying physics operators.
>>
>> I think there exist a (PARAMESH?) option enforcing evolving all blocks in
>> the simulation, if necessary. A local modification might be more efficient,
>> though.
>>
>> In the latter case, the next question I would have is whether all blocks
>> coarser than parent blocks do have valid data.
>>
>> Tomek
>> --
>> Ross Parkin wrote:
>>>
>>> Hi Seyit,
>>>
>>>
>>>
>>> Quoting Seyit Hocuk <seyit at astro.rug.nl>:
>>>
>>>
>>>>
>>>> Hi community,
>>>>
>>>> I've switched to Flash3.1.1a (from version 2.5), but when I try to run
>>>> my setup, comparable to the Jeans setup, I encounter problems. If I am
>>>> running without cooling and with a uniform grid (lref_max=lref_min),
>>>> then everything seems ok. However, if I add the cooling module OR I
>>>> have adaptive refinement, zero or negative densities/internal energies
>>>> appear (or lower than the "small" values). These cause the refinement
>>>> to be messed up and of course leads to crashes. The error outpusts are
>>>> shown below. From there, I notice that the error might come from guard
>>>> cell filling.
>>>>
>>>> I test the standard Jeans setup and the same problems occur.
>>>> Just changing and adding in flash.par:
>>>> lrefine_max = 5
>>>> refine_cutoff_1 = 1.0
>>>>
>>>> On several occasions I have checked the Eos_wrapped.F90 file and found
>>>> the location where it might go wrong;
>>>> just before reading solnData eint/dens was ok, but after this the
>>>> values are crazy. Btw., I don't use massFractions ad haven't
>>>> implemented Multispecies and no species are initiated.
>>>>
>>>>       eosData(pres+1:pres+vecLen) = &
>>>>            solnData(PRES_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>       eosData(dens+1:dens+vecLen) = &
>>>>            solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>       eosData(temp+1:temp+vecLen) = &
>>>>            solnData(TEMP_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>       eosData(gamc+1:gamc+vecLen) = &
>>>>            solnData(GAMC_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>       eosData(eint+1:eint+vecLen) = energyInternal(1:vecLen)
>>>>
>>>>
>>>>       call
>>>>
>>>> Eos(mode,vecLen,eosData)!,massFraction)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>>
>>>>
>>>> Another thing that I notice but is likely completely unrelated is, that
>>>> with this latest Flash version, the upper level blocks are blocky and
>>>> have absurd values. Though, the highest refined level is always ok.
>>>> Could it be related, as it needs parent blocks for refinement? Then
>>>> again, how is this related to cooling when there is no refinement. I
>>>> double checked with Jeans and Sedov test problems; it was the same
>>>> thing.
>>>>
>>>>
>>>
>>> I can't help you with the Jeans simulation but I have noticed a  similar
>>> problem with the coarsest level blocks when you are using  source terms.
>>> Specifically, my problem was that blocks on the coarsest  refinement level
>>> were not being acted on by source terms. I tracked  down the root of this
>>> problem; it seems that the blockList passed from  Driver_evolveFlash into
>>> Driver_sourceTerms is only for leaf blocks and  not all blocks. You can get
>>> around this by adding a call to  Grid_getlistofblocks and requesting all
>>> blocks before you start  applying source terms. I hope this helps.
>>>
>>> Cheers,
>>>
>>> Ross Parkin
>>>
>>>
>>>
>>>
>>>>
>>>> Some info on my environment:
>>>> Kernel: Centos 5.2
>>>> Machine: Linux ulubey 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST
>>>> 2008 x86_64 x86_64 x86_64 GNU/Linux
>>>> ifort version: (IFORT) 10.1 20080602
>>>> hdf5 version: 1.8.1
>>>> MPICH version. 2-1.0.7
>>>> gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
>>>>
>>>>
>>>>
>>>>
>>>> A snippet from the log file when the warnings appear.
>>>>
>>>> [GRID amr_refine_derefine] min leaf blks 76    max leaf blks 76    tot
>>>> leaf blks 76
>>>> [ 04-08-2009  18:10:42.268 ] [GRID amr_refine_derefine]: refinement
>>>> complete
>>>> [ 04-08-2009  18:10:42.273 ] WARNING after amr_prolong: min.
>>>> unk(EINT_VAR)=6685180.741266931 PE=0 block=18 type=1
>>>> [ 04-08-2009  18:10:42.274 ] step: n=37 t=1.414748E+12 dt=1.000000E+11
>>>> [ 04-08-2009  18:10:42.279 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=6162624.299265089 PE=0 block=18 type=1
>>>> [ 04-08-2009  18:10:42.297 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=6174477.607051291 PE=0 block=18 type=1
>>>> .
>>>> .
>>>> [ 04-08-2009  18:10:42.939 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=6274674.035376545 PE=0 block=32 type=1
>>>> [ 04-08-2009  18:10:42.939 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=6259353.579602077 PE=0 block=33 type=1
>>>> [ 04-08-2009  18:10:42.992 ] [gr_hgRecordNodeTypeState] max refine level
>>>> =: 5
>>>> [ 04-08-2009  18:10:43.054 ] [IO_writeCheckpoint] open: type=checkpoint
>>>> name=xSH-A1_hdf5_chk_0002
>>>> [ 04-08-2009  18:10:43.059 ] [IO_writeCheckpoint] close:
>>>> type=checkpoint name=xSH-A1_hdf5_chk_0002
>>>> [ 04-08-2009  18:10:43.059 ] step: n=40 t=2.014748E+12 dt=1.000000E+11
>>>> [ 04-08-2009  18:10:43.064 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=5604765.671211276 PE=0 block=18 type=1
>>>> [ 04-08-2009  18:10:43.083 ] WARNING after gc filling: min.
>>>> unk(EINT_VAR)=5320601.064666561 PE=0 block=18 type=1
>>>> [ 04-08-2009  18:10:43.112 ] [gr_hgRecordNodeTypeState] max refine level
>>>> =: 5
>>>> [ 04-08-2009  18:10:43.161 ] WARNING after gc filling: min.
>>>> unk(DENS_VAR)=-0.7514521682254683594983E-21 PE=0 block=8 type=2
>>>> [ 04-08-2009  18:10:43.161 ] WARNING after gc filling: min.
>>>> unk(DENS_VAR)=-0.1537578263572373951611E-20 PE=0 block=13 type=2
>>>>
>>>>
>>>>
>>>> Another snippet from the log file when the crash occurs (more frequent
>>>> for cooling runs).
>>>>
>>>> [ 04-08-2009  17:51:23.737 ] [IO writePlotfile] open: type=plotfile
>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>> [ 04-08-2009  17:51:23.738 ] [IO_writePlotfile] close: type=plotfile
>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>> [ 04-08-2009  17:51:23.738 ] [Driver_evolveFlash]: Entering evolution
>>>> loop
>>>> [ 04-08-2009  17:51:23.739 ] step: n=1 t=0.000000E+00 dt=1.000000E+02
>>>> [ 04-08-2009  17:51:23.776 ] [gr_hgRecordNodeTypeState] max refine level
>>>> =: 4
>>>> [Eos_wrapped] ERROR Density or Internal Energy are zero after a call to
>>>> EOS!
>>>> [ 04-08-2009  17:51:23.816 ] [DRIVER_ABORT]: Driver_abort() called by
>>>> PE           0
>>>> [ 04-08-2009  17:51:23.816 ] abort_message: [Eos_wrapped] ERROR Density
>>>> or Internal Energy are zero after a call to EOS!
>>>>
>>>
>>>
>>>
>
>