[FLASH-USERS] Negative/zero densities and internal energies after filling gc
Kevin Olson
Kevin.M.Olson at drexel.edu
Fri Apr 10 12:23:12 EDT 2009
All,
I have one comment on the use of "advance_all_levels" in Paramesh.
This option only has an effect on guardcell filling. If it is set to
.true. Paramesh will assume that parent block data which is needed for
guardcell filling has been updated by the user (i.e. FLASH in this
case) rather than doing a default interpolation to the parent block.
Since, FLASH does not guarantee that parent blocks will have valid data
(as far as I know) then setting "advance_all_levels" to .true. could do
more harm that good.
Kevin Olson
On Apr 10, 2009, at 11:54 AM, Anshu Dubey wrote:
> Seyit and Ross
>
> FLASH evolves the solution only on the leaf blocks, and as such all
> the provided solvers are guaranteed to be applied to only the leaf
> blocks. If parents or ancestors have any valid data, it is because of
> restriction from their children (with some very minor exceptions). If
> you are interested in evolving at all levels, you will have make sure
> that other solvers also operate on all blocks, by changing "LEAF" to
> "ALLBLOCKS" in calls to Grid_getListOfBlocks. And you will have to
> enable the "advance_all_levels" mode in Paramesh. However, you should
> be aware that some of the solvers cannot be trivially changed in this
> way, for example active particles
> and self gravity.
>
> There will be some other changes necessary, since different parts of
> FLASH operate under the assumption that only leaf blocks are
> propagated. At the least, various calls to Grid_restrictAllLevels in
> IO_* subroutines and a call to amr_restrict in
> Grid_updateRefinement.F90 should be removed or disabled.
>
> If it isn't really necessary to evolve the solution on all levels
> at the same time, we recommend that you stick with evolving only on
> the leaf
> blocks - especially if this is the first time using FLASH3
>
> We were able to reproduce negative values in density in the Jeans
> problem in some of the guardcells of parent blocks after changing the
> refinement criterion. Those guardcell values are coming from the
> fine-coarse boundaries where same level neighbor of a "parent" block
> is an "ancestor" block. Ancestor blocks do not always have valid
> values. These guardcells have no impact on solution as long as you are
> advancing on LEAF blocks only, so they can be safely ignored. We
> verified by filling the ancestor blocks with zeros, and in the parents
> blocks in question, the zeroes showed up in the exact same pattern
> that we had seen with the -ve values.
>
> We don't yet understand why these warnings are triggered in the case
> of Jeans but not in many other problems. To reduce or suppress the
> annoying messages, you could to either of two things:
> 1) Remove or comment out the line '#define DEBUG_CONSCONV'
> in gr_sanitizeDataAfterInterp.F90. You should then see only
> messages with one line per block / variable on standard output, no
> dumping of the blocks contents.
> 2) Change gr_sanitizeDataAfterInterp so it only checks leaf blocks, not
> leaf and parent blocks. That is, you might suppressing the checking
> or the output if (nodetype(block)==2) .
>
> Although we didn't see any negative density values in the interior of
> parent blocks, or anywhere in the LEAF blocks, if you are getting
> them, then there is cause for concern. I wouldn't simply take absolute
> values, because something else may be going wrong with your
> simulation. You cannot rely on those results.
>
> We hope that the above explanation will be helpful in solving your
> problems. If you are seeing some behavior that is different from
> what I described above, we'd like to know more about it.
>
> The flash code group.
>
> On Thu, Apr 9, 2009 at 11:49 AM, Seyit Hocuk <seyit at astro.rug.nl>
> wrote:
>> Hi,
>>
>> Ross, I tried but your trick didn't work unfortunately. It even made
>> it more
>> instable as it seemed. Tomek might be right.
>>
>> To everybody,
>>
>> I've reduced the problem a bit. Jeans' specific refinement is bad. I
>> think
>> it doesn't work properly, so I commented some of the refinement
>> criteria in
>> Grid_markRefineDerefine and used Grid_markRefineSpecialized for the
>> threshold criteria instead, which works fine. So that's solved.
>>
>> I also set convertToConsvdForMeshCalls to .True. in runtime parameters
>> instead of using convertToConsvdInMeshInterp, which should have worked
>> better because it is supposed to avoid spurius things in paramesh3+.
>> However
>> convertToConsvdInMeshInterp calls at some point
>> gr_sanitizeDataAfterInterp,
>> which creates all the weirdness. It is my humble opinion that
>> something goes
>> wrong with conservation there.
>>
>> Lastly, I did a very bad thing to solve the negative densities. In
>> Eos_wrapped, where the densities become negative after getting the
>> density
>> from solnData (it seems the value is ok, just the sign changed), I
>> use the
>> absolute value to get passed that problem
>> (ASB(solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)))). I
>> know
>> this isn't the way, but everything looks ok for now. Until a better
>> solution
>> is found, I will continue like this.
>>
>> With my limited knowledge and understanding of the code, I could only
>> come
>> up with this trick.
>>
>> Can anybody come up with a better solution or has anybody else
>> implemented
>> cooling successfully?
>>
>> Thanks,
>> Seyit
>>
>>
>>
>>
>> Tomasz Plewa wrote:
>>>
>>> This would imply there is an inconsistency in applying physics
>>> operators.
>>>
>>> I think there exist a (PARAMESH?) option enforcing evolving all
>>> blocks in
>>> the simulation, if necessary. A local modification might be more
>>> efficient,
>>> though.
>>>
>>> In the latter case, the next question I would have is whether all
>>> blocks
>>> coarser than parent blocks do have valid data.
>>>
>>> Tomek
>>> --
>>> Ross Parkin wrote:
>>>>
>>>> Hi Seyit,
>>>>
>>>>
>>>>
>>>> Quoting Seyit Hocuk <seyit at astro.rug.nl>:
>>>>
>>>>
>>>>>
>>>>> Hi community,
>>>>>
>>>>> I've switched to Flash3.1.1a (from version 2.5), but when I try to
>>>>> run
>>>>> my setup, comparable to the Jeans setup, I encounter problems. If
>>>>> I am
>>>>> running without cooling and with a uniform grid
>>>>> (lref_max=lref_min),
>>>>> then everything seems ok. However, if I add the cooling module OR I
>>>>> have adaptive refinement, zero or negative densities/internal
>>>>> energies
>>>>> appear (or lower than the "small" values). These cause the
>>>>> refinement
>>>>> to be messed up and of course leads to crashes. The error outpusts
>>>>> are
>>>>> shown below. From there, I notice that the error might come from
>>>>> guard
>>>>> cell filling.
>>>>>
>>>>> I test the standard Jeans setup and the same problems occur.
>>>>> Just changing and adding in flash.par:
>>>>> lrefine_max = 5
>>>>> refine_cutoff_1 = 1.0
>>>>>
>>>>> On several occasions I have checked the Eos_wrapped.F90 file and
>>>>> found
>>>>> the location where it might go wrong;
>>>>> just before reading solnData eint/dens was ok, but after this the
>>>>> values are crazy. Btw., I don't use massFractions ad haven't
>>>>> implemented Multispecies and no species are initiated.
>>>>>
>>>>> eosData(pres+1:pres+vecLen) = &
>>>>>
>>>>> solnData(PRES_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(dens+1:dens+vecLen) = &
>>>>>
>>>>> solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(temp+1:temp+vecLen) = &
>>>>>
>>>>> solnData(TEMP_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(gamc+1:gamc+vecLen) = &
>>>>>
>>>>> solnData(GAMC_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(eint+1:eint+vecLen) = energyInternal(1:vecLen)
>>>>>
>>>>>
>>>>> call
>>>>>
>>>>> Eos(mode,vecLen,eosData)!,massFraction)!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>>> !!!!!!!!!!!!!!!!!!!!!!
>>>>>
>>>>>
>>>>> Another thing that I notice but is likely completely unrelated is,
>>>>> that
>>>>> with this latest Flash version, the upper level blocks are blocky
>>>>> and
>>>>> have absurd values. Though, the highest refined level is always ok.
>>>>> Could it be related, as it needs parent blocks for refinement? Then
>>>>> again, how is this related to cooling when there is no refinement.
>>>>> I
>>>>> double checked with Jeans and Sedov test problems; it was the same
>>>>> thing.
>>>>>
>>>>>
>>>>
>>>> I can't help you with the Jeans simulation but I have noticed a
>>>> similar
>>>> problem with the coarsest level blocks when you are using source
>>>> terms.
>>>> Specifically, my problem was that blocks on the coarsest
>>>> refinement level
>>>> were not being acted on by source terms. I tracked down the root
>>>> of this
>>>> problem; it seems that the blockList passed from
>>>> Driver_evolveFlash into
>>>> Driver_sourceTerms is only for leaf blocks and not all blocks. You
>>>> can get
>>>> around this by adding a call to Grid_getlistofblocks and
>>>> requesting all
>>>> blocks before you start applying source terms. I hope this helps.
>>>>
>>>> Cheers,
>>>>
>>>> Ross Parkin
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Some info on my environment:
>>>>> Kernel: Centos 5.2
>>>>> Machine: Linux ulubey 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16
>>>>> 11:57:43 EST
>>>>> 2008 x86_64 x86_64 x86_64 GNU/Linux
>>>>> ifort version: (IFORT) 10.1 20080602
>>>>> hdf5 version: 1.8.1
>>>>> MPICH version. 2-1.0.7
>>>>> gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> A snippet from the log file when the warnings appear.
>>>>>
>>>>> [GRID amr_refine_derefine] min leaf blks 76 max leaf blks 76
>>>>> tot
>>>>> leaf blks 76
>>>>> [ 04-08-2009 18:10:42.268 ] [GRID amr_refine_derefine]: refinement
>>>>> complete
>>>>> [ 04-08-2009 18:10:42.273 ] WARNING after amr_prolong: min.
>>>>> unk(EINT_VAR)=6685180.741266931 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:42.274 ] step: n=37 t=1.414748E+12
>>>>> dt=1.000000E+11
>>>>> [ 04-08-2009 18:10:42.279 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6162624.299265089 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:42.297 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6174477.607051291 PE=0 block=18 type=1
>>>>> .
>>>>> .
>>>>> [ 04-08-2009 18:10:42.939 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6274674.035376545 PE=0 block=32 type=1
>>>>> [ 04-08-2009 18:10:42.939 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6259353.579602077 PE=0 block=33 type=1
>>>>> [ 04-08-2009 18:10:42.992 ] [gr_hgRecordNodeTypeState] max refine
>>>>> level
>>>>> =: 5
>>>>> [ 04-08-2009 18:10:43.054 ] [IO_writeCheckpoint] open:
>>>>> type=checkpoint
>>>>> name=xSH-A1_hdf5_chk_0002
>>>>> [ 04-08-2009 18:10:43.059 ] [IO_writeCheckpoint] close:
>>>>> type=checkpoint name=xSH-A1_hdf5_chk_0002
>>>>> [ 04-08-2009 18:10:43.059 ] step: n=40 t=2.014748E+12
>>>>> dt=1.000000E+11
>>>>> [ 04-08-2009 18:10:43.064 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=5604765.671211276 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:43.083 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=5320601.064666561 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:43.112 ] [gr_hgRecordNodeTypeState] max refine
>>>>> level
>>>>> =: 5
>>>>> [ 04-08-2009 18:10:43.161 ] WARNING after gc filling: min.
>>>>> unk(DENS_VAR)=-0.7514521682254683594983E-21 PE=0 block=8 type=2
>>>>> [ 04-08-2009 18:10:43.161 ] WARNING after gc filling: min.
>>>>> unk(DENS_VAR)=-0.1537578263572373951611E-20 PE=0 block=13 type=2
>>>>>
>>>>>
>>>>>
>>>>> Another snippet from the log file when the crash occurs (more
>>>>> frequent
>>>>> for cooling runs).
>>>>>
>>>>> [ 04-08-2009 17:51:23.737 ] [IO writePlotfile] open: type=plotfile
>>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>>> [ 04-08-2009 17:51:23.738 ] [IO_writePlotfile] close:
>>>>> type=plotfile
>>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>>> [ 04-08-2009 17:51:23.738 ] [Driver_evolveFlash]: Entering
>>>>> evolution
>>>>> loop
>>>>> [ 04-08-2009 17:51:23.739 ] step: n=1 t=0.000000E+00
>>>>> dt=1.000000E+02
>>>>> [ 04-08-2009 17:51:23.776 ] [gr_hgRecordNodeTypeState] max refine
>>>>> level
>>>>> =: 4
>>>>> [Eos_wrapped] ERROR Density or Internal Energy are zero after a
>>>>> call to
>>>>> EOS!
>>>>> [ 04-08-2009 17:51:23.816 ] [DRIVER_ABORT]: Driver_abort() called
>>>>> by
>>>>> PE 0
>>>>> [ 04-08-2009 17:51:23.816 ] abort_message: [Eos_wrapped] ERROR
>>>>> Density
>>>>> or Internal Energy are zero after a call to EOS!
>>>>>
>>>>
>>>>
>>>>
>>
>>
>
More information about the flash-users
mailing list