[FLASH-USERS] Negative/zero densities and internal energies after filling gc
Seyit Hocuk
seyit at astro.rug.nl
Tue Apr 14 08:14:50 EDT 2009
Hi Anshu,
Thanks for your reply. Of the solutions you proposed for the warnings
(not on the crash), I have a few remarks.
> To reduce or suppress the
> annoying messages, you could to either of two things:
> 1) Remove or comment out the line '#define DEBUG_CONSCONV'
> in gr_sanitizeDataAfterInterp.F90. You should then see only
> messages with one line per block / variable on standard output, no
> dumping of the blocks contents.
> 2) Change gr_sanitizeDataAfterInterp so it only checks leaf blocks, not
> leaf and parent blocks. That is, you might suppressing the checking
> or the output if (nodetype(block)==2) .
Doing 1) still gives the warnings although like you say at a reduced amount.
With 2) you must mean if "nodetype(block)==1" for the leaf/child blocks.
However, that also does not completely remove the issue. The values can
still fall below the "small" values. I will still stick with the old
"convertToConsvdForMeshCalls" parameter.
Deducted from your idea though, a colleague of mine suggested to do the
cooling on only the leaf blocks, which has solved the negative/zero
values when implementing cooling. Now I use this solution instead of
using the abs() of the solnData.
Kind Regards,
Seyit
Anshu Dubey wrote:
> Seyit and Ross
>
> FLASH evolves the solution only on the leaf blocks, and as such all
> the provided solvers are guaranteed to be applied to only the leaf
> blocks. If parents or ancestors have any valid data, it is because of
> restriction from their children (with some very minor exceptions). If
> you are interested in evolving at all levels, you will have make sure
> that other solvers also operate on all blocks, by changing "LEAF" to
> "ALLBLOCKS" in calls to Grid_getListOfBlocks. And you will have to
> enable the "advance_all_levels" mode in Paramesh. However, you should
> be aware that some of the solvers cannot be trivially changed in this
> way, for example active particles
> and self gravity.
>
> There will be some other changes necessary, since different parts of
> FLASH operate under the assumption that only leaf blocks are
> propagated. At the least, various calls to Grid_restrictAllLevels in
> IO_* subroutines and a call to amr_restrict in
> Grid_updateRefinement.F90 should be removed or disabled.
>
> If it isn't really necessary to evolve the solution on all levels
> at the same time, we recommend that you stick with evolving only on the leaf
> blocks - especially if this is the first time using FLASH3
>
> We were able to reproduce negative values in density in the Jeans
> problem in some of the guardcells of parent blocks after changing the
> refinement criterion. Those guardcell values are coming from the
> fine-coarse boundaries where same level neighbor of a "parent" block
> is an "ancestor" block. Ancestor blocks do not always have valid
> values. These guardcells have no impact on solution as long as you are
> advancing on LEAF blocks only, so they can be safely ignored. We
> verified by filling the ancestor blocks with zeros, and in the parents
> blocks in question, the zeroes showed up in the exact same pattern
> that we had seen with the -ve values.
>
> We don't yet understand why these warnings are triggered in the case
> of Jeans but not in many other problems. To reduce or suppress the
> annoying messages, you could to either of two things:
> 1) Remove or comment out the line '#define DEBUG_CONSCONV'
> in gr_sanitizeDataAfterInterp.F90. You should then see only
> messages with one line per block / variable on standard output, no
> dumping of the blocks contents.
> 2) Change gr_sanitizeDataAfterInterp so it only checks leaf blocks, not
> leaf and parent blocks. That is, you might suppressing the checking
> or the output if (nodetype(block)==2) .
>
> Although we didn't see any negative density values in the interior of
> parent blocks, or anywhere in the LEAF blocks, if you are getting
> them, then there is cause for concern. I wouldn't simply take absolute
> values, because something else may be going wrong with your
> simulation. You cannot rely on those results.
>
> We hope that the above explanation will be helpful in solving your
> problems. If you are seeing some behavior that is different from
> what I described above, we'd like to know more about it.
>
> The flash code group.
>
> On Thu, Apr 9, 2009 at 11:49 AM, Seyit Hocuk <seyit at astro.rug.nl> wrote:
>
>> Hi,
>>
>> Ross, I tried but your trick didn't work unfortunately. It even made it more
>> instable as it seemed. Tomek might be right.
>>
>> To everybody,
>>
>> I've reduced the problem a bit. Jeans' specific refinement is bad. I think
>> it doesn't work properly, so I commented some of the refinement criteria in
>> Grid_markRefineDerefine and used Grid_markRefineSpecialized for the
>> threshold criteria instead, which works fine. So that's solved.
>>
>> I also set convertToConsvdForMeshCalls to .True. in runtime parameters
>> instead of using convertToConsvdInMeshInterp, which should have worked
>> better because it is supposed to avoid spurius things in paramesh3+. However
>> convertToConsvdInMeshInterp calls at some point gr_sanitizeDataAfterInterp,
>> which creates all the weirdness. It is my humble opinion that something goes
>> wrong with conservation there.
>>
>> Lastly, I did a very bad thing to solve the negative densities. In
>> Eos_wrapped, where the densities become negative after getting the density
>> from solnData (it seems the value is ok, just the sign changed), I use the
>> absolute value to get passed that problem
>> (ASB(solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)))). I know
>> this isn't the way, but everything looks ok for now. Until a better solution
>> is found, I will continue like this.
>>
>> With my limited knowledge and understanding of the code, I could only come
>> up with this trick.
>>
>> Can anybody come up with a better solution or has anybody else implemented
>> cooling successfully?
>>
>> Thanks,
>> Seyit
>>
>>
>>
>>
>> Tomasz Plewa wrote:
>>
>>> This would imply there is an inconsistency in applying physics operators.
>>>
>>> I think there exist a (PARAMESH?) option enforcing evolving all blocks in
>>> the simulation, if necessary. A local modification might be more efficient,
>>> though.
>>>
>>> In the latter case, the next question I would have is whether all blocks
>>> coarser than parent blocks do have valid data.
>>>
>>> Tomek
>>> --
>>> Ross Parkin wrote:
>>>
>>>> Hi Seyit,
>>>>
>>>>
>>>>
>>>> Quoting Seyit Hocuk <seyit at astro.rug.nl>:
>>>>
>>>>
>>>>
>>>>> Hi community,
>>>>>
>>>>> I've switched to Flash3.1.1a (from version 2.5), but when I try to run
>>>>> my setup, comparable to the Jeans setup, I encounter problems. If I am
>>>>> running without cooling and with a uniform grid (lref_max=lref_min),
>>>>> then everything seems ok. However, if I add the cooling module OR I
>>>>> have adaptive refinement, zero or negative densities/internal energies
>>>>> appear (or lower than the "small" values). These cause the refinement
>>>>> to be messed up and of course leads to crashes. The error outpusts are
>>>>> shown below. From there, I notice that the error might come from guard
>>>>> cell filling.
>>>>>
>>>>> I test the standard Jeans setup and the same problems occur.
>>>>> Just changing and adding in flash.par:
>>>>> lrefine_max = 5
>>>>> refine_cutoff_1 = 1.0
>>>>>
>>>>> On several occasions I have checked the Eos_wrapped.F90 file and found
>>>>> the location where it might go wrong;
>>>>> just before reading solnData eint/dens was ok, but after this the
>>>>> values are crazy. Btw., I don't use massFractions ad haven't
>>>>> implemented Multispecies and no species are initiated.
>>>>>
>>>>> eosData(pres+1:pres+vecLen) = &
>>>>> solnData(PRES_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(dens+1:dens+vecLen) = &
>>>>> solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(temp+1:temp+vecLen) = &
>>>>> solnData(TEMP_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(gamc+1:gamc+vecLen) = &
>>>>> solnData(GAMC_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>>>>> eosData(eint+1:eint+vecLen) = energyInternal(1:vecLen)
>>>>>
>>>>>
>>>>> call
>>>>>
>>>>> Eos(mode,vecLen,eosData)!,massFraction)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>>>
>>>>>
>>>>> Another thing that I notice but is likely completely unrelated is, that
>>>>> with this latest Flash version, the upper level blocks are blocky and
>>>>> have absurd values. Though, the highest refined level is always ok.
>>>>> Could it be related, as it needs parent blocks for refinement? Then
>>>>> again, how is this related to cooling when there is no refinement. I
>>>>> double checked with Jeans and Sedov test problems; it was the same
>>>>> thing.
>>>>>
>>>>>
>>>>>
>>>> I can't help you with the Jeans simulation but I have noticed a similar
>>>> problem with the coarsest level blocks when you are using source terms.
>>>> Specifically, my problem was that blocks on the coarsest refinement level
>>>> were not being acted on by source terms. I tracked down the root of this
>>>> problem; it seems that the blockList passed from Driver_evolveFlash into
>>>> Driver_sourceTerms is only for leaf blocks and not all blocks. You can get
>>>> around this by adding a call to Grid_getlistofblocks and requesting all
>>>> blocks before you start applying source terms. I hope this helps.
>>>>
>>>> Cheers,
>>>>
>>>> Ross Parkin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Some info on my environment:
>>>>> Kernel: Centos 5.2
>>>>> Machine: Linux ulubey 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST
>>>>> 2008 x86_64 x86_64 x86_64 GNU/Linux
>>>>> ifort version: (IFORT) 10.1 20080602
>>>>> hdf5 version: 1.8.1
>>>>> MPICH version. 2-1.0.7
>>>>> gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> A snippet from the log file when the warnings appear.
>>>>>
>>>>> [GRID amr_refine_derefine] min leaf blks 76 max leaf blks 76 tot
>>>>> leaf blks 76
>>>>> [ 04-08-2009 18:10:42.268 ] [GRID amr_refine_derefine]: refinement
>>>>> complete
>>>>> [ 04-08-2009 18:10:42.273 ] WARNING after amr_prolong: min.
>>>>> unk(EINT_VAR)=6685180.741266931 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:42.274 ] step: n=37 t=1.414748E+12 dt=1.000000E+11
>>>>> [ 04-08-2009 18:10:42.279 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6162624.299265089 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:42.297 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6174477.607051291 PE=0 block=18 type=1
>>>>> .
>>>>> .
>>>>> [ 04-08-2009 18:10:42.939 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6274674.035376545 PE=0 block=32 type=1
>>>>> [ 04-08-2009 18:10:42.939 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=6259353.579602077 PE=0 block=33 type=1
>>>>> [ 04-08-2009 18:10:42.992 ] [gr_hgRecordNodeTypeState] max refine level
>>>>> =: 5
>>>>> [ 04-08-2009 18:10:43.054 ] [IO_writeCheckpoint] open: type=checkpoint
>>>>> name=xSH-A1_hdf5_chk_0002
>>>>> [ 04-08-2009 18:10:43.059 ] [IO_writeCheckpoint] close:
>>>>> type=checkpoint name=xSH-A1_hdf5_chk_0002
>>>>> [ 04-08-2009 18:10:43.059 ] step: n=40 t=2.014748E+12 dt=1.000000E+11
>>>>> [ 04-08-2009 18:10:43.064 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=5604765.671211276 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:43.083 ] WARNING after gc filling: min.
>>>>> unk(EINT_VAR)=5320601.064666561 PE=0 block=18 type=1
>>>>> [ 04-08-2009 18:10:43.112 ] [gr_hgRecordNodeTypeState] max refine level
>>>>> =: 5
>>>>> [ 04-08-2009 18:10:43.161 ] WARNING after gc filling: min.
>>>>> unk(DENS_VAR)=-0.7514521682254683594983E-21 PE=0 block=8 type=2
>>>>> [ 04-08-2009 18:10:43.161 ] WARNING after gc filling: min.
>>>>> unk(DENS_VAR)=-0.1537578263572373951611E-20 PE=0 block=13 type=2
>>>>>
>>>>>
>>>>>
>>>>> Another snippet from the log file when the crash occurs (more frequent
>>>>> for cooling runs).
>>>>>
>>>>> [ 04-08-2009 17:51:23.737 ] [IO writePlotfile] open: type=plotfile
>>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>>> [ 04-08-2009 17:51:23.738 ] [IO_writePlotfile] close: type=plotfile
>>>>> name=xSH-A1_hdf5_plt_cnt_0000
>>>>> [ 04-08-2009 17:51:23.738 ] [Driver_evolveFlash]: Entering evolution
>>>>> loop
>>>>> [ 04-08-2009 17:51:23.739 ] step: n=1 t=0.000000E+00 dt=1.000000E+02
>>>>> [ 04-08-2009 17:51:23.776 ] [gr_hgRecordNodeTypeState] max refine level
>>>>> =: 4
>>>>> [Eos_wrapped] ERROR Density or Internal Energy are zero after a call to
>>>>> EOS!
>>>>> [ 04-08-2009 17:51:23.816 ] [DRIVER_ABORT]: Driver_abort() called by
>>>>> PE 0
>>>>> [ 04-08-2009 17:51:23.816 ] abort_message: [Eos_wrapped] ERROR Density
>>>>> or Internal Energy are zero after a call to EOS!
>>>>>
>>>>>
>>>>
>>>>
>>
More information about the flash-users
mailing list