[FLASH-USERS] Negative/zero densities and internal energies after filling gc

[Ross Parkin]

Hello again,

The source term I've been using is a bit like the Stir unit; it  
applies a driving force to a fluid flow. I had set the base grid to be  
8 by 8 blocks in 2D, and during some tests I noticed that the values  
for the driving force in unrefined blocks were zeros. This meant that  
the source term was not being applied to unrefined blocks. My 'quick  
fix' was to call for a list of ALL_BLKS at the start of the source term.

Regards,

Ross



Quoting Tomasz Plewa <tplewa at fsu.edu>:

>        If it is not a mesh problem, solver is to blame for requesting
> invalid data.
>
> So I remain curious why solvers in Seyit and Ross problems need those
> data.
>
> Tomek
> --
> Kevin Olson wrote:
>
>   All,I have one comment on the use of "advance_all_levels" in
> Paramesh. This option only has an effect on guardcell filling.  If it
> is set to .true. Paramesh will assume that parent block data which is
> needed for guardcell  filling has been updated by the user (i.e. FLASH
> in this case) rather than doing a default interpolation to the parent
> block. Since, FLASH does not guarantee that parent blocks will have
> valid data (as far as I know) then setting "advance_all_levels" to
> .true. could do more harm that good.Kevin OlsonOn Apr 10, 2009, at
> 11:54 AM, Anshu Dubey wrote:
>
>> Seyit and RossFLASH evolves the solution only on the leaf blocks,
>> and as such all the provided solvers are guaranteed to be applied
> to
>> only the leaf blocks. If parents or ancestors have any valid data,
>> it is because of restriction from their children (with some very
>> minor exceptions). If you are interested in evolving at all levels,
>> you will have make sure that other solvers also operate on all
>> blocks, by changing \"LEAF\" to \"ALLBLOCKS\" in calls to
>> Grid_getListOfBlocks. And you will have to enable the
>> \"advance_all_levels\" mode in Paramesh. However, you should be
>> aware that some of the solvers cannot be trivially changed in this
>> way, for example active particles and self gravity.There will be
>> some other changes necessary, since different parts of FLASH
> operate
>> under the assumption that only leaf blocks are propagated. At the
>> least, various calls to Grid_restrictAllLevels in IO_* subroutines
>> and a call to amr_restrict in Grid_updateRefinement.F90 should be
>> removed or disabled.If it isn't really necessary to evolve the
>> solution on all levels at the same time, we recommend that you
> stick
>> with evolving only on the leaf blocks - especially if this is the
>> first time using FLASH3We were able to reproduce negative values in
>> density in the Jeans problem in some of the guardcells of parent
>> blocks after changing the refinement criterion. Those guardcell
>> values are coming from the fine-coarse boundaries where same level
>> neighbor of a \"parent\" block is an \"ancestor\" block. Ancestor
>> blocks do not always  have valid values. These guardcells have no
>> impact on solution as long as you are advancing on LEAF blocks
> only,
>> so they can be  safely ignored.   We verified by filling the
>> ancestor blocks with zeros, and in the parents blocks in question,
>> the zeroes showed up in the exact same pattern that we had seen
> with
>> the -ve values.We don't yet understand why these warnings are
>> triggered in the case of Jeans but not in many other problems.  To
>> reduce or suppress the annoying messages, you could to either of
> two
>> things: 1) Remove or comment out the line '#define DEBUG_CONSCONV'
>> in gr_sanitizeDataAfterInterp.F90.  You should then see only
>> messages with one line per block / variable on standard output, no
>> dumping of the blocks contents. 2) Change
> gr_sanitizeDataAfterInterp
>> so it only checks leaf blocks, not leaf and parent blocks. That is,
>> you might suppressing the checking or the output if
>> (nodetype(block)==2) .Although we didn't see any negative density
>> values in the interior of parent blocks, or anywhere in the LEAF
>> blocks, if you are getting them, then there is cause for concern. I
>> wouldn't simply take absolute values, because something else may be
>> going wrong with your simulation. You cannot rely on those
>> results.We hope that the above explanation will be helpful in
>> solving your problems.  If you are seeing  some behavior that is
>> different from what I described above, we'd like to know more about
>> it.The flash code group.On Thu, Apr 9, 2009 at 11:49 AM, Seyit
> Hocuk
>>  href=\"mailto:seyit at astro.rug.nl\"><seyit at astro.rug.nl>
>> wrote:
>>
>>   Hi,Ross, I tried but your trick didn't work unfortunately. It
>> even made it more instable as it seemed. Tomek might be right.To
>> everybody,I've reduced the problem a bit. Jeans' specific
> refinement
>> is bad. I think it doesn't work properly, so I commented some of
> the
>> refinement criteria in Grid_markRefineDerefine and used
>> Grid_markRefineSpecialized for the threshold criteria instead,
> which
>> works fine. So that's solved.I also set convertToConsvdForMeshCalls
>> to .True. in runtime parameters instead of using
>> convertToConsvdInMeshInterp, which should have worked better
> because
>> it is supposed to avoid spurius things in paramesh3+. However
>> convertToConsvdInMeshInterp calls at some point
>> gr_sanitizeDataAfterInterp, which creates all the weirdness. It is
>> my humble opinion that something goes wrong with conservation
>> there.Lastly, I did a very bad thing to solve the negative
>> densities. In Eos_wrapped, where the densities become negative
> after
>> getting the density from solnData (it seems the value is ok, just
>> the sign changed), I use the absolute value to get passed that
>> problem
>> (ASB(solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)))).
> I
>> know this isn't the way, but everything looks ok for now. Until a
>> better solution is found, I will continue like this.With my limited
>> knowledge and understanding of the code, I could only come up with
>> this trick.Can anybody come up with a better solution or has
> anybody
>> else implemented cooling successfully?Thanks, SeyitTomasz Plewa
>> wrote:
>>
>>   This would imply there is an inconsistency in applying physics
>> operators.I think there exist a (PARAMESH?) option enforcing
>> evolving all blocks in the simulation, if necessary. A local
>> modification might be more efficient, though.In the latter case,
> the
>> next question I would have is whether all blocks coarser than
> parent
>> blocks do have valid data.Tomek -- Ross Parkin wrote:
>>
>>
>>   Hi Seyit,Quoting Seyit Hocuk
> href=\"mailto:seyit at astro.rug.nl\"><seyit at astro.rug.nl>:
>>
>>
>>   Hi community,I've switched to Flash3.1.1a (from version 2.5), but
>> when I try to run my setup, comparable to the Jeans setup, I
>> encounter problems. If I am running without cooling and with a
>> uniform grid (lref_max=lref_min), then everything seems ok.
> However,
>> if I add the cooling module OR I have adaptive refinement, zero or
>> negative densities/internal energies appear (or lower than the
>> \"small\" values). These cause the refinement to be messed up and
> of
>> course leads to crashes. The error outpusts are shown below. From
>> there, I notice that the error might come from guard cell filling.I
>> test the standard Jeans setup and the same problems occur. Just
>> changing and adding in flash.par: lrefine_max = 5 refine_cutoff_1 =
>> 1.0On several occasions I have checked the Eos_wrapped.F90 file and
>> found the location where it might go wrong; just before reading
>> solnData eint/dens was ok, but after this the values are crazy.
>> Btw., I don't use massFractions ad haven't implemented Multispecies
>> and no species are initiated.eosData(pres+1:pres+vecLen) = &
>> solnData(PRES_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>> eosData(dens+1:dens+vecLen) = &
>> solnData(DENS_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>> eosData(temp+1:temp+vecLen) = &
>> solnData(TEMP_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>> eosData(gamc+1:gamc+vecLen) = &
>> solnData(GAMC_VAR,range(LOW,IAXIS):range(HIGH,IAXIS),j,k)
>> eosData(eint+1:eint+vecLen) =
>>
> energyInternal(1:vecLen)callEos(mode,vecLen,eosData)!,massFraction)!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>> !!!!!!!!!!!!!!!!!!!!!!Another thing that I notice but is likely
>> completely unrelated is, that with this latest Flash version, the
>> upper level blocks are blocky and have absurd values. Though, the
>> highest refined level is always ok. Could it be related, as it
> needs
>> parent blocks for refinement? Then again, how is this related to
>> cooling when there is no refinement. I double checked with Jeans
> and
>> Sedov test problems; it was the same thing.
>
>   I can't help you with the Jeans simulation but I have noticed a
> similar problem with the coarsest level blocks when you are using
> source terms. Specifically, my problem was that blocks on the coarsest
> refinement level were not being acted on by source terms. I tracked
> down the root of this problem; it seems that the blockList passed from
> Driver_evolveFlash into Driver_sourceTerms is only for leaf blocks and
>  not all blocks. You can get around this by adding a call to
> Grid_getlistofblocks and requesting all blocks before you start
> applying source terms. I hope this helps.Cheers,Ross Parkin
>
>> Some info on my environment: Kernel: Centos 5.2 Machine: Linux
>> ulubey 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008
> x86_64
>> x86_64 x86_64 GNU/Linux ifort version: (IFORT) 10.1 20080602 hdf5
>> version: 1.8.1 MPICH version. 2-1.0.7 gcc version 4.1.2 20071124
>> (Red Hat 4.1.2-42)A snippet from the log file when the warnings
>> appear.[GRID amr_refine_derefine] min leaf blks 76    max leaf blks
>> 76 tot leaf blks 76 [ 04-08-2009  18:10:42.268 ] [GRID
>> amr_refine_derefine]: refinement complete [ 04-08-2009
> 18:10:42.273
>> ] WARNING after amr_prolong: min. unk(EINT_VAR)=6685180.741266931
>> PE=0 block=18 type=1 [ 04-08-2009  18:10:42.274 ] step: n=37
>> t=1.414748E+12 dt=1.000000E+11 [ 04-08-2009  18:10:42.279 ] WARNING
>> after gc filling: min. unk(EINT_VAR)=6162624.299265089 PE=0
> block=18
>> type=1 [ 04-08-2009  18:10:42.297 ] WARNING after gc filling: min.
>> unk(EINT_VAR)=6174477.607051291 PE=0 block=18 type=1 . . [
>> 04-08-2009  18:10:42.939 ] WARNING after gc filling: min.
>> unk(EINT_VAR)=6274674.035376545 PE=0 block=32 type=1 [ 04-08-2009
>> 18:10:42.939 ] WARNING after gc filling: min.
>> unk(EINT_VAR)=6259353.579602077 PE=0 block=33 type=1 [ 04-08-2009
>> 18:10:42.992 ] [gr_hgRecordNodeTypeState] max refine level =: 5 [
>> 04-08-2009  18:10:43.054 ] [IO_writeCheckpoint] open:
>> type=checkpoint name=xSH-A1_hdf5_chk_0002 [ 04-08-2009
> 18:10:43.059
>> ] [IO_writeCheckpoint] close: type=checkpoint
>> name=xSH-A1_hdf5_chk_0002 [ 04-08-2009  18:10:43.059 ] step: n=40
>> t=2.014748E+12 dt=1.000000E+11 [ 04-08-2009  18:10:43.064 ] WARNING
>> after gc filling: min. unk(EINT_VAR)=5604765.671211276 PE=0
> block=18
>> type=1 [ 04-08-2009  18:10:43.083 ] WARNING after gc filling: min.
>> unk(EINT_VAR)=5320601.064666561 PE=0 block=18 type=1 [ 04-08-2009
>> 18:10:43.112 ] [gr_hgRecordNodeTypeState] max refine level =: 5 [
>> 04-08-2009  18:10:43.161 ] WARNING after gc filling: min.
>> unk(DENS_VAR)=-0.7514521682254683594983E-21 PE=0 block=8 type=2 [
>> 04-08-2009  18:10:43.161 ] WARNING after gc filling: min.
>> unk(DENS_VAR)=-0.1537578263572373951611E-20 PE=0 block=13
>> type=2Another snippet from the log file when the crash occurs (more
>> frequent for cooling runs).[ 04-08-2009  17:51:23.737 ] [IO
>> writePlotfile] open: type=plotfile name=xSH-A1_hdf5_plt_cnt_0000 [
>> 04-08-2009  17:51:23.738 ] [IO_writePlotfile] close: type=plotfile
>> name=xSH-A1_hdf5_plt_cnt_0000 [ 04-08-2009  17:51:23.738 ]
>> [Driver_evolveFlash]: Entering evolution loop [ 04-08-2009
>> 17:51:23.739 ] step: n=1 t=0.000000E+00 dt=1.000000E+02 [
> 04-08-2009
>>  17:51:23.776 ] [gr_hgRecordNodeTypeState] max refine level =: 4
>> [Eos_wrapped] ERROR Density or Internal Energy are zero after a
> call
>> to EOS! [ 04-08-2009  17:51:23.816 ] [DRIVER_ABORT]: Driver_abort()
>> called by PE           0 [ 04-08-2009  17:51:23.816 ]
> abort_message:
>> [Eos_wrapped] ERROR Density or Internal Energy are zero after a
> call
>> to EOS!
>
>