<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
Dear Yi-Hao / All,<br>
<br>
Just to quickly chime in I am 100% certain this is the issue I reported in 2016. I got as far as convincing myself it was an issue at boundaries where magnetic field has an oblique component relative to the boundary. If you read the chain of mail from there,
one of the tests showed it only occurs when you have a magnetic field gradient oblique to the Cartesian grid (1D current sheer in force balance, shows these artefacts once it's rotated relative to the Cartesian grid). Something about the interpolation/refinement
in the ghosts seems to numerically break the force balance.<br>
<br>
This can be shown for a "fixed mesh" where automatic refinement /dererinement was turned off. this supports what Yi-Hao says, it seemed to be to do with the interpolation/injection on coarse / fine boundaries in general, as opposed to something that only occurs
as refinement levels change. I imagine it would also happen as boundaries change but felt that it would be harder to notice these artefacts there.<br>
<br>
Unfortunately, I never got any further fixing it so can't be much help. It's a pretty serious issue for null point reconnection problems in my opinion and I ultimately decided flash wasn't the right tool for the job. That was very disappointing because otherwise
the code has some features that would have been excellent.... I also wonder if similar AMR + finite volume MHD codes exhibit this issue.<br>
<br>
Cheers, <br>
<br>
Jonathan
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> flash-users-bounces@flash.uchicago.edu <flash-users-bounces@flash.uchicago.edu> on behalf of Yi-Hao Chen <ychen@astro.wisc.edu><br>
<b>Sent:</b> 28 March 2019 22:41:18<br>
<b>To:</b> Dongwook Lee<br>
<b>Cc:</b> flash-users@flash.uchicago.edu<br>
<b>Subject:</b> Re: [FLASH-USERS] Magnetic fields at refinement boundaries in USM</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Dongwook,
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668m_-9119656752045206656gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div>Thanks for the explanation about the HLLC solver. I really appreciate your efforts on developing the code and answering all the questions.</div>
<div><br>
</div>
<div>In my experience, the anomalies appear only at the boundaries between two different refinement levels, especially where the refinement levels stay the same for a long time. They seem to dissipate when one side refines or derefines to the same level as
the other. That leads me to think that the interpolation (or the restriction) of the guardcells in PARAMESH might be the problem (involving
<span style="font-family:monospace,monospace">mpi_amr_1blk_guardcell_c_to_f</span>), particularly for the magnetic face variables as you suspect. I am wondering if other people have seen similar artifacts at other places.<br>
</div>
<div><br>
</div>
<div>However, following up my suspicion, I tried setting <span style="font-family:monospace,monospace">
interp_mask_face[x,y,z]</span> to 0, but it doesn't seem to help. There is also a divergence cleaning subroutine
<span style="font-family:monospace,monospace">amr_1blk_fc_clean_div</span>, but it is not used (<span style="font-family:monospace,monospace">prol_fc_clean_divb</span> is set to .false.).<br>
</div>
</div>
<div><br>
</div>
<div>I have spent a lot of time trying to find the root of the problem. If there is any discussion going on among FLASH developers, I would be happy to contribute my experience on this bug.<br>
<br>
Thanks,</div>
<div><br>
</div>
<div>Yi-Hao</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><span style="font-family:monospace,monospace"><br>
</span></div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Wed, Mar 27, 2019 at 9:39 PM Dongwook Lee <<a href="mailto:dongwook@flash.uchicago.edu" target="_blank">dongwook@flash.uchicago.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Dear Yi-Hao,
<div><br>
</div>
<div>As John already has pointed out, this is a known problem for years. As the USM code developer, I tried my best to debug this issue in the past but no gain. </div>
<div>I suspect the issue could be related to how PARAMESH (the default AMR library in FLASH) operates for the magnetic face variables as the refinement levels progressively change during simulations.</div>
<div><br>
</div>
<div>Having said that, I am writing this email to clarify some of the concerns you mentioned in your previous email.<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Mar 26, 2019 at 11:12 AM Yi-Hao Chen <<a href="mailto:ychen@astro.wisc.edu" target="_blank">ychen@astro.wisc.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div>Hi John,</div>
<div><br>
</div>
<div>Thanks for getting back to me quickly.</div>
<div><br>
</div>
<div>I thought that HLLC solve is primarily for Hydro simulations. I am using the Hybrid solver in MHD, so that should include HLLD or HLL solvers.<br>
</div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Although HLLC only resolves three waves (i.e., two fast shocks and a contact discontinuity) and is missing the rest MHD waves (i.e., two Alfven waves and two slow shocks), you can still use HLLC in your MHD simulations. The HLLC solver in FLASH implements
the MHD version of HLLC by S. Li (<a href="https://www.sciencedirect.com/science/article/pii/S0021999104003857" target="_blank">https://www.sciencedirect.com/science/article/pii/S0021999104003857</a>) which is generally good for most practical cases. As John
mentioned, the use of HLLC over HLLD may reduce/delay the anomalies. You might want to just use HLLC (or HLL) only instead of the hybrid Riemann solver just to avoid any further inconsistency in combining the two different Riemann solvers at fine-coarse block
boundaries.</div>
<div><br>
</div>
<div>As I have left the Flash Center, I cannot fully commit myself to debuging this issue, but it will be great if this problem is resolved in the near future (by the current Flash team or by me).</div>
<div><br>
</div>
<div>Best,</div>
<div>Dongwook</div>
<div><br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"></div>
<div>The Balsara interpolation affects only the prolongation process, i.e. when interpolating newly created finer blocks, and has no effect on the guardcell filling between coarser and finer block. The guardcell filling process for the face-center fields seems
to use neither direction injection nor Balsara injection. It is coded to uses linear interpolation and does not have a runtime parameter for other choices, which puzzles me. I am wondering the motivation behind this choice and looking for insights whether
this might cause the problem.<br>
</div>
<div> <br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div></div>
<div>Best,</div>
<div>Yi-Hao<br>
</div>
<div dir="ltr"><a href="mailto:ychen@astro.wisc.edu" target="_blank"></a></div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Mar 26, 2019 at 12:07 PM John Zuhone <<a href="mailto:jzuhone@gmail.com" target="_blank">jzuhone@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>Hi Yi-Hao,
<div><br>
</div>
<div>This is a known problem, and you are likely seeing the same issue that you saw on the mailing list earlier. </div>
<div><br>
</div>
<div>Sometimes it helps to use the HLLC solver instead of HLLD, though it does not remove this artifact entirely in my experience. </div>
<div><br>
</div>
<div>There is also balsara interpolation for magnetic fields, but I have found that neither this or the straight injection interpolation helps either. </div>
<div><br>
</div>
<div>Best,</div>
<div><br>
</div>
<div>John</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Mar 26, 2019, at 1:01 PM, Yi-Hao Chen <<a href="mailto:ychen@astro.wisc.edu" target="_blank">ychen@astro.wisc.edu</a>> wrote:</div>
<br class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315Apple-interchange-newline">
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Dear FLASH community,
<div><br>
</div>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>
<div>I have been trying to find the cause of a refinement boundary problem when using USM solver in cartesian 3D simulations. The problem appears as the accumulation of magnetic fields at the finer edge at the refinement boundary. I am not sure how to reproduce
the problem in a simple setting, but this seems to be prevalent in the simulations that I have been running. Usually, the anomalies dissipated after many timesteps and went unnoticed. However, it occasionally causes a few cells to drop to extremely small or
even negative density and crashes the simulation. I am wondering if anyone has seen similar behaviors.</div>
</div>
<div><br>
</div>
<div>Here is a snapshot showing the By field in x-y plane. The quiver arrows indicate the fluid velocity.<br>
</div>
<div>
<div><span id="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315cid:ii_jtpy5byy0"><Group_L438_hdf5_plt_cnt_0339_Slice_z_magnetic_field_y.png></span>
<br>
</div>
</div>
<div><br>
</div>
<div>
<div>Some relavent parameters I used: RiemannSolver="Hybrid", order=3, slopelimiter="mc", CFL=0.4, energfix = .true.</div>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>A few solutions that I have tried:</div>
<div>1. Lower CFL to 0.2 <br>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px 40px; border:medium none; padding:0px">
<div>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>-> does not help</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>2. Use E_upwind = .true. </div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px 40px; border:medium none; padding:0px">
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>-> Enabling upwind scheme for E fields does seem to lower the frequency that the anomaly happens, but does not get rid of the problem completely.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>3. Refine or derefine the neighboring block to make the interface <span class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-Punctuation x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-only-ins x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr-alert x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr_21 x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr_gramm x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr_ x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr_inline_cards x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-gr_disable_anim_appear x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-replaceWithoutSep" id="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail-21" style="display:inline; border-bottom:2px solid transparent; background-repeat:no-repeat; color:inherit; font-size:inherit">not</span> at
the coarse-fine block boundary. </div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px 40px; border:medium none; padding:0px">
<div>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail-m_4868212182492332594gmail-m_9206329192388601315m_6055753200755198736gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div>-> This does prevent or dissipate the anomaly but does not seem to be a good general solution and I don't know how to identify the problem block on the fly.</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
<div>A similar problem was discussed on the mailing list in 2016, although I am not sure the problem is the same.</div>
<div><a href="http://flash.uchicago.edu/pipermail/flash-users/2016-May/001962.html" target="_blank">http://flash.uchicago.edu/pipermail/flash-users/2016-May/001962.html</a></div>
<div>Following the discussion there, I found that the interpolation method for face-center variables in guardcells is set by
<font face="monospace, monospace">interp_mask_face[x,y,z]</font>. They are initialized to be 1 (linear) in
<font face="monospace, monospace">amr_initialize.F90</font>. During the creation of new child blocks, they are temporarily set to 0 (if prolMethod="injection_prol" as default) in <font face="monospace, monospace">Simulation_customizeProlong.F90</font>, and
are reverted to the old value after the prolongation.</div>
<div><br>
</div>
<div>I have two questions here:</div>
<div>1. Why does FLASH use two different interpolation methods during (a) prolongation, or the creation of child blocks, and (b) guardcell filling from coarse to fine block? In the discussion linked above, it was mentioned that using different treatments was
based on experience with applications. I would like to learn more about this if possible.</div>
<div>2. It appears to me that the direct injection (0th order) is used in prolongation because it simply preserves the divergence-free nature of the B fields. I am not sure, but I suspect linear interpolation does not always preserve the divergence-free fields.
Thus, wouldn't it be a problem using linear interpolation for guardcell filling?</div>
<div><br>
</div>
<div>Any suggestions or possible directions to look into are very much appreciated.<br>
</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Yi-Hao</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail-m_7847993222019213138gmail-m_-1562988082618715668gmail-m_-9119656752045206656gmail-m_4194286075534737573gmail-m_1967926401830040057gmail_signature">
<div dir="ltr">
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000"><br>
=========================================</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000">Dongwook Lee, Ph.D., Associate Professor</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000">Applied Mathematics<br>
</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000">University of California, Santa Cruz</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000">Baskin Engineering, Room 353C</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000">1156 High Street, Santa Cruz, CA 95064</font></div>
<div style="font-size:12.8px"><font size="2" face="arial, helvetica, sans-serif" color="#000000"><a href="https://users.soe.ucsc.edu/~dongwook/" target="_blank">https://users.soe.ucsc.edu/~dongwook/</a></font></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
This message is intended solely for the addressee and may contain confidential and/or legally privileged information. Any use, disclosure or reproduction without the sender’s explicit consent is unauthorised and may be unlawful. If you have received this message
in error, please notify Northumbria University immediately and permanently delete it. Any views or opinions expressed in this message are solely those of the author and do not necessarily represent those of the University. Northumbria University email is provided
by Microsoft Office365 and is hosted within the EEA, although some information may be replicated globally for backup purposes. The University cannot guarantee that this message or any attachment is virus free or has not been intercepted and/or amended.
</body>
</html>