[FLASH-USERS] Magnetic fields at refinement boundaries in USM

[Yi-Hao Chen]

Hi Dongwook,

Thanks for the explanation about the HLLC solver. I really appreciate your efforts on developing the code and answering all the questions.

In my experience, the anomalies appear only at the boundaries between two different refinement levels, especially where the refinement levels stay the same for a long time. They seem to dissipate when one side refines or derefines to the same level as the other. That leads me to think that the interpolation (or the restriction) of the guardcells in PARAMESH might be the problem (involving mpi_amr_1blk_guardcell_c_to_f), particularly for the magnetic face variables as you suspect. I am wondering if other people have seen similar artifacts at other places.

However, following up my suspicion, I tried setting interp_mask_face[x,y,z] to 0, but it doesn't seem to help. There is also a divergence cleaning subroutine amr_1blk_fc_clean_div, but it is not used (prol_fc_clean_divb is set to .false.).

I have spent a lot of time trying to find the root of the problem. If there is any discussion going on among FLASH developers, I would be happy to contribute my experience on this bug.

Thanks,

Yi-Hao


On Wed, Mar 27, 2019 at 9:39 PM Dongwook Lee <dongwook at flash.uchicago.edu<mailto:dongwook at flash.uchicago.edu>> wrote:
Dear Yi-Hao,

As John already has pointed out, this is a known problem for years. As the USM code developer, I tried my best to debug this issue in the past but no gain.
I suspect the issue could be related to how PARAMESH (the default AMR library in FLASH) operates for the magnetic face variables as the refinement levels progressively change during simulations.

Having said that, I am writing this email to clarify some of the concerns you mentioned in your previous email.

On Tue, Mar 26, 2019 at 11:12 AM Yi-Hao Chen <ychen at astro.wisc.edu<mailto:ychen at astro.wisc.edu>> wrote:
Hi John,

Thanks for getting back to me quickly.

I thought that HLLC solve is primarily for Hydro simulations. I am using the Hybrid solver in MHD, so that should include HLLD or HLL solvers.


Although HLLC only resolves three waves (i.e., two fast shocks and a contact discontinuity) and is missing the rest MHD waves (i.e., two Alfven waves and two slow shocks), you can still use HLLC in your MHD simulations. The HLLC solver in FLASH implements the MHD version of HLLC by S. Li (https://www.sciencedirect.com/science/article/pii/S0021999104003857) which is generally good for most practical cases. As John mentioned, the use of HLLC over HLLD may reduce/delay the anomalies. You might want to just use HLLC (or HLL) only instead of the hybrid Riemann solver just to avoid any further inconsistency in combining the two different Riemann solvers at fine-coarse block boundaries.

As I have left the Flash Center, I cannot fully commit myself to debuging this issue, but it will be great if this problem is resolved in the near future (by the current Flash team or by me).

Best,
Dongwook

The Balsara interpolation affects only the prolongation process, i.e. when interpolating newly created finer blocks, and has no effect on the guardcell filling between coarser and finer block. The guardcell filling process for the face-center fields seems to use neither direction injection nor Balsara injection. It is coded to uses linear interpolation and does not have a runtime parameter for other choices, which puzzles me. I am wondering the motivation behind this choice and looking for insights whether this might cause the problem.

Best,
Yi-Hao
<mailto:ychen at astro.wisc.edu>


On Tue, Mar 26, 2019 at 12:07 PM John Zuhone <jzuhone at gmail.com<mailto:jzuhone at gmail.com>> wrote:
Hi Yi-Hao,

This is a known problem, and you are likely seeing the same issue that you saw on the mailing list earlier.

Sometimes it helps to use the HLLC solver instead of HLLD, though it does not remove this artifact entirely in my experience.

There is also balsara interpolation for magnetic fields, but I have found that neither this or the straight injection interpolation helps either.

Best,

John

On Mar 26, 2019, at 1:01 PM, Yi-Hao Chen <ychen at astro.wisc.edu<mailto:ychen at astro.wisc.edu>> wrote:

Dear FLASH community,

I have been trying to find the cause of a refinement boundary problem when using USM solver in cartesian 3D simulations. The problem appears as the accumulation of magnetic fields at the finer edge at the refinement boundary. I am not sure how to reproduce the problem in a simple setting, but this seems to be prevalent in the simulations that I have been running. Usually, the anomalies dissipated after many timesteps and went unnoticed. However, it occasionally causes a few cells to drop to extremely small or even negative density and crashes the simulation. I am wondering if anyone has seen similar behaviors.

Here is a snapshot showing the By field in x-y plane. The quiver arrows indicate the fluid velocity.
<Group_L438_hdf5_plt_cnt_0339_Slice_z_magnetic_field_y.png>

Some relavent parameters I used: RiemannSolver="Hybrid", order=3, slopelimiter="mc", CFL=0.4, energfix = .true.

A few solutions that I have tried:
1. Lower CFL to 0.2
-> does not help
2. Use E_upwind = .true.
-> Enabling upwind scheme for E fields does seem to lower the frequency that the anomaly happens, but does not get rid of the problem completely.
3. Refine or derefine the neighboring block to make the interface not at the coarse-fine block boundary.
-> This does prevent or dissipate the anomaly but does not seem to be a good general solution and I don't know how to identify the problem block on the fly.

A similar problem was discussed on the mailing list in 2016, although I am not sure the problem is the same.
http://flash.uchicago.edu/pipermail/flash-users/2016-May/001962.html
Following the discussion there, I found that the interpolation method for face-center variables in guardcells is set by interp_mask_face[x,y,z]. They are initialized to be 1 (linear) in amr_initialize.F90. During the creation of new child blocks, they are temporarily set to 0 (if prolMethod="injection_prol" as default) in Simulation_customizeProlong.F90, and are reverted to the old value after the prolongation.

I have two questions here:
1. Why does FLASH use two different interpolation methods during (a) prolongation, or the creation of child blocks, and (b) guardcell filling from coarse to fine block? In the discussion linked above, it was mentioned that using different treatments was based on experience with applications. I would like to learn more about this if possible.
2. It appears to me that the direct injection (0th order) is used in prolongation because it simply preserves the divergence-free nature of the B fields. I am not sure, but I suspect linear interpolation does not always preserve the divergence-free fields. Thus, wouldn't it be a problem using linear interpolation for guardcell filling?

Any suggestions or possible directions to look into are very much appreciated.

Best Regards,
Yi-Hao




--

=========================================
Dongwook Lee, Ph.D., Associate Professor
Applied Mathematics
University of California, Santa Cruz
Baskin Engineering, Room 353C
1156 High Street, Santa Cruz, CA 95064
https://users.soe.ucsc.edu/~dongwook/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20190328/59cb033d/attachment-0001.htm>