[FLASH-USERS] Magnetic fields at refinement boundaries in USM

Fri Mar 29 03:21:56 EDT 2019

Dear Yi-Hao / All,

Just to quickly chime in I am 100% certain this is the issue I reported in 2016. I got as far as convincing myself it was an issue at boundaries where magnetic field has an oblique component relative to the boundary. If you read the chain of mail from there, one of the tests showed it only occurs when you have a magnetic field gradient oblique to the Cartesian grid (1D current sheer in force balance, shows these artefacts once it's rotated relative to the Cartesian grid).  Something about the interpolation/refinement in the ghosts seems to numerically break the force balance.

This can be shown for a "fixed mesh" where automatic refinement /dererinement was turned off. this supports what Yi-Hao says, it seemed to be to do with the interpolation/injection on coarse / fine boundaries in general, as opposed to something that only occurs as refinement levels change. I imagine it would also happen as boundaries change but felt that it would be harder to notice these artefacts there.

Unfortunately, I never got any further fixing it so can't be much help. It's a pretty serious issue for null point reconnection problems in my opinion and I ultimately decided flash wasn't the right tool for the job. That was very disappointing because otherwise the code has some features that would have been excellent.... I also wonder if similar AMR + finite volume MHD codes exhibit this issue.

Cheers,

Jonathan
________________________________
From: flash-users-bounces at flash.uchicago.edu <flash-users-bounces at flash.uchicago.edu> on behalf of Yi-Hao Chen <ychen at astro.wisc.edu>
Sent: 28 March 2019 22:41:18
To: Dongwook Lee
Cc: flash-users at flash.uchicago.edu
Subject: Re: [FLASH-USERS] Magnetic fields at refinement boundaries in USM

Hi Dongwook,

Thanks for the explanation about the HLLC solver. I really appreciate your efforts on developing the code and answering all the questions.

In my experience, the anomalies appear only at the boundaries between two different refinement levels, especially where the refinement levels stay the same for a long time. They seem to dissipate when one side refines or derefines to the same level as the other. That leads me to think that the interpolation (or the restriction) of the guardcells in PARAMESH might be the problem (involving mpi_amr_1blk_guardcell_c_to_f), particularly for the magnetic face variables as you suspect. I am wondering if other people have seen similar artifacts at other places.

However, following up my suspicion, I tried setting interp_mask_face[x,y,z] to 0, but it doesn't seem to help. There is also a divergence cleaning subroutine amr_1blk_fc_clean_div, but it is not used (prol_fc_clean_divb is set to .false.).

I have spent a lot of time trying to find the root of the problem. If there is any discussion going on among FLASH developers, I would be happy to contribute my experience on this bug.

Thanks,

Yi-Hao

On Wed, Mar 27, 2019 at 9:39 PM Dongwook Lee <dongwook at flash.uchicago.edu<mailto:dongwook at flash.uchicago.edu>> wrote:
Dear Yi-Hao,

As John already has pointed out, this is a known problem for years. As the USM code developer, I tried my best to debug this issue in the past but no gain.
I suspect the issue could be related to how PARAMESH (the default AMR library in FLASH) operates for the magnetic face variables as the refinement levels progressively change during simulations.

Having said that, I am writing this email to clarify some of the concerns you mentioned in your previous email.

On Tue, Mar 26, 2019 at 11:12 AM Yi-Hao Chen <ychen at astro.wisc.edu<mailto:ychen at astro.wisc.edu>> wrote:
Hi John,

Thanks for getting back to me quickly.

I thought that HLLC solve is primarily for Hydro simulations. I am using the Hybrid solver in MHD, so that should include HLLD or HLL solvers.

Although HLLC only resolves three waves (i.e., two fast shocks and a contact discontinuity) and is missing the rest MHD waves (i.e., two Alfven waves and two slow shocks), you can still use HLLC in your MHD simulations. The HLLC solver in FLASH implements the MHD version of HLLC by S. Li (https://www.sciencedirect.com/science/article/pii/S0021999104003857) which is generally good for most practical cases. As John mentioned, the use of HLLC over HLLD may reduce/delay the anomalies. You might want to just use HLLC (or HLL) only instead of the hybrid Riemann solver just to avoid any further inconsistency in combining the two different Riemann solvers at fine-coarse block boundaries.

As I have left the Flash Center, I cannot fully commit myself to debuging this issue, but it will be great if this problem is resolved in the near future (by the current Flash team or by me).

Best,
Dongwook

The Balsara interpolation affects only the prolongation process, i.e. when interpolating newly created finer blocks, and has no effect on the guardcell filling between coarser and finer block. The guardcell filling process for the face-center fields seems to use neither direction injection nor Balsara injection. It is coded to uses linear interpolation and does not have a runtime parameter for other choices, which puzzles me. I am wondering the motivation behind this choice and looking for insights whether this might cause the problem.

Best,
Yi-Hao
<mailto:ychen at astro.wisc.edu>

On Tue, Mar 26, 2019 at 12:07 PM John Zuhone <jzuhone at gmail.com<mailto:jzuhone at gmail.com>> wrote:
Hi Yi-Hao,

This is a known problem, and you are likely seeing the same issue that you saw on the mailing list earlier.

Sometimes it helps to use the HLLC solver instead of HLLD, though it does not remove this artifact entirely in my experience.

There is also balsara interpolation for magnetic fields, but I have found that neither this or the straight injection interpolation helps either.

Best,

John

On Mar 26, 2019, at 1:01 PM, Yi-Hao Chen <ychen at astro.wisc.edu<mailto:ychen at astro.wisc.edu>> wrote:

Dear FLASH community,

I have been trying to find the cause of a refinement boundary problem when using USM solver in cartesian 3D simulations. The problem appears as the accumulation of magnetic fields at the finer edge at the refinement boundary. I am not sure how to reproduce the problem in a simple setting, but this seems to be prevalent in the simulations that I have been running. Usually, the anomalies dissipated after many timesteps and went unnoticed. However, it occasionally causes a few cells to drop to extremely small or even negative density and crashes the simulation. I am wondering if anyone has seen similar behaviors.

Here is a snapshot showing the By field in x-y plane. The quiver arrows indicate the fluid velocity.
<Group_L438_hdf5_plt_cnt_0339_Slice_z_magnetic_field_y.png>

Some relavent parameters I used: RiemannSolver="Hybrid", order=3, slopelimiter="mc", CFL=0.4, energfix = .true.

A few solutions that I have tried:
1. Lower CFL to 0.2
-> does not help
2. Use E_upwind = .true.
-> Enabling upwind scheme for E fields does seem to lower the frequency that the anomaly happens, but does not get rid of the problem completely.
3. Refine or derefine the neighboring block to make the interface not at the coarse-fine block boundary.
-> This does prevent or dissipate the anomaly but does not seem to be a good general solution and I don't know how to identify the problem block on the fly.

A similar problem was discussed on the mailing list in 2016, although I am not sure the problem is the same.
http://flash.uchicago.edu/pipermail/flash-users/2016-May/001962.html
Following the discussion there, I found that the interpolation method for face-center variables in guardcells is set by interp_mask_face[x,y,z]. They are initialized to be 1 (linear) in amr_initialize.F90. During the creation of new child blocks, they are temporarily set to 0 (if prolMethod="injection_prol" as default) in Simulation_customizeProlong.F90, and are reverted to the old value after the prolongation.

I have two questions here:
1. Why does FLASH use two different interpolation methods during (a) prolongation, or the creation of child blocks, and (b) guardcell filling from coarse to fine block? In the discussion linked above, it was mentioned that using different treatments was based on experience with applications. I would like to learn more about this if possible.
2. It appears to me that the direct injection (0th order) is used in prolongation because it simply preserves the divergence-free nature of the B fields. I am not sure, but I suspect linear interpolation does not always preserve the divergence-free fields. Thus, wouldn't it be a problem using linear interpolation for guardcell filling?

Any suggestions or possible directions to look into are very much appreciated.

Best Regards,
Yi-Hao

--

=========================================
Dongwook Lee, Ph.D., Associate Professor
Applied Mathematics
University of California, Santa Cruz
Baskin Engineering, Room 353C
1156 High Street, Santa Cruz, CA 95064
https://users.soe.ucsc.edu/~dongwook/

This message is intended solely for the addressee and may contain confidential and/or legally privileged information. Any use, disclosure or reproduction without the sender's explicit consent is unauthorised and may be unlawful. If you have received this message in error, please notify Northumbria University immediately and permanently delete it. Any views or opinions expressed in this message are solely those of the author and do not necessarily represent those of the University. Northumbria University email is provided by Microsoft Office365 and is hosted within the EEA, although some information may be replicated globally for backup purposes. The University cannot guarantee that this message or any attachment is virus free or has not been intercepted and/or amended.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20190329/d081f3c0/attachment.htm>