[FLASH-USERS] hangup with laser2Din3D
Norbert Flocke
flocke at flash.uchicago.edu
Fri Apr 22 16:10:01 EDT 2016
Mario,
You can try to increase MAXBLOCKS and see what happens.
Norbert
On Fri, 22 Apr 2016, Mario Manuel wrote:
> Hi Norbert,
>
> I have read the section you mentioned and tried to set the lens up in both ways. The ‘crash’, more of an infinite loop or something, occurs with both setups when run in 3Din2D set to true. However, as one might expect, the 2D lens definition works with 3Din2D set to false and produces the image that I attached in the last email.
>
> I get the same ‘crash’ behavior when running the LaserSlab simulation (using the setup in the manual) with the example.par setup which uses 3Din2D = true… and when I run a setup in cartesian, LaserSlab runs fine. So I don’ t think that this is something specific to my simulated laser setup or the packages I’ve implemented since LaserSlab doesn’t work with the default setup. I’ve run out of ideas for the moment on how to proceed, but any other thoughts are much appreciated. You say that there may be a memory issue. Should I increase MAXBLOCKS in the setup? Is there another flag that might help with the memory issue? I am running on a BG/Q system, Vulcan, at LLNL.
>
> Cheers,
> Mario
>
> ______________________________________
>
> Dr. Mario Manuel
> Einstein Postdoctoral Fellow, Smithsonian Astrophysical Observatory
> mmanuel at umich.edu
> 617-899-8381
>
> University of Michigan
> Space Research Building
> 2455 Hayward Street
> Ann Arbor, MI 48109
> Office: 734-764-4585
> ______________________________________
>
> On Apr 22, 2016, at 3:00 PM, Norbert Flocke <flocke at flash.uchicago.edu> wrote:
>
>> Hi Mario,
>>
>> From the comments right next to your ed_lens[X,Y,Z] runtime parameters (the X referring to radial position and the Y to axial position) I can see that you set up the laser beams in the original cylindrical 2D coordinates, which is what you should not do. The ed_lens[X,Y,Z] and the ed_target[X,Y,Z] must be set up using the expanded 3D space. The cylindrical domain is placed inside the 3D coordinate system such that
>> the 3D z axis corresponds now to the cylinder axis and the 3D (x,y) plane contains the circular cross sections of the cylinder. I would suggest you read section 16.4.7 '3D Laser Ray Tracing in 2D Cylindrical Symmetry' in
>> the user's guide to get an idea and a feeling for the 3D in 2D laser simulations.
>>
>> Other than that I can only tell you that it seems to be somewhat unlikely that your crashes are due to the laser, especially since they occur suddenly without any message. If something is wrong with the laser setup (for example placing targets outside of the domain or lenses inside the domain) the laser package will tell you that. Those crashes you get seem to be more like memory issues.
>>
>> Best,
>> Norbert
>>
>>
>> On Thu, 21 Apr 2016, Mario Manuel wrote:
>>
>>> Hi Norbert,
>>>
>>> I am not sure. I found it quite confusing, so to check things I thought I'd look
>>> at it in visit. The code that doesn't run defines the lasers using:
>>>
>>> --------------------------------------------------------------------------------------------------------
>>> --------------------------------------------------------------------------------------------------------
>>> useEnergyDeposition = .true.
>>> ed_maxRayCount = 100000
>>> ed_gradOrder = 2
>>>
>>> # Activate 3D-in-2D ray trace:
>>> ed_laser3Din2D = .true.
>>> ed_laser3Din2DwedgeAngle = 0.5
>>>
>>> ### LASER IO OPTIONS ###
>>> ed_useLaserIO = .true.
>>> ed_laserIOMaxNumberOfPositions = 10000
>>> ed_laserIOMaxNumberOfRays = 128
>>>
>>> ed_numberOfPulses = 3
>>> ed_numberOfBeams = 3
>>>
>>> # Define Beam 1: 2/7 of the power 2.84e12*(2/7)
>>> # approximate lens location for OMEGA (R_cham=3.33m) define angles relative to target normal
>>> #---------- CONE 1 at 32 degrees, y=R*cos(theta), x=R*sin(theta)
>>> ed_numberOfSections_1 = 4
>>> ed_time_1_1 = 0.0
>>> ed_time_1_2 = 0.1e-09
>>> ed_time_1_3 = 1.1e-09
>>> ed_time_1_4 = 1.2e-09
>>> # power can be adjusted to test cone dependence in the future
>>> ed_power_1_1 = 0.0
>>> ed_power_1_2 = 0.81e+12
>>> ed_power_1_3 = 0.81e+12
>>> ed_power_1_4 = 0.0
>>> # set up the beam profile
>>> ed_lensX_1 = 176.5 # radial position
>>> ed_lensY_1 = 282.4 # axial position
>>> ed_lensZ_1 = 0.0
>>> ed_lensSemiAxisMajor_1 = 14.925
>>> ed_targetX_1 = 0.0e-04
>>> ed_targetZ_1 = 0.0e-04
>>> ed_targetY_1 = 0.0e-04
>>> # define major and minor radii at the target
>>> ed_targetSemiAxisMajor_1 = 500.0e-04
>>> ed_targetSemiAxisMinor_1 = 500.0e-04
>>> ed_pulseNumber_1 = 1
>>> ed_wavelength_1 = 0.351
>>> ed_crossSectionFunctionType_1 = "gaussian2D"
>>> ed_gaussianExponent_1 = 4.3
>>> ed_gaussianRadiusMajor_1 = 250.0e-04
>>> ed_gaussianRadiusMinor_1 = 250.0e-04
>>> ed_numberOfRays_1 = 5000
>>> ed_gridType_1 = "radial2D"
>>> ed_semiAxisMajorTorsionAngle_1= 0.0
>>> ed_semiAxisMajorTorsionAxis_1 = "x"
>>>
>>>
>>> # Define Beam 2: 3/7 of the power 2.84e12*(3/7)
>>> # approximate lens location for OMEGA (R_cham=3.33m) define angles relative to target normal
>>> #-------------- CONE 2 at 42 degrees, y=R*cos(theta), x=R*sin(theta)
>>> ed_numberOfSections_2 = 4
>>> ed_time_2_1 = 0.0
>>> ed_time_2_2 = 0.1e-09
>>> ed_time_2_3 = 1.1e-09
>>> ed_time_2_4 = 1.2e-09
>>> # power can be adjusted to test cone dependence in the future
>>> ed_power_2_1 = 0.0
>>> ed_power_2_2 = 1.22e+12
>>> ed_power_2_3 = 1.22e+12
>>> ed_power_2_4 = 0.0
>>> # set up the beam profile
>>> ed_lensX_2 = 222.8 # radial position
>>> ed_lensY_2 = 247.5 # axial position
>>> ed_lensZ_2 = 0.0
>>> ed_lensSemiAxisMajor_2 = 14.925
>>> ed_targetX_2 = 0.0e-04
>>> ed_targetZ_2 = 0.0e-04
>>> ed_targetY_2 = 0.0e-04
>>> # define major and minor radii at the target
>>> ed_targetSemiAxisMajor_2 = 500.0e-04
>>> ed_targetSemiAxisMinor_2 = 500.0e-04
>>> # laser profile from with SG4 profile
>>> ed_pulseNumber_2 = 2
>>> ed_wavelength_2 = 0.351
>>> ed_crossSectionFunctionType_2 = "gaussian2D"
>>> ed_gaussianExponent_2 = 4.3
>>> ed_gaussianRadiusMajor_2 = 250.0e-04
>>> ed_gaussianRadiusMinor_2 = 250.0e-04
>>> ed_numberOfRays_2 = 5000
>>> ed_gridType_2 = "radial2D"
>>> ed_semiAxisMajorTorsionAngle_2= 0.0
>>> ed_semiAxisMajorTorsionAxis_2 = "x"
>>>
>>>
>>> # Define Beam 3: 2/7 of the power 2.84e12*(2/7)
>>> # approximate lens location for OMEGA (R_cham=3.33m) define angles relative to target normal
>>> #------------- CONE 3 at 50 degrees, y=R*cos(theta), x=R*sin(theta)
>>> ed_numberOfSections_3 = 4
>>> ed_time_3_1 = 0.0
>>> ed_time_3_2 = 0.1e-09
>>> ed_time_3_3 = 1.1e-09
>>> ed_time_3_4 = 1.2e-09
>>> # power can be adjusted to test cone dependence in the future
>>> ed_power_3_1 = 0.0
>>> ed_power_3_2 = 0.81e+12
>>> ed_power_3_3 = 0.81e+12
>>> ed_power_3_4 = 0.0
>>> # set up the beam profile
>>> ed_lensX_3 = 255.1 # radial position
>>> ed_lensY_3 = 214.0 # axial position
>>> ed_lensZ_3 = 0.0
>>> ed_lensSemiAxisMajor_3 = 14.925
>>> ed_targetX_3 = 0.0e-04
>>> ed_targetZ_3 = 0.0e-04
>>> ed_targetY_3 = 0.0e-04
>>> # define major and minor radii at the target
>>> ed_targetSemiAxisMajor_3 = 500.0e-04
>>> ed_targetSemiAxisMinor_3 = 500.0e-04
>>> ed_pulseNumber_3 = 3
>>> ed_wavelength_3 = 0.351
>>> ed_crossSectionFunctionType_3 = "gaussian2D"
>>> ed_gaussianExponent_3 = 4.3
>>> ed_gaussianRadiusMajor_3 = 250.0e-04
>>> ed_gaussianRadiusMinor_3 = 250.0e-04
>>> ed_numberOfRays_3 = 5000
>>> ed_gridType_3 = "radial2D"
>>> ed_semiAxisMajorTorsionAngle_3= 0.0
>>> ed_semiAxisMajorTorsionAxis_3 = "x"
>>>
>>> --------------------------------------------------------------------------------------------------------
>>> --------------------------------------------------------------------------------------------------------
>>>
>>> however when I change the lines:
>>>
>>> ed_laser3Din2D = .false.
>>> ed_crossSectionFunctionType_X = "gaussian1D"
>>> ed_gridType_X = "regular1D"
>>>
>>> The code runs and the attached image shows a sample output early in the simulation. The geometry at least looks correct in that there are 3 beams coming in at different angles onto the target. I don't quite understand why they don't seem to be centered on 0,0, but perhaps I am not saving enough rays for this to look correct.
>>>
>>> For what its worth, I have also compiled the LaserSlab simulation on the system I'm using with the standard setup script in the 4.2.2 manual. When I run LaserSlab with the example.par setup, it reproduces the behavior by hanging up right after the output line 'Initial dt verified'. That to me suggests something more fundamental is going wrong on my system.
>>>
>>> Any input is very much welcomed.
>>>
>>> Cheers,
>>> Mario
>>>
>>> ______________________________________
>>>
>>> Dr. Mario Manuel
>>> Einstein Postdoctoral Fellow, Smithsonian Astrophysical Observatory
>>> mmanuel at umich.edu
>>> 617-899-8381
>>>
>>> University of Michigan
>>> Space Research Building
>>> 2455 Hayward Street
>>> Ann Arbor, MI 48109
>>> Office: 734-764-4585
>>> ______________________________________
>>>
>>>
>>> \
>>>
>>> On Apr 21, 2016, at 4:15 PM, Norbert Flocke <flocke at flash.uchicago.edu> wrote:
>>>
>>>> Hi Manuel,
>>>>
>>>> Are you sure you are setting up the beams correctly for this 3D in 2D simulation? The reason I am asking is because it is easy to get confused
>>>> between the cylindrical FLASH coordinates (x,y) -> (R,z) and the 3D beam coordinates you need to use. In your application (cylindrical) the FLASH y coordinate corresponds to the beam 3D z coordinate.
>>>>
>>>> Best,
>>>> Norbert
>>>>
>>>>
>>>> On Thu, 21 Apr 2016, Mario Manuel wrote:
>>>>
>>>>> Hello FLASH users,
>>>>>
>>>>> I am running a FLASH4.2.2 simulation that is a combination of the LaserSlab and FieldLoop
>>>>> example simulations. It is a laser-matter interaction (multi-material, 3T, tabulated EOSs) with MHD as well. I recently implemented a more complex laser setup with multiple beams and wanted to use the ed_laser3Din2D = .true. for a more accurate ray tracing in the R-Z geometry that I am using. After fixing all the initialization errors, I got it to run, but it never gets in to the first step. The simulation hangs up, but doesn't throw an error. The output looks like:
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> Wed Apr 20 20:57:51 PDT 2016
>>>>> NODES= 40
>>>>> NTASKS= 1280
>>>>> [Driver_initParallel]: Called MPI_Init_thread - requested level 2, given level 2
>>>>> [Driver_initParallel]: Number of OpenMP threads in each parallel region 2
>>>>> NOTE: Enabling curvilinear, cartesian_pm/cylindrical_pm/spherical_pm/polar_pm so far was F T F F
>>>>> MaterialProperties initialized
>>>>> Cosmology initialized
>>>>> [eos_tabBrowseIonmix4Tables] IONMIX4 file found: he-imx-005.cn4
>>>>> in eos_inittabulated, tableName = he-imx-005.cn4
>>>>> in eos_inittabulated, tableName = he-imx-005.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, tableName = he-imx-005.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> [eos_tabBrowseIonmix4Tables] IONMIX4 file found: polystyrene-imx-002.cn4
>>>>> in eos_inittabulated, tableName = he-imx-005.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, tableName = polystyrene-imx-002.cn4
>>>>> in eos_inittabulated, tableName = polystyrene-imx-002.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, tableName = polystyrene-imx-002.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> in eos_inittabulated, tableName = polystyrene-imx-002.cn4
>>>>> in eos_inittabulated, groupName = -none-
>>>>> Source terms initialized
>>>>> iteration, no. not moved = 0 0
>>>>> refined: total leaf blocks = 3
>>>>> refined: total blocks = 3
>>>>> INFO: Grid_fillGuardCells is ignoring masking.
>>>>> iteration, no. not moved = 0 2
>>>>> iteration, no. not moved = 1 0
>>>>> refined: total leaf blocks = 12
>>>>> refined: total blocks = 15
>>>>> iteration, no. not moved = 0 13
>>>>> iteration, no. not moved = 1 2
>>>>> iteration, no. not moved = 2 0
>>>>> refined: total leaf blocks = 48
>>>>> refined: total blocks = 63
>>>>> iteration, no. not moved = 0 60
>>>>> iteration, no. not moved = 1 13
>>>>> iteration, no. not moved = 2 0
>>>>> refined: total leaf blocks = 192
>>>>> refined: total blocks = 255
>>>>> iteration, no. not moved = 0 251
>>>>> iteration, no. not moved = 1 60
>>>>> iteration, no. not moved = 2 0
>>>>> refined: total leaf blocks = 768
>>>>> refined: total blocks = 1023
>>>>> iteration, no. not moved = 0 283
>>>>> iteration, no. not moved = 1 101
>>>>> iteration, no. not moved = 2 0
>>>>> refined: total leaf blocks = 216
>>>>> refined: total blocks = 287
>>>>> iteration, no. not moved = 0 50
>>>>> iteration, no. not moved = 1 12
>>>>> iteration, no. not moved = 2 0
>>>>> refined: total leaf blocks = 96
>>>>> refined: total blocks = 127
>>>>> Finished with Grid_initDomain, no restart
>>>>> Ready to call Hydro_init
>>>>> [Hydro_init] The upwind electric field construction is NOT allowed for diffusion.
>>>>> Hydro initialized
>>>>> Gravity initialized
>>>>> Initial dt verified
>>>>> *** Wrote checkpoint file to /p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_chk_0000 ****
>>>>> *** Wrote plotfile to /p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_plt_cnt_0000 ****
>>>>> Initial plotfile written
>>>>> Driver init all done
>>>>> srun: got SIGCONT
>>>>> srun: forcing job termination
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>> and the last lines of the log file look like:
>>>>>
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> [ 04-20-2016 21:18:56.632 ] [GRID amr_refine_derefine]: redist. phase. tot blks requested: 287
>>>>> [GRID amr_refine_derefine] min blks 0 max blks 1 tot blks 287
>>>>> [GRID amr_refine_derefine] min leaf blks 0 max leaf blks 1 tot leaf blks 216
>>>>> [ 04-20-2016 21:18:56.654 ] [GRID amr_refine_derefine]: refinement complete
>>>>> [ 04-20-2016 21:18:56.847 ] [GRID gr_expandDomain]: iteration=6, create level=5
>>>>> [ 04-20-2016 21:18:56.891 ] [GRID amr_refine_derefine]: initiating refinement
>>>>> [ 04-20-2016 21:18:56.896 ] [GRID amr_refine_derefine]: redist. phase. tot blks requested: 127
>>>>> [GRID amr_refine_derefine] min blks 0 max blks 1 tot blks 127
>>>>> [GRID amr_refine_derefine] min leaf blks 0 max leaf blks 1 tot leaf blks 96
>>>>> [ 04-20-2016 21:18:56.919 ] [GRID amr_refine_derefine]: refinement complete
>>>>> [ 04-20-2016 21:18:57.110 ] [GRID gr_expandDomain]: iteration=7, create level=5
>>>>> [ 04-20-2016 21:18:57.116 ] [gr_ensureValidNeighborInfo] found mpi_pattern_id: -10
>>>>> [ 04-20-2016 21:18:57.124 ] [mpi_amr_comm_setup]: buffer_dim_send=157, buffer_dim_recv=1
>>>>> [ 04-20-2016 21:18:57.130 ] [gr_hypreInit]: HYPRE_RELEASE_NAME=hypre, HYPRE_RELEASE_VERSION=2.8.0b, HYPRE_RELEASE_DATE=2011/11/14
>>>>> [ 04-20-2016 21:18:57.136 ] memory: bg heap use (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:57.141 ] memory: bg heap peak (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:57.145 ] memory: bg heap free (MB): 324.20 (min) 332.95 (max) 327.56 (avg)
>>>>> [ 04-20-2016 21:18:57.155 ] memory: bg heap use (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:57.169 ] memory: bg heap peak (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:57.263 ] memory: bg heap free (MB): 324.20 (min) 332.95 (max) 327.56 (avg)
>>>>> [ 04-20-2016 21:18:58.199 ] memory: bg heap use (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:58.205 ] memory: bg heap peak (MB): 126.46 (min) 135.21 (max) 131.85 (avg)
>>>>> [ 04-20-2016 21:18:58.209 ] memory: bg heap free (MB): 324.26 (min) 333.01 (max) 327.63 (avg)
>>>>> [ 04-20-2016 21:18:58.407 ] [IO_writeCheckpoint] open: type=checkpoint name=/p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_chk_0000
>>>>> [ 04-20-2016 21:18:59.943 ] [io_writeData]: wrote 127 blocks
>>>>> [ 04-20-2016 21:19:00.596 ] [IO_writeCheckpoint] close: type=checkpoint name=/p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_chk_0000
>>>>> [ 04-20-2016 21:19:00.682 ] [IO_writePlotfile] open: type=plotfile name=/p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_plt_cnt_0000
>>>>> [ 04-20-2016 21:19:01.269 ] [io_writeData]: wrote 127 blocks
>>>>> [ 04-20-2016 21:19:01.695 ] [IO_writePlotfile] close: type=plotfile name=/p/lscratchv/manuel3/ACSEL15B_CH/ACSEL15B_hdf5_plt_cnt_0000
>>>>> [ 04-20-2016 21:19:01.706 ] memory: bg heap use (MB): 134.27 (min) 135.21 (max) 134.74 (avg)
>>>>> [ 04-20-2016 21:19:01.711 ] memory: bg heap peak (MB): 134.27 (min) 151.21 (max) 134.77 (avg)
>>>>> [ 04-20-2016 21:19:01.716 ] memory: bg heap free (MB): 324.26 (min) 325.20 (max) 324.73 (avg)
>>>>> [ 04-20-2016 21:19:01.722 ] [Driver_evolveFlash]: Entering evolution loop
>>>>> [ 04-20-2016 21:19:01.727 ] step: n=1 t=0.000000E+00 dt=1.000000E-15
>>>>> [ 04-20-2016 21:19:01.746 ] [mpi_amr_comm_setup]: buffer_dim_send=27933, buffer_dim_recv=1
>>>>> [ 04-20-2016 21:19:01.783 ] [hy_uhd_unsplit]: gcNeed(MAGI_FACE_VAR,MAG_FACE_VAR) - FACES
>>>>> [ 04-20-2016 21:19:01.792 ] [mpi_amr_comm_setup]: buffer_dim_send=2333, buffer_dim_recv=1
>>>>> [ 04-20-2016 21:19:01.886 ] [mpi_amr_comm_setup]: buffer_dim_send=27933, buffer_dim_recv=1
>>>>> [ 04-20-2016 21:19:01.913 ] [mpi_amr_comm_setup]: buffer_dim_send=25757, buffer_dim_recv=1
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>> -------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>> Everything runs fine when I switch the flag to ed_laser2Din3D = .false.
>>>>>
>>>>> Has anyone come across this type of thing before? Are there issues with the 2Din3D package and some other packages I may have implemented? Any thoughts are much welcomed.
>>>>>
>>>>> Cheers,
>>>>> Mario
>>>>>
>>>>> ______________________________________
>>>>>
>>>>> Dr. Mario Manuel
>>>>> Einstein Postdoctoral Fellow, Smithsonian Astrophysical Observatory
>>>>> mmanuel at umich.edu
>>>>> 617-899-8381
>>>>>
>>>>> University of Michigan
>>>>> Space Research Building
>>>>> 2455 Hayward Street
>>>>> Ann Arbor, MI 48109
>>>>> Office: 734-764-4585
>>>>> ______________________________________
>>>>>
>>>>>
>>>
>>>
>
>
More information about the flash-users
mailing list