<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style><style type="text/css"></style>
</head>
<body style="word-wrap:break-word" fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi John,
<div><br>
</div>
<div>Thanks for your suggestions!</div>
<div><br>
</div>
<div>1) We excluded the Hydro for two reasons: the Hydro has usually basically no effect on the DM particles (especially not on the positions where matter concentrates), and we used both split/PPM and unsplit/StaggeredMesh without any change in the outcome.</div>
<div><br>
</div>
<div>2) We used the unsplit and split driver by choosing split/PPM or unsplit/StaggeredMesh. Unfortunately I already deleted the PPM data, but I have setup another simulation with split/PPM, which should be finished by tomorrow.</div>
<div><br>
</div>
<div>3) We will try this and report back.</div>
<div><br>
</div>
<div>4) I've attached and uploaded the setup_call and setup_units files.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Josef</div>
<div><br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF393393" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>Von:</b> John ZuHone [jzuhone@milkyway.gsfc.nasa.gov]<br>
<b>Gesendet:</b> Sonntag, 08. September 2013 15:48<br>
<b>An:</b> Stöckl, Josef<br>
<b>Cc:</b> flash-users@flash.uchicago.edu<br>
<b>Betreff:</b> Re: [FLASH-USERS] Strange discrepancy in Dark Matter structure of cosmological simulation<br>
</font><br>
</div>
<div></div>
<div>Josef,
<div><br>
</div>
<div>First, thank you for the very thorough investigation you've done of this problem. </div>
<div><br>
</div>
<div>I did have a few thoughts on maybe how to try to weed this one out:</div>
<div><br>
</div>
<div>1) I wonder a little bit about this:</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div>
<div style="direction:ltr; font-family:Tahoma; font-size:10pt"><span style="font-size:10pt">Since this simulation basically only includes cosmology, self-gravity, active particles and hydro, we can only make out three possible points, which could influence
the DM evolution: the cosmology, the gravity or the LeapfrogCosmo unit. </span></div>
</div>
</blockquote>
</div>
<div>
<div>
<div style="direction:ltr; font-family:Tahoma; font-size:10pt"><span style="font-size:10pt"><br>
</span></div>
<div style="direction:ltr; font-family:Tahoma; font-size:10pt"><span style="font-size:10pt">Why not hydro? Certainly the baryonic component is very small by mass, but in some collapsed regions the effect would not be completely insignificant. I'm guessing you're
right, but I would just throw that one out there for consideration. </span></div>
</div>
</div>
<div><br>
</div>
<div>2) Which Driver unit are you using? Split or Unsplit? Have you tried this with the other one? </div>
<div><br>
</div>
<div>3) Could you try setting up the Zeldovich pancake problem (without hydro) in all three codes? This should provide a clean, unambiguous point of comparison. </div>
<div><br>
</div>
<div>4) Thanks for sending the flash.par, but could you also send along the setup_units and setup_call files? I'd like to see what units are being employed.</div>
<div><br>
</div>
<div>I think that somewhere between these we could get to the bottom of this. </div>
<div><br>
</div>
<div>Best,</div>
<div><br>
</div>
<div>John</div>
<div><br>
<div>
<div>On Sep 7, 2013, at 10:20 AM, Stöckl, Josef <<a href="mailto:Josef.Stoeckl@uibk.ac.at" target="_blank">Josef.Stoeckl@uibk.ac.at</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:medium; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px">
<div style="direction:ltr; font-family:Tahoma; font-size:10pt">Dear FLASH users and devs,
<div><br>
</div>
<div>We have been conducting some comparisons of cosmological simulations conducted with FLASH 4.0.1, Gadget-2 and Enzo. We set up a cosmological volume of 60 Mpc/h box size with Planck cosmological parameters and evolve these initial conditions in all three
codes. For the grid based codes (FLASH, Enzo) we used 256^3 resolution, and for Gadget-2 we used 256^3 DM particles and 256^3 gas particles. </div>
<div><br>
</div>
<div>In doing so we noticed that the dark matter structure FLASH shows a significant difference to both Enzo and Gadget-2, while those two codes show practically perfect agreement. I have attached some projection plots at redshift z=0 to show you the differences.
As you can see, it seems as if the DM structure is less evolved in FLASH. <span style="font-size:10pt">To see, if this is discrepancy is specific to FLASH 4.0.1, we also tried FLASH 3.1, but there is no change to be seen there. </span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">Since this simulation basically only includes cosmology, self-gravity, active particles and hydro, we can only make out three possible points, which could influence the DM evolution: the cosmology, the gravity or the LeapfrogCosmo
unit. </span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">We tried to check the cosmology unit by comparing the dt/dz, z, t and tend values and the timesteps that the simulation takes, but everything is in very good agreement with our calculations and also Enzo. So at least in terms
of general cosmology there doesn't seem to be a problem there.</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">For checking the gravity unit, we switched from Multigrid to bhtree, but this did change absolutely nothing - the agreement between the two gravity units is very good. So we are inclined to exclude the gravity unit as the culprit.</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">This leaves us basically with the LeapFrogCosmo unit as the only point left, where things could go wrong.</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">I have checked everything that I can think of in terms of reading the ICs wrongly into FLASH, but all my checks turn out well (overall mass, min/max/avg density, particle masses, crit dens,...). To further exclude me as a source
of error, a colleague also started with a FLASH 4.0.1 vanilla version from scratch and implemented the IC reading himself, but this also leads to the same results. The ICs have been filled in by assuming both distances and masses to be comoving cgs, but with
the configured cosmology (aka not "over h").</span></div>
<div><span style="font-size:10pt"><br>
</span></div>
<div><span style="font-size:10pt">Since the structures seemed less evolved to us, we decided to try running the simulation into the future and set redshift z = -0.5 as the final z. The result is given as the sim1-future* attachment, and to our surprise, at
this point the FLASH structures match the other two codes structures (with those at z=0).</span></div>
<div>
<div><br>
</div>
<div>We would be thankful for any insights into this, since we are at our wits end.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Josef</div>
<div><br>
</div>
<div>The attachments can also be found at <a href="http://astro-staff.uibk.ac.at/~josef/work/comparison/flash-users/" style="font-size:10pt" target="_blank">http://astro-staff.uibk.ac.at/~josef/work/comparison/flash-users/</a> together with the initial condition
gadget-2 snapshot file ics_* and the resulting FLASH checkpoint 0000 file for simulation 1.</div>
<div><br>
</div>
<div>Attachments:</div>
<div>sim1_* ... the simulation 1 comparison projection plots</div>
<div>sim2_* ... <span style="font-size:10pt">the simulation 2 comparison projection plots</span></div>
<div>sim1-future_* ... simulation 1 evolved until z = -0.5</div>
<div>flash.par ... simulation 1 flash.par with comments removed</div>
<div><br>
</div>
<div><br>
</div>
<div><span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">----------------------------------------------</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">Dr. Josef Stöckl</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">Institute of Astro- and Particle Physics</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">Technikerstrasse 25/8</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">6020 Innsbruck</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">Austria</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">Phone: +43 512 507 52020</span><br style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">
<span style="font-family:'Segoe UI',Helvetica,Arial,sans-serif">E-Mail:<span class="Apple-converted-space"> </span><a href="mailto:josef.stoeckl@uibk.ac.at" target="_blank">josef.stoeckl@uibk.ac.at</a></span></div>
</div>
</div>
<span><sim1_enzo_Projection_z_Density.png></span><span><sim1_enzo_Projection_z_dm_density.png></span><span><sim1_flash4_Projection_z_Density.png></span><span><sim1_flash4_Projection_z_pden.png></span><span><sim1_gadget_Projection_z_Density.png></span><span><sim1_gadget_Projection_z_dm_density.png></span><span><sim1-future_flash4_Projection_z_Density.png></span><span><sim1-future_flash4_Projection_z_pden.png></span><span><sim2_enzo_Projection_z_Density.png></span><span><sim2_enzo_Projection_z_dm_density.png></span><span><sim2_flash4_Projection_z_Density.png></span><span><sim2_flash4_Projection_z_pden.png></span><span><sim2_gadget_Projection_z_Density.png></span><span><sim2_gadget_Projection_z_dm_density.png></span><span><flash.par></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>