[FLASH-USERS] Question about simulation time spent in HYPRE

[Jeremy Melvin]

Hello all,

I am running a 2D spherical Rad-Hydro simulation (I have made some
edits to the RadTrans solver to allow for it to handle the 2D
spherical geometry).

I am using 64 energy groups for a Multi-Group Diffusion simulation and
was wondering if the following profile output in the logfile seemed
reasonable.

accounting unit                       time sec  num calls   secs avg  time pct
******
RadTrans                                   542875.204  33116   16.393    89.761
  Grid_advanceDiffusion              516282.519  ******    0.244    85.364
    gr_hypreSetIniGuess               31996.328    ******    0.015     5.290
    gr_hypreCreateMatrix               2703.867      ******   0.001     0.447
    gr_hypreComputeB                  692.072       ******   0.000     0.114
    gr_hypreSolve                         480558.961  ******   0.227    79.457
       gr_hypreSetupSolver             14.593         ******   0.000     0.002
       HYPRE_ParCSRPCGSetup   301379.330 ******   0.142    49.831
       HYPRE_ParCSRPCGSolve   176461.720 ******    0.083    29.177
       gr_hypreDestroySolver          25.306        ******    0.000     0.004
    gr_hypreUpdateSoln                 50.462        ******    0.000     0.008
******

My concern is that the code is spending almost twice as long (49.8%)
of the time in HYPRE_ParCSRPCGSetup than it is in HYPRE_ParCSRPCGSolve
(29.2%).  Does that seem like it could be the sign of a larger problem
(with implementation, input options, HYPRE configuration, etc..) or
could it make sense for my simulation?

These simulations are using a Uniform Grid and were run without mesh
replication on 120 processors.  I plan to try it with mesh replication
and see if that improves this.

Please let me know if you have any questions or need any additional
information about my simulation.  Thanks!


Jeremy Melvin