[FLASH-USERS] [EXT] Library conflict at the end of make and segmentation fault
Mattia Cipriani
mattia.cipriani at enea.it
Mon Mar 4 16:33:34 EST 2024
Hi all,
I come back on this thread to point out that I tried solving my issue in different ways.
One of them was to recompile hypre with a new version, namely 2.27, with the idea that upgrading FLASH required upgrading hypre too.
I also found an issue with the compilers on my cluster, which made the compiling a bit messy.
I tried to use version 4.7 of FLASH, instead of 4.7.1 I was using previously.
However, my problem was not solved.
FLASH is randomly exiting with a segmentation fault error with the hypre solve.
The compiler I am using if the gnu fortran open mpi compiler version 7.3.0.
I report below the output I am getting with the segmentation fault.
Backtrace for this error:
#0 0x2b6d450e436f in ???
#1 0x2b6d56323762 in ???
#2 0x2b6d45b9b84b in ???
#3 0x2b6d442e1eec in ???
#4 0x2b6d443216ea in ???
#5 0x7710f1 in ???
#6 0x76f044 in ???
#7 0x7c72ed in ???
#8 0x7c785a in ???
#9 0x6f012d in ???
#10 0x6a7fe0 in ???
#11 0x6835fc in ???
#12 0x67496a in ???
#13 0x677854 in ???
#14 0x5592a7 in gr_hypresolve_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/gr_hypreSolve.F90:176
#15 0x42c0ca in grid_advancediffusion_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/Grid_advanceDiffusion.F90:351
#16 0x4650b3 in radtrans_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/RadTrans.F90:419
#17 0x411d8b in driver_evolveflash_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/Driver_evolveFlash.F90:305
#18 0x407c86 in flash
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/Flash.F90:51
#19 0x407c86 in main
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/Sim/Flash.F90:43<http://enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_polystyrene_4.7/Flash.F90:43>
The issue might be related to some cluster configuration, but maybe somebody can indicate some way of debugging this.
Any ideas?
Best regards,
Mattia
---------------------------------------------
Dr Mattia Cipriani, PhD
ENEA
FSN-PLAS-PAX
45, Via E. Fermi
00044 Frascati (Rome)
Italy
Office: Building 12
Room 12223
Tel: +39 069400 5145
Mobile: +39 328 5552077
Skype: matcipriani
Researchgate<https://www.researchgate.net/profile/Mattia-Cipriani-2>
LinkedIn<https://www.linkedin.com/in/mattia-cipriani-0707bb2a/>
ResearcherID AAG-9269-2021<https://publons.com/researcher/AAG-9269-2021/>
ORCID 0000-0002-2500-605X<https://orcid.org/0000-0002-2500-605X>
---------------------------------------------
Il giorno 7 feb 2024, alle ore 14:27, Mattia Cipriani <mattia.cipriani at enea.it> ha scritto:
My version of hypre is 2.15.1.
---------------------------------------------
Dr Mattia Cipriani, PhD
ENEA
FSN-PLAS-PAX
45, Via E. Fermi
00044 Frascati (Rome)
Italy
Office: Building 12
Room 12223
Tel: +39 069400 5145
Mobile: +39 328 5552077
Skype: matcipriani
Researchgate<https://www.researchgate.net/profile/Mattia-Cipriani-2>
LinkedIn<https://www.linkedin.com/in/mattia-cipriani-0707bb2a/>
ResearcherID AAG-9269-2021<https://publons.com/researcher/AAG-9269-2021/>
ORCID 0000-0002-2500-605X<https://orcid.org/0000-0002-2500-605X>
---------------------------------------------
Il giorno 7 feb 2024, alle ore 14:13, Reyes, Adam <adam.reyes at rochester.edu> ha scritto:
Hi Mattia,
Are you using a version of hyper >= 2.29?
We noticed recently that starting with that version the library depends on some initialize/finalize calls that are not being called from FLASH.
*********************************************
Adam Reyes
<FLASH.jpg>
Code Group Leader, Flash Center for Computational Science
Research Scientist, Dept. of Physics and Astronomy
University of Rochester
River Campus: Bausch and Lomb Hall, 369
500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
Email adam.reyes at rochester.edu
Web https://flash.rochester.edu
(he / him / his)
<FLASH-pride-sml.png>
*********************************************
On Feb 7, 2024, at 2:09 PM, Mattia Cipriani <mattia.cipriani at enea.it> wrote:
Dear Adam,
I solved the issue about the conflicting library, which was cluster-related.
I run a simulation which ended with the segmentation fault error.
I list the output from the job on the cluster here, maybe it can help in identifying the issue:
420 1.3202E-10 1.9789E-13 ( 1.377E-02, 2.573E-02, 0.00 ) | 1.979E-13 5.081E+79 8.265E+85 0.4000000
421 1.3222E-10 1.9781E-13 ( 1.389E-02, 2.552E-02, 0.00 ) | 1.978E-13 5.043E+79 8.036E+85 0.4000000
422 1.3242E-10 1.9765E-13 ( 1.389E-02, 2.552E-02, 0.00 ) | 1.976E-13 5.048E+79 7.501E+85 0.4000000
423 1.3262E-10 1.9752E-13 ( 1.389E-02, 2.552E-02, 0.00 ) | 1.975E-13 5.101E+79 7.625E+85 0.4000000
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x2b8611c3336f in ???
#1 0x2b8622d73762 in ???
#2 0x2b86126ea84b in ???
#3 0x2b8610e31c2c in ???
#4 0x2b8610e75c6e in ???
#5 0x6f0270 in ???
#6 0x701354 in ???
#7 0x6981e1 in ???
#8 0x687315 in ???
#9 0x67597d in ???
#10 0x678284 in ???
#11 0x559c43 in gr_hypresolve_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/gr_hypreSolve.F90:181
#12 0x42c41a in grid_advancediffusion_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/Grid_advanceDiffusion.F90:351
#13 0x465623 in radtrans_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/RadTrans.F90:419
#14 0x4120e0 in driver_evolveflash_
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/Driver_evolveFlash.F90:306
#15 0x407fc6 in flash
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/Flash.F90:51
#16 0x407fc6 in main
at /afs/enea.it/project/iner/SIMULAZIONI/FSim/exp_infn-fusion-simsesperimento_SiO2_t1/Flash.F90:43
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 11 with PID 0 on node cresco6x005 exited on signal 11 (Segmentation fault).
—————————————————————————————————————
Thanks,
Mattia
---------------------------------------------
Dr Mattia Cipriani, PhD
ENEA
FSN-PLAS-PAX
45, Via E. Fermi
00044 Frascati (Rome)
Italy
Office: Building 12
Room 12223
Tel: +39 069400 5145
Mobile: +39 328 5552077
Skype: matcipriani
Researchgate<https://urldefense.com/v3/__https://www.researchgate.net/profile/Mattia-Cipriani-2__;!!CGUSO5OYRnA7CQ!exAwgr543bZGIXI5dU7Etu6rYav_iVMUvLAg5qJuoK10ez1YrtR55Bi8vLhP0kcRUchD4quv1loCimTwEzpWu1gyIn_ejLU$>
LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/in/mattia-cipriani-0707bb2a/__;!!CGUSO5OYRnA7CQ!exAwgr543bZGIXI5dU7Etu6rYav_iVMUvLAg5qJuoK10ez1YrtR55Bi8vLhP0kcRUchD4quv1loCimTwEzpWu1gyIM-oODk$>
ResearcherID AAG-9269-2021<https://urldefense.com/v3/__https://publons.com/researcher/AAG-9269-2021/__;!!CGUSO5OYRnA7CQ!exAwgr543bZGIXI5dU7Etu6rYav_iVMUvLAg5qJuoK10ez1YrtR55Bi8vLhP0kcRUchD4quv1loCimTwEzpWu1gyq_XHuv8$>
ORCID 0000-0002-2500-605X<https://urldefense.com/v3/__https://orcid.org/0000-0002-2500-605X__;!!CGUSO5OYRnA7CQ!exAwgr543bZGIXI5dU7Etu6rYav_iVMUvLAg5qJuoK10ez1YrtR55Bi8vLhP0kcRUchD4quv1loCimTwEzpWu1gyf7jAkEA$>
---------------------------------------------
Il giorno 7 feb 2024, alle ore 11:49, Reyes, Adam <adam.reyes at rochester.edu> ha scritto:
Hi Mattia,
I think openmpi provides a “ompi_info” command that should print out all information on how it was built. Somewhere in there should have the compiler version.
*********************************************
Adam Reyes
<FLASH.jpg>
Code Group Leader, Flash Center for Computational Science
Research Scientist, Dept. of Physics and Astronomy
University of Rochester
River Campus: Bausch and Lomb Hall, 369
500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
Email adam.reyes at rochester.edu
Web https://flash.rochester.edu
(he / him / his)
<FLASH-pride-sml.png>
*********************************************
On Feb 7, 2024, at 11:45 AM, Mattia Cipriani <mattia.cipriani at enea.it> wrote:
Hi Adam,
thank you for the reply.
I was aware that this might be the problem, but I don’t know how to check it.
How can I know which is the version of the compiler used for building the mpi library?
---------------------------------------------
Dr Mattia Cipriani, PhD
ENEA
FSN-PLAS-PAX
45, Via E. Fermi
00044 Frascati (Rome)
Italy
Office: Building 12
Room 12223
Tel: +39 069400 5145
Mobile: +39 328 5552077
Skype: matcipriani
Researchgate<https://urldefense.com/v3/__https://www.researchgate.net/profile/Mattia-Cipriani-2__;!!CGUSO5OYRnA7CQ!aazmSSqcrCVxyt9iupeL-dXWjtLDp4mgnGfpdRHPDGN1AXkNZIlpwgoCGjMq3bmNkVsWzm6yxMBbPjxv4Og_PBsmxR3d7uw$>
LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/in/mattia-cipriani-0707bb2a/__;!!CGUSO5OYRnA7CQ!aazmSSqcrCVxyt9iupeL-dXWjtLDp4mgnGfpdRHPDGN1AXkNZIlpwgoCGjMq3bmNkVsWzm6yxMBbPjxv4Og_PBsmKaabxww$>
ResearcherID AAG-9269-2021<https://urldefense.com/v3/__https://publons.com/researcher/AAG-9269-2021/__;!!CGUSO5OYRnA7CQ!aazmSSqcrCVxyt9iupeL-dXWjtLDp4mgnGfpdRHPDGN1AXkNZIlpwgoCGjMq3bmNkVsWzm6yxMBbPjxv4Og_PBsmvZhY6Ck$>
ORCID 0000-0002-2500-605X<https://urldefense.com/v3/__https://orcid.org/0000-0002-2500-605X__;!!CGUSO5OYRnA7CQ!aazmSSqcrCVxyt9iupeL-dXWjtLDp4mgnGfpdRHPDGN1AXkNZIlpwgoCGjMq3bmNkVsWzm6yxMBbPjxv4Og_PBsmlFo2Oo8$>
---------------------------------------------
Il giorno 7 feb 2024, alle ore 11:30, Reyes, Adam <adam.reyes at rochester.edu> ha scritto:
Hi Mattia,
The warning suggests that compiler that you’re using for FLASH is different from the one used to compile the openmpi library. You could check that “mpif90 --version” returns the same GCC 7.3.0 version your warning says that openmpi was built with.
It’s difficult to say if this is related to your seg fault or not without more context about where the error is occurring, either from the stdout or the log file.
*********************************************
Adam Reyes
<FLASH.jpg>
Code Group Leader, Flash Center for Computational Science
Research Scientist, Dept. of Physics and Astronomy
University of Rochester
River Campus: Bausch and Lomb Hall, 369
500 Wilson Blvd. PO Box 270171, Rochester, NY 14627
Email adam.reyes at rochester.edu
Web https://flash.rochester.edu
(he / him / his)
<FLASH-pride-sml.png>
*********************************************
On Feb 7, 2024, at 11:06 AM, Mattia Cipriani <mattia.cipriani at enea.it> wrote:
Hi,
I am recently experiencing some issues with FLASH 4.7.1 simulations, after updating from FLASH 4.5.
I am using the same site-specific Makefile.h which was working before, since the configuration of the cluster did not change from the past.
Also the compilers are the same as before.
I use the setup script to collect all the needed files and objects for compiling the simulation and then I use make to compile it.
At the end of the execution of make, I get the following warning
/bin/ld: warning: libgfortran.so.4, needed by /usr/mpi/gcc730/openmpi-3.1.2/lib/libmpi_usempif08.so, may conflict with libgfortran.so.3
If I run the flash4 executables with mpirun it starts to run ok, but at some apparently random point of the simulation I get a segmentation fault error.
I am wondering if the warning from make could be the reason for the segmentation fault error.
How can I solve the warning?
I know it is a linux-related question rather than a FLASH problem, but I would be glad if you can help or at least give me a feedback if the segmentation fault error might be due to the warning from make.
Best regards,
Mattia
________________________________
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettron ica da cui è stata inviata è da qualificarsi quale strumento aziendale.
La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente viet ate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).
Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mit tente e di provvedere alla sua distruzione. Grazie.
This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) on ly.
Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequen t amendments and GDPR UE 2016/679).
If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e -mail. Thanks.
________________________________
_______________________________________________
flash-users mailing list
flash-users at flash.rochester.edu
For list info, including unsubscribe:
https://flash.rochester.edu/mailman/listinfo/flash-users
________________________________
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettron ica da cui è stata inviata è da qualificarsi quale strumento aziendale.
La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente viet ate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).
Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mit tente e di provvedere alla sua distruzione. Grazie.
This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) on ly.
Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequen t amendments and GDPR UE 2016/679).
If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e -mail. Thanks.
________________________________
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettron ica da cui è stata inviata è da qualificarsi quale strumento aziendale.
La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente viet ate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).
Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mit tente e di provvedere alla sua distruzione. Grazie.
This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) on ly.
Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequen t amendments and GDPR UE 2016/679).
If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e -mail. Thanks.
________________________________
________________________________
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettron ica da cui è stata inviata è da qualificarsi quale strumento aziendale.
La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente viet ate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).
Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mit tente e di provvedere alla sua distruzione. Grazie.
This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) on ly.
Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequen t amendments and GDPR UE 2016/679).
If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e -mail. Thanks.
________________________________
_______________________________________________
flash-users mailing list
flash-users at flash.rochester.edu
For list info, including unsubscribe:
https://flash.rochester.edu/mailman/listinfo/flash-users
________________________________
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettron ica da cui è stata inviata è da qualificarsi quale strumento aziendale.
La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente viet ate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679).
Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mit tente e di provvedere alla sua distruzione. Grazie.
This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only.
Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequen t amendments and GDPR UE 2016/679).
If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e -mail. Thanks.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://flash.rochester.edu/pipermail/flash-users/attachments/20240304/8db46c70/attachment.htm>
More information about the flash-users
mailing list