Example B3a segmentation fault on computing center

I seem to have correctly installed ROOT 6.24/02, Geant4 10.7.p02, and additional libraries on a CentOS 7 (really RHEL 7) based computing center (TACC).
The center uses impi 18.0.2, the intel compilers (icc,icpc) 18.0.2, cmake 3.16.1.

ROOT has been tested to work; I can open a TBrowser to view files.
Geant4 does not have any make errors or cmake configure errors, but the example B3a throws a segmentation fault upon opening the analysis file.

The visualization opens, using the G4UIGAG ui, and it is only upon running an event that the analysis file fails (only the first 166 byte header of the root file is created, so the file is unreadable):

image.png

Index : 2     used in the geometry : Yes
 Material : G4_BRAIN_ICRP
 Range cuts        :  gamma  1 mm     e-  1 mm     e+  1 mm  proton 1 mm 
 Energy thresholds :  gamma  2.92086 keV    e-  361.464 keV    e+  351.877 keV proton 100 keV
 Region(s) which use this couple : 
    DefaultRegionForTheWorld

==================================================================

[New Thread 0x2aaad5252700 (LWP 322471)]
### Run 0 starts.
### Run 0 start.
... create file : scoring.root - done
... open analysis file : scoring.root - done
G4WT1 > /tracking/storeTrajectory 2
G4WT0 > /tracking/storeTrajectory 2
G4WT0 > ### Run 0 starts on worker thread 0.
G4WT0 > ### Run 0 start.
G4WT1 > ### Run 0 starts on worker thread 1.
G4WT1 > ### Run 0 start.
G4WT0 > ... create file : scoring_t0.root - done
G4WT0 > ... open analysis file : scoring_t0.root - done

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2aaac2cb0700 (LWP 320402)]
0x0000000000425cb0 in __intel_avx_rep_memcpy ()
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 freetype-2.8-12.el7_6.1.x86_64 glib2-2.56.1-2.el7.x86_64 glibc-2.17-260.el7_6.3.x86_64 graphite2-1.3.10-1.el7_3.x86_64 harfbuzz-1.7.5-2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libICE-1.0.9-9.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.5-2.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.15-1.el7.x86_64 libXdamage-1.1.4-4.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.3-1.el7.x86_64 libXmu-1.1.2-2.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libXt-1.1.5-3.el7.x86_64 libXxf86vm-1.1.4-1.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libcurl-7.29.0-51.el7.x86_64 libdrm-2.4.91-3.el7.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 libicu-50.1.2-17.el7.x86_64 libidn-1.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-14.1.el7.x86_64 libssh2-1.4.3-12.el7.x86_64 libuuid-2.23.2-59.el7.x86_64 libxcb-1.13-1.el7.x86_64 libxshmfence-1.2-1.el7.x86_64 llvm-private-6.0.1-2.el7.x86_64 mesa-dri-drivers-18.0.5-4.el7_6.x86_64 mesa-libGL-18.0.5-4.el7_6.x86_64 mesa-libglapi-18.0.5-4.el7_6.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.1.el7_6.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.1.el7_6.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7_6.5.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) where
#0  0x0000000000425cb0 in __intel_avx_rep_memcpy ()
#1  0x00002aaab7823394 in CLHEP::MixMaxRng::apply_bigskip(unsigned long long*, unsigned long long*, unsigned int, unsigned int, unsigned int, unsigned int) [clone .a] () from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4clhep.so
#2  0x00002aaab782325d in CLHEP::MixMaxRng::seed_uniquestream(unsigned int, unsigned int, unsigned int, unsigned int) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4clhep.so
#3  0x00002aaab167188b in G4WorkerRunManager::GenerateEvent(int) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#4  0x00002aaab167363e in G4WorkerRunManager::ProcessOneEvent(int) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#5  0x00002aaab167380c in G4WorkerRunManager::DoEventLoop(int, char const*, int) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#6  0x00002aaab1655432 in G4RunManager::BeamOn(int, char const*, int) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#7  0x00002aaab1671266 in G4WorkerRunManager::DoWork() ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#8  0x00002aaab167f301 in G4MTRunManagerKernel::StartThread(G4WorkerThread*) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#9  0x00002aaab7da6eef in std::execute_native_thread_routine (__p=0xaf37e0)
    at ../../../.././libstdc++-v3/src/c++11/thread.cc:83
#10 0x00002aaab858fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00002aaab88a1ead in clone () from /lib64/libc.so.6
(gdb) 

The where command does not give me much information, other than noting that this is somehow related to memory?

image.png

The make commands used follow (the 49 cores are irrelevant - the error is reproducible with a single core):

make clean

cmake -DCMAKE_C_COMPILER=icc \
-DCMAKE_CXX_COMPILER=icpc \
-DCMAKE_PREFIX_PATH=$sharedir/geant4/geant4-install \
-DROOT_INCLUDE_DIRS=$sharedir/root/include \
../B3a

make -j49

(note icc and icpc are the intel compilers required by the computing center)

Interestingly, I get a similar error when running my code (which has been tested on my home machine):

It seems to occur at a different location, however, the where command shows that the error is again related to the analysis file.

image.png

GDB where:

image.png

image.png

Isomer production flag                              1
Internal e- conversion flag                         1
Store e- internal conversion data                   0
Electron internal conversion ID                     2
Correlated gamma emission flag                      0
Max 2J for sampling of angular correlations         10
=======================================================================
Start closing geometry.
G4GeometryManager::ReportVoxelStats -- Voxel Statistics

    Total memory consumed for geometry optimisation:   11 kByte
    Total CPU time elapsed for geometry optimisation: 0 seconds

    Voxelisation: top CPU users:
    Percent   Total CPU    System CPU       Memory  Volume
    -------   ----------   ----------     --------  ----------
       0.00         0.00         0.00           11k world_volume

    Voxelisation: top memory users:
    Percent     Memory      Heads    Nodes   Pointers    Total CPU    Volume
    -------   --------     ------   ------   --------   ----------    ----------
     100.00         11k        19       74        842         0.00    world_volume

=======================================================================
--> G4MTRunManager::CreateAndStartWorkers() --> Initializing workers...
=======================================================================

[New Thread 0x2aaac8582700 (LWP 326347)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2aaac8582700 (LWP 326347)]
0x00002aaab87f3278 in __printf_fp_l () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 freetype-2.8-12.el7_6.1.x86_64 glib2-2.56.1-2.el7.x86_64 glibc-2.17-260.el7_6.3.x86_64 graphite2-1.3.10-1.el7_3.x86_64 harfbuzz-1.7.5-2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libICE-1.0.9-9.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.5-2.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXmu-1.1.2-2.el7.x86_64 libXt-1.1.5-3.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libcurl-7.29.0-51.el7.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 libicu-50.1.2-17.el7.x86_64 libidn-1.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-14.1.el7.x86_64 libssh2-1.4.3-12.el7.x86_64 libuuid-2.23.2-59.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.1.el7_6.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.1.el7_6.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) where
#0  0x00002aaab87f3278 in __printf_fp_l () from /lib64/libc.so.6
#1  0x00002aaab87f1357 in vfprintf () from /lib64/libc.so.6
#2  0x00002aaab881bf39 in vsnprintf () from /lib64/libc.so.6
#3  0x00002aaab7dc222f in std::__convert_from_v (
    __cloc=@0x2aaac8345ad8: 0x2aaab8b69780 <_nl_C_locobj>, __out=__out@entry=0x2aaac8345a50 "", 
    __size=__size@entry=45, __fmt=__fmt@entry=0x2aaac8345ae0 "%.*g")
    at /admin/build/rpms/BUILD/gcc-6.3.0/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/c++locale.h:92
#4  0x00002aaab7def4e2 in std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_float<double> (this=0x2aaab806d010 <(anonymous namespace)::num_put_c>, __s=..., 
    __io=..., __fill=<optimized out>, __mod=__mod@entry=0 '\000', __v=0)
    at /admin/build/rpms/BUILD/gcc-6.3.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/locale_facets.tcc:1005
#5  0x00002aaab7def7f0 in std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put (this=<optimized out>, __s=..., __io=..., __fill=<optimized out>, 
    __v=<optimized out>)
    at /admin/build/rpms/BUILD/gcc-6.3.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/locale_facets.tcc:1159
#6  0x00002aaab7dfb5a5 in put (__v=0, __fill=<optimized out>, __io=..., __s=..., 
    this=0x2aaab806d010 <(anonymous namespace)::num_put_c>)
    at /admin/build/rpms/BUILD/gcc-6.3.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/locale_facets.h:2434
#7  std::ostream::_M_insert<double> (this=0x2aaac8345b98, __v=0)
    at /admin/build/rpms/BUILD/gcc-6.3.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:73
#8  0x00002aaab72d01b7 in G4UIparameter::SetDefaultValue(double) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4intercoms.so
#9  0x00002aaab500422b in G4H1Messenger::CreateH1Cmd() ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#10 0x00002aaab5002db4 in G4H1Messenger::G4H1Messenger(G4VAnalysisManager*) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#11 0x00002aaab4ff277f in G4AnalysisMessenger::G4AnalysisMessenger(G4VAnalysisManager*) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#12 0x00002aaab5215f84 in G4VAnalysisManager::G4VAnalysisManager(G4String const&, bool) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#13 0x00002aaab4d2091b in G4ToolsAnalysisManager::G4ToolsAnalysisManager(G4String const&, bool) [clone .a] () from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#14 0x00002aaab52246d3 in G4RootAnalysisManager::G4RootAnalysisManager(bool) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#15 0x00002aaab5225561 in G4RootAnalysisManager::Instance() ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4analysis.so
#16 0x00000000004a2799 in HistoManager::Book() ()
---Type <return> to continue, or q <return> to quit---
#17 0x00000000004a273f in HistoManager::HistoManager(G4VUserDetectorConstruction*) ()
#18 0x00000000004aba92 in B3aRunAction::B3aRunAction(G4VUserDetectorConstruction*, B3PrimaryGeneratorAction*) ()
#19 0x000000000046adda in B3aActionInitialization::Build() const ()
#20 0x00002aaab167f2da in G4MTRunManagerKernel::StartThread(G4WorkerThread*) ()
   from /work2/07752/as_tacc/utklshare/geant4/geant4-install/lib64/libG4run.so
#21 0x00002aaab7da6eef in std::execute_native_thread_routine (__p=0xb342330)
    at ../../../.././libstdc++-v3/src/c++11/thread.cc:83
#22 0x00002aaab858fdd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00002aaab88a1ead in clone () from /lib64/libc.so.6

While both errors are called by the Analysis manager, the first error is a memcpy error, while the second is due to a failed printf.

Similar CMake commands were used for this program as well.

As far as I can guess, these errors may be due to either ROOT and Geant4 having compatibility issues, or bad library linkage.

I would really appreciate any light you can shed on this matter. Thank you.

Could you please try and upload the attachments again please - I’m not able to see them. Alternately, if this is terminal output, you can copy and paste it in your comment between triple backtick blocks e.g.

```
type or paste code here
```

to display as

type or paste code here

Thank you for the immediate response; I just edited my post.
Also, I have tried (and failed) to install newer versions of gcc (to have an updated gdb debugger).
As the system is a computing center, I am not allowed superuser (sudo) access.

You shouldn’t need to install GCC yourself. If you’re on a computing cluster, the cluster should provide you with different GCC versions which you can select. If your cluster uses “modules”, you may check to see whether GCC/4.9.3 or even more recent (I think GCC/9.3.0 is the most recent) version are available. You may also want to contact your sysadmin and ask.

The system mainly uses intel compilers, and has GCC 6.3 installed (that is what I used to produce the above backtraces). I tried and failed to install GCC 9.3,10.1, and 11.1 (on my own), but assumed GCC 6.3 would suffice for debugging.

Is 6.3 too old a GCC version?
I am in correspondence with the sysadmin, and he mentioned that any new updates/upgrades to packages will not occur for a couple of months (until the next maintenance day) - which is why I am installing Geant4 and the other necessary packages myself.
I can try again to upgrade GCC if it is necessary (this time with the help of the sysadmin).

GCC 6.3 should be plenty, since 4.9.3 is the “minimum required” version for G4.

Thank you. Do you have any thoughts on the segmentation faults mentioned previously?

Anyone? Any information would be greatly appreciated.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.