Problems compiling without visualization libraries

Getting Started
1

Hi,
I have made a simple calorimeter, similar to the B4a example on my local CMS tier2 cluster. Interactively it runs fine, but when I try to run it on our condor system, it crashes because it can not find the graphics librarires like /usr/lib64/libX11.so. These are not installed on the batch nodes, only on the interactive nodes. I tried doing the cmake using -DWITH_GEANT4_UIVIS=OFF and I also tried setting this directly to OFF in the CMakeLists.txt, but the exe still seems to pick up the graphics libraries in /usr/lib64. I enclose a dump of a session (including pulling the code from gitlab) below, including the “grep” that shows that it is pulling in these libraries. Any suggestions on how to do a link that wouldn’t include these libraries would be appreciated.

[eno@hepcms-in2 dualReadout]$ git clone ssh://git@gitlab.cern.ch:7999/eno/cepc_calotiming.git

Initialized empty Git repository in /data/users/eno/dualReadout/cepc_calotiming/.git/

remote: Enumerating objects: 211, done.

remote: Counting objects: 100% (211/211), done.

remote: Compressing objects: 100% (101/101), done.

remote: Total 211 (delta 134), reused 172 (delta 107)

Receiving objects: 100% (211/211), 113.57 KiB, done.

Resolving deltas: 100% (134/134), done.

[eno@hepcms-in2 dualReadout]$ cd cepc_calotiming/

[eno@hepcms-in2 cepc_calotiming]$ source ./g4env.sh

Setting up the environment for Geant4 10.5 for use with CMake

[eno@hepcms-in2 cepc_calotiming]$ cmake -DGeant4_DIR=/cvmfs/geant4.cern.ch/geant4/10.5/x86_64-slc6-gcc63-opt/lib64/GEANT4-10.5.0 -DWITH_GEANT4_UIVIS=OFF

– The C compiler identification is GNU 6.2.0

– The CXX compiler identification is GNU 6.2.0

– Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin/gcc

– Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin/gcc – works

– Detecting C compiler ABI info

– Detecting C compiler ABI info - done

– Detecting C compile features

– Detecting C compile features - done

– Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin/g++

– Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin/g++ – works

– Detecting CXX compiler ABI info

– Detecting CXX compiler ABI info - done

– Detecting CXX compile features

– Detecting CXX compile features - done

– Configuring done

– Generating done

– Build files have been written to: /home/eno/dualReadout/cepc_calotiming

[eno@hepcms-in2 cepc_calotiming]$ grep -r “/usr/lib64” *

CMakeFiles/CMakeOutput.log: collapse library dir [/usr/lib/…/lib64] ==> [/usr/lib64]

CMakeFiles/CMakeOutput.log: implicit dirs: [/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib]

CMakeFiles/CMakeOutput.log: collapse library dir [/usr/lib/…/lib64] ==> [/usr/lib64]

CMakeFiles/CMakeOutput.log: implicit dirs: [/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib]

CMakeFiles/3.11.1/CMakeCCompiler.cmake:set(CMAKE_C_IMPLICIT_LINK_DIRECTORIES “/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib”)

CMakeFiles/3.11.1/CMakeCXXCompiler.cmake:set(CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES “/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib”)

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXm.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libSM.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libICE.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libX11.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXext.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXt.so

CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXmu.so

[eno@hepcms-in2 cepc_calotiming]$

2

You probably need Geant4 itself built without visualization – because the Geant4 libraries were built with visualization, some of them are linked to the libraries that do not exist so it really doesn’t matter how you build your downstream application because you are going to end up trying to link to Geant4 libraries that have symbols that cannot be resolved.

3

Do you know if maybe there is some way I can fake it out? Copy the libraries to some local area that is mounted in the batch modes and tell it to look there during the build?

4

Also, in case there are people working at CERN on this, this is my g4env.sh. If there is some way to get those libraries via cvmfs or get the geant library built without visualization there?

source /cvmfs/sft.cern.ch/lcg/contrib/gcc/6.3/x86_64-slc6/setup.sh
source /cvmfs/geant4.cern.ch/geant4/10.5/x86_64-slc6-gcc63-opt-MT/CMake-setup.sh
export CXX=which g++
export CC=which gcc
export PATH=$PATH:/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.11.1/Linux-x86_64/bin
source /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/setup.sh
source $ROOTSYS/bin/thisroot.sh

5

Yes, you could copy the libraries to some folder and add the path to that folder to LIBRARY_PATH and LD_LIBRARY_PATH in the same way you appended PATH above. The former is used during compilation, the latter during runtime. Run ldd on the library files and look for any entries that are “not found” to help figure out which libraries you need to copy over.

6

I personally have found the cvmfs configuration to be a huge mess with respect to installations. I’m not surprised you are having issues. I gave up trying to debug a MT version of the LZ code because there was no possible environment configuration that let me compile and run the code and use the GBD debugger because the environment I was required to use included a different Python version than the one GDB wanted so it was importing Python 2 code from Python 3 or maybe vice versa. Good luck.

7

Before the cmake and the make, I did
export LIBRARY_PATH=$LIBRARY_PATH:/home/eno/dualReadout/fakelib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/eno/dualReadout/fakelib

and I did put the libraries there
[eno@hepcms-in1 cepc_calotiming]$ ls /home/eno/dualReadout/fakelib/
libGL.so libGLU.so libICE.so libSM.so libX11.so libXext.so libXm.so libXmu.so libXt.so

but the build still got them from /usr/lib64
[eno@hepcms-in1 cepc_calotiming]$ grep -r “/usr/lib64” *
CMakeFiles/CMakeOutput.log: collapse library dir [/usr/lib/…/lib64] ==> [/usr/lib64]
CMakeFiles/CMakeOutput.log: implicit dirs: [/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/home/eno/dualReadout/fakelib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib]
CMakeFiles/CMakeOutput.log: collapse library dir [/usr/lib/…/lib64] ==> [/usr/lib64]
CMakeFiles/CMakeOutput.log: implicit dirs: [/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/home/eno/dualReadout/fakelib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib]
CMakeFiles/3.11.1/CMakeCCompiler.cmake:set(CMAKE_C_IMPLICIT_LINK_DIRECTORIES “/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/home/eno/dualReadout/fakelib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib”)
CMakeFiles/3.11.1/CMakeCXXCompiler.cmake:set(CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES “/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc/x86_64-pc-linux-gnu/6.2.0;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib/gcc;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib64;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64;/lib64;/usr/lib64;/cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-slc6-gcc62-opt/lib;/home/eno/dualReadout/fakelib;/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib”)
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXm.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libSM.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libICE.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libX11.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXext.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXt.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libXmu.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libGL.so
CMakeFiles/CEPC_CaloTiming.dir/build.make:CEPC_CaloTiming: /usr/lib64/libGLU.so

Is there something else I need to do for this to work? for it to get the libraries from the local copy I made?

8

The /usr paths are still there as I’d guess that even with UI/Vis turned off, there’ll still be a link to G4interfaces which does link to system graphics libs as configured on the LCG install.

Is the application that gets run on the batch nodes installed, i.e. from where ‘make install’ was run? The LD_LIBRARY_PATH setting will only make sense at application runtime, i.e. in your batch script you should be able to do something like

export LD_LIBRARY_PATH=/where/you/copied/X11libs:$LD_LIBRARY_PATH
...
run executable

If that doesn’t work, then it would be best to build a local Geant4 without any ui/vis.