VecGeom and Cuda Errors

ParticleTruthSeeker · August 26, 2021, 1:43pm

I have successfully built G4.10.7.2 with vecgeom and the cuda backend however when I attempt to build an application I get the following errors

/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::CudaManager()':
CudaManager.cpp:(.text+0x60b): undefined reference to `cudaDeviceSetLimit'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::AllocateNavIndexOnCoproc()':
CudaManager.cpp:(.text+0xc97): undefined reference to `vecgeom::cuda::InitDeviceNavIndexPtr(void*, int)'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::AllocatePlacedVolumesOnCoproc()':
CudaManager.cpp:(.text+0x193a): undefined reference to `vecgeom::cuda::InitDeviceCompactPlacedVolBufferPtr(void*)'
/usr/bin/ld: CudaManager.cpp:(.text+0x19a6): undefined reference to `vecgeom::cxx::DevicePtr<char>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1a69): undefined reference to `vecgeom::cuda::InitDeviceCompactPlacedVolBufferPtr(void*)'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::AllocateGeometry()':
CudaManager.cpp:(.text+0x1ab7): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::LogicalVolume>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1b30): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::LogicalVolume>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1bfd): undefined reference to `vecgeom::cxx::DevicePtr<char>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1c61): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::Transformation3D>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1cde): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::Transformation3D>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1ce6): undefined reference to `vecgeom::cxx::DevicePtr<char>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1d29): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::Vector<vecgeom::cuda::VPlacedVolume const*> >::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1d6c): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::VPlacedVolume const*>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1e02): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::Vector<vecgeom::cuda::VPlacedVolume const*> >::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x1e14): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::VPlacedVolume const*>::SizeOf()'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::Synchronize()':
CudaManager.cpp:(.text+0x2493): undefined reference to `vecgeom::cxx::DevicePtr<vecgeom::cuda::VPlacedVolume const*>::SizeOf()'
/usr/bin/ld: CudaManager.cpp:(.text+0x24e0): undefined reference to `void vecgeom::cxx::DevicePtr<vecgeom::cuda::Vector<vecgeom::cuda::VPlacedVolume const*> >::Construct<vecgeom::cxx::DevicePtr<vecgeom::cuda::VPlacedVolume const*>, unsigned long>(vecgeom::cxx::DevicePtr<vecgeom::cuda::VPlacedVolume const*>, unsigned long) const'
/usr/bin/ld: CudaManager.cpp:(.text+0x25f9): undefined reference to `cudaMemcpy'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::CudaManager::PrintGeometry() const':
CudaManager.cpp:(.text+0xc11): undefined reference to `vecgeom::cuda::CudaManagerPrintGeometry(vecgeom::cuda::VPlacedVolume const*)'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(CudaManager.cpp.o): in function `vecgeom::cxx::DevicePtrImpl<char, vecgeom::cxx::DevicePtr<char> >::Allocate(unsigned long)':
CudaManager.cpp:(.text._ZN7vecgeom3cxx13DevicePtrImplIcNS0_9DevicePtrIcEEE8AllocateEm[_ZN7vecgeom3cxx13DevicePtrImplIcNS0_9DevicePtrIcEEE8AllocateEm]+0xd): undefined reference to `vecgeom::cxx::DevicePtr<char>::SizeOf()'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaCheckError(cudaError)':
Interface.cpp:(.text+0x11): undefined reference to `cudaGetErrorString'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaCheckError()':
Interface.cpp:(.text+0x35): undefined reference to `cudaGetLastError'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaAssertError()':
Interface.cpp:(.text+0x65): undefined reference to `cudaGetLastError'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaMalloc(void**, unsigned int)':
Interface.cpp:(.text+0x83): undefined reference to `cudaMalloc'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaCopyToDevice(void*, void const*, unsigned int)':
Interface.cpp:(.text+0x98): undefined reference to `cudaMemcpy'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaCopyFromDevice(void*, void const*, unsigned int)':
Interface.cpp:(.text+0xa8): undefined reference to `cudaMemcpy'
/usr/bin/ld: /usr/local/lib/libvecgeom.a(Interface.cpp.o): in function `vecgeom::cxx::CudaFree(void*)':
Interface.cpp:(.text+0xb1): undefined reference to `cudaFree'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/Collimator.dir/build.make:376: Collimator] Error 1

This seems like some sort of linker error though I don’t understand what the problem is.

Any help would be greatly appreciated.

Cheers

gcosmo · August 26, 2021, 1:58pm

The Cuda backend is not supported in Geant4.
You should configure VecGeom to use either the Scalar or Vector backends.

ParticleTruthSeeker · August 26, 2021, 4:56pm

Ok I rebuilt Vecgeom with the vc backend as before but without cuda and then rebuilt G4.

Now I have this error

/usr/bin/ld: /home/user/GEANT4-10.7.2/G4-Build/lib/libG4geometry.a(G4UPolyhedra.cc.o):(.data.rel.ro._ZTVN7vecgeom3cxx30CommonUnplacedVolumeImplHelperINS0_24PolyhedronImplementationIL11EInnerRadii0EL10EPhiCutout0EEENS0_15VUnplacedVolumeEEE[_ZTVN7vecgeom3cxx30CommonUnplacedVolumeImplHelperINS0_24PolyhedronImplementationIL11EInnerRadii0EL10EPhiCutout0EEENS0_15VUnplacedVolumeEEE]+0x70): undefined reference to `vecgeom::cxx::VUnplacedVolume::DistanceToInVec(vecgeom::cxx::Vector3D<Vc_1::Vector<double, Vc_1::VectorAbi::Sse> > const&, vecgeom::cxx::Vector3D<Vc_1::Vector<double, Vc_1::VectorAbi::Sse> > const&, Vc_1::Vector<double, Vc_1::VectorAbi::Sse> const&) const'
/usr/bin/ld: /home/user/GEANT4-10.7.2/G4-Build/lib/libG4geometry.a(G4UPolyhedra.cc.o):(.data.rel.ro._ZTVN7vecgeom3cxx30CommonUnplacedVolumeImplHelperINS0_24PolyhedronImplementationIL11EInnerRadii0EL10EPhiCutout0EEENS0_15VUnplacedVolumeEEE[_ZTVN7vecgeom3cxx30CommonUnplacedVolumeImplHelperINS0_24PolyhedronImplementationIL11EInnerRadii0EL10EPhiCutout0EEENS0_15VUnplacedVolumeEEE]+0x80): undefined reference to `vecgeom::cxx::VUnplacedVolume::SafetyToInVec(vecgeom::cxx::Vector3D<Vc_1::Vector<double, Vc_1::VectorAbi::Sse> > const&) const'
collect2: error: ld returned 1 exit status

amadio · August 27, 2021, 9:16am

The error above indicates that there is a potential mismatch between compile flags used for VecGeom and Geant4, i.e. which SIMD instructions are enabled for each one.

ParticleTruthSeeker · August 27, 2021, 10:20am

Is it only possible to enable a single instruction set in VecGeom through VECGEOM_VECTOR? The boolean variables do not accept changes after this is set. In which case Vc would also have to be built with that single instruction?
I disabled all simd instructions and rebuilt vc veccore vecgeom and geant4 but the problem is the same. Unless the builds don’t overwrite the previous installations.

amadio · August 27, 2021, 11:46am

What do you mean by this? Are you running make install for the builds you have?

ParticleTruthSeeker · August 27, 2021, 11:47am

Yes i am rebuilding in the same directory with make then make install. I do not manually delete the build and installation directories each time I alter the cmake parameters. I hope this is not an issue.

amadio · August 27, 2021, 11:53am

I think you should start from scratch just to make sure. VecGeom will install less things with CUDA=OFF and if you just overwrite you will have leftovers from previous builds.

ParticleTruthSeeker · August 27, 2021, 9:30pm

I deleted everything and recompiled with no SIMD extensions and it worked. However this seems to me slightly underwhelming considering the vector backend is being used for performance reasons.

weller · August 28, 2021, 1:33pm

what do you mean by „underwhelming“? can you please post your comparisons in numbers?

ParticleTruthSeeker · August 28, 2021, 1:39pm

Thats a good point. I will run some benchmarks with the various options and see what the difference is as far as the vc backend and simd instruction sets, if I manage to make them work together. Underwhelming in the sense that I seem to be unable to activate hardware simd instructions that accelerate the vc backend. Several presentations I have seen show that for those workloads you can get a x2 speedup with avx for instance.

I also have a slightly different problem now that while multiple threads are activated they dont seem to br fully utilised as seen in the system monitor.

system · September 4, 2021, 1:39pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.