Multithreading Hits/General issue

Recording, Visualizing and Persisting Data
1

Hello all,

I am running into an issue with multithreading and data storing. In particular, I was getting an error from my analysis class’ FillData function, specifically with lines like
SDman->GetCollectionID("correct collection name"))

First off, and probably most importantly, sometimes when I first run my executable, I get a segmentation error and/or double-free. When I re-enter the command, my run starts and errors eventually follow.

The errors include segmentation errors and lines like G4WT5 > < BCDHitsCollection> is not found across all my HitsCollections. Some threads don’t get this issue, others do. The segmentation errors eventually crash my run.

For example, Thread 4 will output these lines and then continue on with the rest of the FillData function, whereas Thread 10 will successfully find all collections. Shortly after, I will get a segmentation error, but with all threads outputting at the same time, it is hard for me to pinpoint what the cause is.

G4WT4 > Starting SDman->GetCollectionIDs
G4WT4 > <SCDHitsCollection> is not found.
G4WT4 > <TCDHitsCollection> is not found.
G4WT4 > <CalHitsCollection> is not found.
G4WT4 > <WHitsCollection> is not found.
G4WT4 > <NDHitsCollection> is not found.
G4WT4 > [SCD, TCD, BCD, CAL, W, ND] [-1, -1, 0, -1, -1, -1]
G4WT4 > FILLING PRIMARY NTUPLE…

G4WT10 > Starting SDman->GetCollectionIDs
G4WT10 > [SCD, TCD, BCD, CAL, W, ND] [4, 0, 1, 2, 3, 5]
G4WT10 > FILLING PRIMARY NTUPLE…

*Other threads also output, I just omitted them for readability.

Another interesting thing I noticed when I upped my verbosity was this output:

G4WT8 > In CALSD::ProcessHits()
G4WT9 > In CALSD::ProcessHits()
G4WT2 > In CALSD::ProcessHits()
… for over 100 lines in a row…
G4WT3 > In CALSD::ProcessHits()
G4WT6 > In CALSD::ProcessHits()
G4WT4 > In CALSD::ProcessHits()
G4WT8 > In CALSD::ProcessHits()
G4WT0 > In CALSD::ProcessHits()
… with the occasional…
G4WT1 > MyHit::MyHit():pLogV()
G4WT9 > In CALSD::ProcessHits()
… and on and on…

I have an idea that it may come from my calorimeter class, namely with the private variable

private:
int* CellID;

This variable is instantiated in my CALSD constructor,
CellID = new G4int[numberOfCells]; // numberOfCells is a G4int parameter
and later used in lines
if(CellID[copyID]==-1) ,
CellID[copyID] = (G4int) aHitCollection->insert( aHit ) - 1; ,
and MyHit* aHit = (*aHitCollection)[CellID[copyID]];

My worry is that this array used in my calorimeter is not thread-local, causing errors when multiple threads access it.

Any help would be appreciated, and I hope my message is detailed enough!

Thank you all very much,
Shivom

Geant4 Version: v11.3
Operating System: Linux

2

Update (and further questions):
I tested to see if all threads were accessing the same CellID variable, and I think the issue lies elsewhere (printing out CellID each time it was accessed gave me a different address for each thread).

When running ASan to see my memory issues, I get a few different error messages. In one scenario, events are simulated for a few seconds and then crash, the error message suggesting these lines are the issue:

Summary

===========================================================
#7 0x0000000000441c92 in G4THitsCollection< MyHit>::operatorunsigned long) const ()
#8 0x00000000004ed2c1 in SCDSD::ProcessHits(G4Step*, G4TouchableHistory*) ()
#9 0x00007f3e30cf99b9 in G4SteppingManager::Stepping() () from /home/shivompandey/G4/lib64/libG4tracking.so
#10 0x00007f3e30d0c92b in G4TrackingManager::ProcessOneTrack(G4Track*) () from /home/shivompandey/G4/lib64/libG4tracking.so
#11 0x00007f3e30d51e23 in G4EventManager::DoProcessing(G4Event*, std::vector<G4Track*, std::allocator<G4Track*> >, bool) () from /home/shivompandey/G4/lib64/libG4event.so
#12 0x00007f3e30e7506c in G4WorkerRunManager::DoEventLoop(int, char const, int) () from /home/shivompandey/G4/lib64/libG4run.so
#13 0x00007f3e30e22c4e in G4RunManager::BeamOn(int, char const*, int) () from /home/shivompandey/G4/lib64/libG4run.so
#14 0x00007f3e30e787e1 in G4WorkerRunManager::DoWork() () from /home/shivompandey/G4/lib64/libG4run.so
#15 0x00007f3e30e1a174 in G4MTRunManagerKernel::StartThread(G4WorkerThread*) () from /home/shivompandey/G4/lib64/libG4run.so
#16 0x00007f3e2cadbad4 in std::execute_native_thread_routine (__p=0x6030016046e0) at …/…/…/…/…/libstdc+±v3/src/c++11/thread.cc:82
#17 0x00007f3e2c689d22 in start_thread (arg=) at pthread_create.c:443
#18 0x00007f3e2c70ed40 in clone3 () at …/sysdeps/unix/sysv/linux/x86_64/clone3.S:81

===========================================================

In another scenario, my code crashes right after each run action creation for threads. The error message for that case is:

Summary

G4WT5 > Inside MyRunAction::MyRunAction() Constructor
G4WT1 > Inside MyRunAction::MyRunAction() Constructor
G4WT0 > Inside MyRunAction::MyRunAction() Constructor
G4WT3 > Inside MyRunAction::MyRunAction() Constructor
G4WT2 > Inside MyRunAction::MyRunAction() Constructor
G4WT6 > Inside MyRunAction::MyRunAction() Constructor
G4WT4 > Inside MyRunAction::MyRunAction() Constructor
G4WT7 > Inside MyRunAction::MyRunAction() Constructor
G4WT10 > Inside MyRunAction::MyRunAction() Constructor
G4WT8 > Inside MyRunAction::MyRunAction() Constructor
G4WT9 > Inside MyRunAction::MyRunAction() Constructor
==40030==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200235d3f0 at pc 0x7ff1b6c3a077 bp 0x7ff18c73d6c0 sp 0x7ff18c73ce68
READ of size 16 at 0x60200235d3f0 thread T1
#0 0x7ff1b6c3a076 in __interceptor_memmove (/lib64/libasan.so.6+0x3a076)
#1 0x7ff1b4744e07 in void std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> >::_M_realloc_insert<G4VSensitiveDetector* const&>(__gnu_cxx::__normal_iterator<G4VSensitiveDetector**, std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> > >, G4VSensitiveDetector* const&) (/home/shivompandey/G4/lib64/libG4run.so+0xb8e07)
#2 0x7ff1b44be82d in G4SDStructure::AddNewDetector(G4VSensitiveDetector*, G4String const&) (/home/shivompandey/G4/lib64/libG4digits_hits.so+0x3882d)
#3 0x7ff1b44bba09 in G4SDManager::AddNewDetector(G4VSensitiveDetector*) (/home/shivompandey/G4/lib64/libG4digits_hits.so+0x35a09)
#4 0x46d690 in MyDetectorConstruction::ConstructSDandField() (/home/shivompandey/G4/MyProgram/main+0x46d690)
#5 0x7ff1b474b372 in G4WorkerRunManager::InitializeGeometry() (/home/shivompandey/G4/lib64/libG4run.so+0xbf372)
#6 0x7ff1b46f9c9f in G4RunManager::Initialize() (/home/shivompandey/G4/lib64/libG4run.so+0x6dc9f)
#7 0x7ff1b46f0166 in G4MTRunManagerKernel::StartThread(G4WorkerThread*) (/home/shivompandey/G4/lib64/libG4run.so+0x64166)
#8 0x7ff1b02dbad3 (/lib64/libstdc++.so.6+0xdbad3)
#9 0x7ff1afe89d21 in start_thread /usr/src/debug/glibc-2.34-125.el9_5.1.alma.2.x86_64/nptl/pthread_create.c:443
#10 0x7ff1aff0ed3f in __clone3 (/lib64/libc.so.6+0x10ed3f)

0x60200235d3f0 is located 0 bytes inside of 16-byte region [0x60200235d3f0,0x60200235d400)
freed by thread T4 here:
#0 0x7ff1b6cb73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x7ff1b4744e1b in void std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> >::_M_realloc_insert<G4VSensitiveDetector* const&>(__gnu_cxx::__normal_iterator<G4VSensitiveDetector**, std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> > >, G4VSensitiveDetector* const&) (/home/shivompandey/G4/lib64/libG4run.so+0xb8e1b)

previously allocated by thread T4 here:
#0 0x7ff1b6cb6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
#1 0x7ff1b4744e4d in void std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> >::_M_realloc_insert<G4VSensitiveDetector* const&>(__gnu_cxx::__normal_iterator<G4VSensitiveDetector**, std::vector<G4VSensitiveDetector*, std::allocator<G4VSensitiveDetector*> > >, G4VSensitiveDetector* const&) (/home/shivompandey/G4/lib64/libG4run.so+0xb8e4d)

Thread T1 created by T0 here:
#0 0x7ff1b6c587d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
#1 0x7ff1b02dbba8 in std:: thread::_M_start_thread(std::unique_ptr<std::thread ::_State, std::default_delete< std:: thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xdbba8)

Thread T4 created by T0 here:
#0 0x7ff1b6c587d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
#1 0x7ff1b02dbba8 in std:: thread::_M_start_thread(std::unique_ptr<std:: thread::_State, std::default_delete< std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xdbba8)

Another question I have is the best way to implement an analysis class. In sequential mode, I had a static analysis class/instance that did all the booking and filling. To adapt it for multithreading, I have replaced all the ROOT Tree filling with the G4AnalysisManager class and tried G4ThreadLocalSingleton, but I still am unsure about the best thread-friendly method (G4ThreadLocalSingleton, non-singleton approach, mutex, or abandon the external class and just use AnalysisManager in RunAction).

I fear that I’ve dug myself into a hole of multi-thread confusion.

Thank you for the help,
Shivom