So I have a simulation which appears to be working exactly like I want when running in single thread mode. However, when I switch to multithreaded mode an issue arises:
First, in my RunAction::RunAction I create the ntuple to save to my run specifc metadata in:
RunAction::RunAction()
{
auto analysisManager = G4AnalysisManager::Instance();
analysisManager->SetFileName("Shells");
analysisManager->SetDefaultFileType("root");
analysisManager->SetNtupleMerging(true);
analysisManager->CreateNtuple("Run", "Run specific information");
analysisManager->CreateNtupleIColumn("RandomSeed");
analysisManager->FinishNtuple(0);
}
Then in the RunAction::EndOfRunAction I save the data:
However, I get this warning from the master thread:
-------- WWWW ------- G4Exception-START -------- WWWW -------
*** G4Exception : Analysis_W001
issued by : G4TNtupleManager<NT,FT>::FillNtupleTColumn
Ntuple 0 does not exist.
*** This is just a warning message. ***
-------- WWWW -------- G4Exception-END --------- WWWW -------
I was able to recreate this in the B4c example by adding analysisManager->FillNtupleDColumn(0, 10); to the RunAction::EndOfRunAction. Does anyone know why this happens and how to fix it?
You should not call the Fill functions on master. The analysis manager is designed for collecting data during event processing, which in MT mode happens on workers. The analysis manager instance on master then take care of merging analysis objects which is triggered by the call to the Write function.
You can fix your code by adding a test:
if (! isMaster) {
analysisManager->FillNtupleIColumn(0, 0, CLHEP::HepRandom::getTheSeed());
analysisManager->AddNtupleRow(0);
}
Thank you that has solved this issue! I have one more issue that is very similar to this, in my EventAction::EventAction I am trying to create an ntuple to save the event specific data which contains ~12 vectors:
Then in my EventAction::EndOfEventAction I fill the columns. However, I’m getting a similar warning:
G4WT1 >
-------- WWWW ------- G4Exception-START -------- WWWW -------
*** G4Exception : Analysis_W001
issued by : G4RootPNtupleManager::FillNtupleTColumn
ntuple id= 1 does not exist.
*** This is just a warning message. ***
-------- WWWW -------- G4Exception-END --------- WWWW -------
and an eventual segfault. I assume the only way to fix this is to move the ntuple creation to the RunAction and use getters from the EventAction, like what is done in example B5? Or do you know of a better way to handle many vectors like this?
Yes, when ntuple merging is active, ntuple must be created also on master, that’s why we create all objects in the RunAction constructor.
So you need to add the getters in a similar way as demonstrated in example B5.
Thank you Ivana, I managed to get everything mostly working now. I just have 1 last issue (i hope), I’m trying to save run specific information (for example the seed i use) but because im only using Fill when not on the master thread:
if (! isMaster) {
analysisManager->FillNtupleIColumn(0, 0, CLHEP::HepRandom::getTheSeed());
analysisManager->AddNtupleRow(0);
}
I’m getting entries for each thread (instead of just 1) and storing the threads’ seeds rather the master’s one.
When ntuple merging is active, the ntuple on the master collects data from worker threads and it cannot be filled by the user calls. This design reflects the fact that when Geant4 runs in multithreaded mode, the event processing is performed on workers only, and so we do not expect filling data on master.
You can use the Geant4 UI command for saving random number status:
/random/setSavingFlag true
This will activate saving the random generator status at each begin of run, but the status will be then saved on both master and workers.
So is it not possible to only save the master run specific information to an ntuple in a multithreaded simulation?
I basically want to run a simulation consisting of 1 run and many events over 128 workers. And have an ntuple with 1 entry for the run specific metadata (seed, macro info, git ID). So I ideally wouldn’t store this info on every worker.
So is it not possible to only save the master run specific information to an ntuple in a multithreaded simulation?
Unfortunately, as I wrote above, such use case was not considered in the design.
I basically want to run a simulation consisting of 1 run and many events over 128 workers. And have an ntuple with 1 entry for the run specific metadata (seed, macro info, git ID). So I ideally wouldn’t store this info on every worker.
As a work-around, if you are sure that your number of events guarantees that all workers participate in event processing, you can call Fill only on one selected worker, eg. the one with thread id = 0.
Something like this could work:
Hi, @ivana ! We have the same use case in SuperCDMS – we store geometry layout information, job-level (not event-thread level) random seeds, and collect timing data in the master thread. Because our simulation framework predated a lot of G4Analysis, our OutputManager talks to ROOT directly. It handles both master and worker thread accumulation via a global singleton with mutexes (ugh).
