Histogram uses too much memory

Hello,
I am trying to create 3d histogram with dimensions 10000x10000x2 as:

G4AnalysisManager* analysis = G4AnalysisManager::Instance();
analysis->SetDefaultFileType(“hdf5”);
analysis->CreateH3(“Name”, “Title”, 10000, 0.0, 1.0, 10000, 0.0, 1.0, 2, 0.0, 60.0);

I am creating histogram in RunAction class constructor. RunAction class is allocated in ActionIntialization::BuildForMaster() and ActionIntialization::Build().
Geant4 verison: geant4-11-00-patch-03 [MT]. Number of used threads is 1.

My approximation is the histogram size should be around (with underflow and overflow bins): 10002x10002x4x(4 bytes) ≈ 1.5 Gb. However, after starting the program, I see that the memory used for histogram creating is 50 Gb.

My question is why the size of histogram is so big? Maybe I creating histogram in wrong way or its normal memory usage for such histogram?

There are a few things that are likely getting you a large factor; let’s see if they get you x33. . .

First, the histogram will be storing doubles, not floats, so that’s 8 bytes per value (x2).

Next, each bin stores more than the accumulated sum of values: it also stores the count of entries (integers, so x1.5), the sum of the weights (x2) and the sum of squared values (x2). All those are needed to compute error bars.

That gets us to a factor of x12, but not x33. I don’t know whether there’s additional overhead for the HDF5 format, or whether G4Analysis has it’s own internal representation across all file types.

With plain ROOT, there’s another factor of about 2 overhead, because it keeps both a memory copy and a write-to-file buffer copy of things.

1 Like

Thank you! Now the G4 histogram structure is much clearer.
With ROOT allocates about the same memory size. Probably HDF5 also has overhead as ROOT.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.