Working with HDF5

Recording, Visualizing and Persisting Data
1

Dear Experts,

Given that a simulation outputs a lot of data, simple txt files would become huge and increasingly difficult to manage. Hence, I was thinking of using HDF5 format to store the data.
Are there any examples that can show how to use this feature (as it is the case for .root files)?

Maybe someone has additional suggestions/experience regarding the idea to use HDF5. Especially using it for Geant Applications in multithread mode. What I learned so far is, that HDF5 doesn’t provide full thread safety for C++ applications.

Thanks a lot in advance!

2

@Nikita I’m currently attempting to get some code running that saves the output of my simulation to an HDF5 file. The problem that I currently have is with keeping the file open and passing new events to it in order to free up memory. I’m using the hdf5.h with version 10.5. While I haven’t yet gotten this to work for my case, perhaps I can be of some assistance in getting you started with yours.

3

Hi - very nice to hear that you try something similar! Some further insight is much appreciated. Do you have a working example or some code snippet that can show how to approach this problem?

Also, I guess you’re not using a multithreaded application? Because that was another issue I was facing. HDF5 appears not to work with multithreading.

Anyway, I’d be very interested to learn what you already have - thanks in advance!

4

Hi,

You can use any Geant4 example with G4 analysis manager (eg. basic examples B4 and B5), and just to change the output type by changing the g4root.hh include inB4[5]Analysis.hh with g4hdf5.hh:
//#include “g4root.hh”
#include “g4hdf5.hh”

If there is activated ntuple merging, like in example B4, then you need to comment out this setting, as it is not available with Hdf5:
// analysisManager->SetNtupleMerging(true);

Besides this incompatibility no other changes are required for switching the output type.

Best regards,

5

Hi @Nikita,

I apologize for the delayed response. I’ve been under the weather lately and haven’t been able to get any work done. Here is some snippets of the code I’m working with:

G4int CobainEventAction::OpenFile(const G4String filename) {
	G4int fROOTOutputTree = 0;
	G4int fROOTOutputFile = 2;
 	std::cerr << "CobainEventAction::OpenFile\n";
	hid_t           file, space, dset, dcpl;
  herr_t          status;
  H5D_layout_t    layout;
  hsize_t         dims[2] = {DIM0, DIM1},
                  start[2],
                  stride[2],
                  count[2],
                  block[2];
	G4int             wdata[DIM0][DIM1],
									rdata[DIM0][DIM1],
									i, j;


  //    Initialize data to "1", to make it easier to see the selections.


    for (i=0; i<DIM0; i++)
        for (j=0; j<DIM1; j++)
            wdata[i][j] = 1;





    //  Create a new file using the default properties.

    file = H5Fcreate (filename, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
		std::cout << "h5file = " << file <<std::endl;


		//FillData(file, space, dcpl, dset);
		space = H5Screate_simple (2, dims, NULL);
	  std::cout << "h5 space = " << space <<std::endl;

		dcpl = H5Pcreate (H5P_DATASET_CREATE);
	  std::cout << "h5 dcpl = " << dcpl <<std::endl;
		//status = H5Pset_chunk (dcpl, 2, chunk);

		dset = H5Dcreate (file, DATASET, H5T_STD_I32LE, space, H5P_DEFAULT, dcpl, H5P_DEFAULT);
		status = H5Dwrite (dset, H5T_NATIVE_INT, H5S_ALL, space, H5P_DEFAULT,
								wdata[0]);
		std::cout << "status " << status << std::endl;


    printf ("Original Data:\n");
    for (i=0; i<DIM0; i++) {
        printf (" [");
        for (j=0; j<DIM1; j++)
            printf (" %3d", wdata[i][j]);
        printf ("]\n");
    }


	 status = H5Pclose (dcpl);
	 status = H5Dclose (dset);
	 status = H5Sclose (space);
	 //status = H5Fclose (file);
	 status = H5Funmount (file, filename);
	 hid_t f = H5Freopen(file);


	 //H5Funmount(file, filename);



	if (fROOTOutputFile != 0) {
		std::cerr << "closing output file\n";

  	}

  	
  	if (fROOTOutputFile == 0) {
    	G4cerr << "Could not open HDF5 output file "<< filename << std::endl;
		return 0, 0;
  	}

	return file, space;
}

void CobainEventAction::FillData(G4int eventNumber,G4String particleName,G4double energyDeposited,G4int volume, hid_t dset)//(file, space, dcpl, dset)
{
			hid_t           file, dcpl;    /* Handles */
	  	hsize_t         dims[2] = {DIM0, DIM1};
			G4int             wdata[DIM0][DIM1],          /* Write buffer */
		                  rdata[DIM0][DIM1],          /* Read buffer */
		                  i, j;
	    /*
	     * Write the data to the dataset.
	     */
			 /*
				* Create dataspace.  Setting maximum size to NULL sets the maximum
				* size to be the current size.
				*/
			//G4String filename = "test.h5";
			
			std::cout << "h5 file = " << dset << std::endl;
			hid_t h5f = H5Freopen(dset);


			std::cout << "h5 dset = " << dset <<std::endl;
			 std::cout << "dims = " << dims << std::endl;
			 std::cout << "space = " << space << std::endl;
			 std::cout << "event = " << eventNumber << std:: endl;
			 wdata[0][0] = float(0);
			 wdata[0][1] = eventNumber;
			 wdata[0][2] = energyDeposited;
			 wdata[0][3] = volume;
			 std::cout << "wdata = " << wdata << std::endl;
			 printf ("\nData as read from disk by hyperslab:\n");
 	    for (G4int i=0; i<DIM0; i++) {
 	        printf (" [");
 	        for (j=0; j<DIM1; j++)
 	            printf (" %3d", wdata[i][j]);
 	        printf ("]\n");
 	    }
	    status = H5Dwrite (dset, H5T_NATIVE_INT, H5S_ALL, space, H5P_DEFAULT,
	                wdata[0]);
			std::cout << "status " << status << std::endl;
}

I also have my entire code up on github at:
https://github.com/glicka/AdamG4

6

Hi @glicka, @ivana!

I’m currently caught up in another problem, hence my very slow response. Sorry for that!
I will have a look into the code - the given suggestions might already get me close to what I need.

Thanks a lot!