Use in multithreaded application

[bojohnson5]

I'm writing some software to control data collection. I'm attempting to write out the data I collect with your crate but am running into problems from what I think is my program being multithreaded. I have a couple of data-taking threads and then a data processing thread and where the data processing occurs I open an HDF5 file and try to write. But my program quits as soon as I open the file. Is this crate usable in my environment? Or is there something I'm not aware of with how Rust works in this situation? I have tested with a normal file using std::fs and there isn't an issue there.

Thanks for the help!

[magnusuMET]

hdf5-c is not safe in a multithreaded environment due to e.g. global caches. In this crate all accesses to hdf5-c are serialised, and the most efficient access pattern is having one thread doing reading/writing, as it sounds like you are doing. What sort of errors do you get? If the failure is silent, what signals are sent?

[magnusuMET]

Very strange that the error just disappeared.

You can do a resize at the beginning where you guess the size of the input then resize back to actual size to save a lot of calls to resize. You might want to batch data before saving, the most efficient way depends on your application, but you can try ~4MB or so if you plan on chunking and compressing the data.

For use of write_slice it would be something like the following

 // ds = ... 3D?
// arr = ...data from the instrument
// istep
ds.write_slice(arr, (istep, .., ..))?;

[bojohnson5]

Yeah I agree. It must have been how I was handling the errors and then just using unwrap avoided the issues?

And those ellipses in the call to write_slice, are those actually there or do I need to fill those in?

[magnusuMET]

The ellipses are supposed to be there if you intend to write everything along that dimension, it is a RangeFull meant to signify ALL.

[bojohnson5]

I appreciate all the help @magnusuMET!

[bojohnson5]

I'll close this since the multithreading issue appears to be gone and open another discussion for other questions I might have!