Best way to write a stream of images?

[Axel-Jacobsen]

Howdy!

We are trying to write a stream of images to disk at ~30 FPS - that is, we want our loop of get image -> do work -> write image to run at ~30 FPS. Compute is restricted, as we are running on a Rasberry Pi 4. We have the zarr writer in a thread so it can write each image in the background (using a ThreadPoolExecutor)

We are using zarr as follows:

• Top level store is a ZipStore (self.store = zarr.ZipStore(filename, mode='x'))
• We have a group using that store (self.group = zarr.group(store=self.store))
• For each image, we create a new array (self.group.array(name, data=data, compressor=None))

We get each image at up to ~50 FPS. Each image is (772, 1032) u8s.

Our problem is this: when running the program with all of the work mentioned above, we think that we can't write to disk fast enough - submitting to the thread pool is fast, but process memory keeps building up until we run out of memory. When not doing work, we can write fast enough and memory doesn't build up.

Speeding up writing to disk would be very helpful. We suspect that some tweaking of how we are saving could be the ticket. Thanks for the help!

[rabernat]

Have you tried using a regular directory store to see if the ZipStore is potentially limiting your throughput?

[Axel-Jacobsen]

The directory store and the zip store give the same performance. Could it be the way we are creating arrays - i.e. one image per array? Would it be more efficient to have many images stored in the same array?

[rabernat]

Would it be more efficient to have many images stored in the same array?

Yes because no metadata operations are required to write new data. The fastest performance would be to pre-initialize the array to the shape you expect it to have and then write independent chunks in parallel from threads.

[jakirkham]

Yeah agree with Ryan. Would add that specifying chunking to align with the size of each image (or set of images if working with multiple at once) would also help.

Can always zip up the result at the end if one needs it all in one file.

[shenker]

This thread has some relevant discussion: https://forum.image.sc/t/ome-zarr-chunking-questions/66794

[Axel-Jacobsen]

Sweet! Thanks all. I'll give these ideas a try.

[Axel-Jacobsen]

A quick thank you to you all! Here are the results:

The magic trick was from [here](constantinpape/z5#186 (comment) | width=100). You can initialize an empty zarr array of arbitrary size, and it will only allocate memory for initialized chunks. So making the chunk size chunks=(h,w,1) you can assign images as they come into the array like normal.