OME-Zarr rendering for big images

[will-moore]

In #619 I have started looking at OMERO.figure support for OME-Zarr images.

This has shown some nice progress, using zarrita.js to load zarr chunks and https://github.com/will-moore/ome-zarr.js to render the chunks to rgb images.
This can handle any channel colours and includes rendering with a large number of different LUTs (corresponding to all those currently in OMERO).

This rendering approach is very simplistic (uses a for loop to iterate through pixels, adding rgb values) but it works fine when we are viewing a whole image. We can choose an appropriate dataset resolution level, for example, this 2k x 2k image
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0101A/13457227.zarr
we load the 1k x 1k resolution, and the 4 active channels are rendered from zarr chunks to RGB in ~50 ms on my laptop.

This may not match the WebGL rendering speeds achieved by e.g. Deck.gl in vizarr which renders while sliding the channel slider, but is certainly sufficient for OMERO.figure which only renders when you stop the slider.

Even "BIG" images don't really cause a problem because we can choose an appropriate resolution level to load.

Currently, the zarrita array is kept in memory for each image, so that we don't need to re-load .zarray etc on each render.
In fact, we are also caching the zarr chunks too (e.g. cache key is /Z/C/T), so that for each re-render of the channel sliders, we don't need to load any more data.

The limitations so-far encountered:

• When we need to load more chunks (e.g. move the Z/T slider), both the figure panel and the right panel viewer request the image to be rendered at the same time and so we load duplicate data. This could be fixed without too much work by keeping track of active requests and not duplicating them.
• When we zoom in to a non-big Image (e.g. the image above) we still render the whole image plane each time - same behaviour as when rendering from OMERO. For zarr, we could be cleverer and try to render just a portion of the image (although for the image above, each Z/T plane is a single chunk up to 2k x 2k for highest resolution, so this wouldn't help for that image!).
• For BIG images (tiled images in OMERO), OMERO.figure doesn't load tiles but instead requests the viewport region to be rendered /render_image_region/ at the appropriate resolution. We currently do the same with zarr data and render the region we want at the correct resolution. However, because we're loading a slightly different region each time, it's hard to cache by region and we're not loading tiles. So, we end up loading a lot more data each re-render.

These issues, along with other concerns about the usability of working with BIG Images in OMERO.figure and the report at https://nasa-impact.github.io/zarr-visualization-report/ have prompted me to step back and consider various alternative options for image viewers in Figure.

Viv, Deck.gl

This is the tech used by vizarr. It is most convenient to use within a React application. I have tried some examples (a while back) at https://github.com/will-moore/viv-examples including without React.

Pros:

• Good zarr and OME-Zarr support
• Big Image viewer handling tiling etc

Cons:

• Not possible to use a separate Deck.gl canvas for every panel in a figure due to memory issues. I tried this for rendering a bunch of thumbnails in a gallery at https://github.com/ome/ome-zarr-catalog/blob/230ccd8ef33b91765b0ba297769056faf7630388/src/Viewer.jsx but the page would run out of memory with about 6 or more Deck.gl canvases on the same page.
• Limited rendering of colors and no support for LUTs?
• No built-in support for Shapes etc. Would probably have to use existing Shapes-editor over the top. There are Deck.gl layers for shapes, but I think allowing users to edit them would be a big job.

Carbonplan maps

As discussed in the report above, https://github.com/carbonplan/maps uses map-box map with a layer that uses regl to render zarr data using WebGL. E.g. see Tiles at https://github.com/carbonplan/maps/blob/main/src/tiles.js#L207.

Pros:

• As for Deck.gl, nice smooth big image rendering of zarr data.
• LUTs support on the demo (not looked at the implementation yet)

Cons:

• How easy is it to handle non-map data? And to integrate it into a custom viewer?
• As above, no easy support for ROI Shape rendering, drawing and editing

However, it may be possible to adapt the Tile rendering code that uses regl and add support for OME-Zarr to iviewer...

OMERO.iviewer

The iviewer uses a custom OpenLayers map that was originally designed to be a standalone viewer (without all the Aurelia.js UI dependencies). Although it is no-longer built as a separate standalone JS bundle, it should be possible to revive this, allowing the viewer to be used outside of the iviewer app (which could be very handy for anyone wanting to build custom viewers).

If we can add zarr support to iviewer, then it might be possible to use the viewer within OMERO.figure.

Pros:

• We already have all the Shape Editing logic we want.
• It would be useful for other apps and use-cases.
• Adding zarr support to iviewer would also be useful in it's own right.
• We get a 1st-class big image viewer in OMERO.figure.

Cons:

• Increase the bundle size of OMERO.figure (include OpenLayers etc)
• Lots of integration work to do
• Not sure if it would work to use iviewer for all the panels on a figure, especially if we're using WebGL regl etc. Would it suffer memory issues as with Deck.gl?
• Might need to use existing ome-zarr.js rendering for all the figure panels and just use iviewer for the right panel and ROIs dialog. But then we'd still be loading the zarr data twice when the right panel and figure panels are updated.

Stick with existing approach

Pros:

• it mostly works pretty well.
• All existing code works too - Shape-Editor, overlay of multiple images when batch-selecting panels,

Cons:

• No tile-based approach for big images mean that we load more zarr data