Given location find `root_id`/ `segment_id`

[Mohinta2892]

Hi !

I have been trying to understand how to use the caveclient to programmatically fetch the root_id given a XYZ location.

Is there an easy way to do this?

Thanks!

[bdpedigo]

Spatial lookups like this can be done with cloudvolume. Assuming cloudvolume is installed (pip install caveclient[cv]), you can do something like this:

from caveclient import CAVEclient

client = CAVEclient(datastack_name)
cv = client.info.segmentation_cloudvolume()
cv.scattered_points(
   points, # points to look up
   coord_resolution=cv.meta.resolution(0), # resolution those points are specified in, here I specify voxel space
   agglomerate=False # lookup supervoxels, if you want root set agglomerate=True but this adds overhead
)

[Mohinta2892]

Thank you so much @bdpedigo!
This has worked :)

[Mohinta2892]

Hi @bdpedigo ,

Is it possible to do the opposite?
Given a root_id find all node locations in space (volume) that belong to that root_id?
Currently, I can perform mesh.get(root_id) but cannot call skel.get(root_id).

[bdpedigo]

what do you mean by "node locations" here? specifically skeleton nodes?

[bdpedigo]

if so, there is a skeleton service in CAVE, but that would have to be already enabled and running for your datastack: https://caveconnectome.github.io/CAVEclient/api/skeleton/

the cloudvolume interface for getting skeletons will work if you have many skeletons already computed in a cloud bucket somewhere, and they are formatted in the way neuroglancer/cloudvolume expects with an info file etc.

there is also a level2 node cache service https://caveconnectome.github.io/CAVEclient/api/l2cache/ which gives something close to the skeleton nodes (just the locations of small chunks of the segmentation), but again, I am not sure if that is enabled for the datastack you are working on

[Mohinta2892]

Thank you @bdpedigo. I checked that these services are not enabled for our datastack. They seem to require additional ingestions to other BigTables to enable them. Is that correct?

As another workaround, I tried to download the proofread datastack like below. However, I am unable to find root_ids after correction in this downloaded precomputed volume.
Is it because the underlying fragments never gets changed and the updates only happen in a cave table like a mapping change between old to new fragment ids (this is my current understanding)?
If so, how can we extract a proofread volume from FlyWire using CAVE?

    client = CAVEclient(datastack_name=datastack_name, server_address=server_address)
    source_cv = client.info.segmentation_cloudvolume(agglomerate=True) # get the agglom version

    info = source_cv.info.copy()

    dest_cv = CloudVolume(
        output_path,
        mip=0,
        info=info,
        cache=False,
        parallel=False,
        non_aligned_writes=True
    )
    dest_cv.commit_info()

[bdpedigo]

I tried to download the proofread datastack

I am not sure what you mean here - are you wanting to download a small cutout? The entire datastack would normally be quite large.

I am also not sure what you are trying to do with your code snippet, it looks like you are just preparing a bucket/path for writing data but have only written the info file.

how can we extract a proofread volume from FlyWire

again, what exactly do you mean here? the entire segmentation would be quite massive so maybe another question is what do you want to do with the entire segmentation?

[bdpedigo]

They seem to require additional ingestions to other BigTables to enable them. Is that correct?

I am not entirely sure what is required, but this is for FlyWire? or something else? @fcollman or @ceesem know more about what would be involved in setting these up - I know there are some brief instructions here https://github.com/CAVEconnectome#deployment but not sure how up to date the instructions are

[Mohinta2892]

Sorry, should have provided more context.

We have a neuron segmentation model that we would like to fine-tune on proofread data. Fine-tuning needs to happen on complete neurons.
So we have proofread a bunch of neurons to completion in FlyWire and now we need extract their segmentation out of Flywire to use as training data.
It is quite large, but I am not sure how else extract only the data corresponding to the regions (ids) that have been proofread.

Re. the snippet: Yes, i am creating a bucket for now, which would be written to chunk by chunk. As I can load the volume at once, I pull non-overlapping chunks via multiple workers that write to this bucket independently.