Motion trajectories & pose extracted data for animal beh research

[yarikoptic]

Your idea

Popular relevant toolkits used

• SLEAP (attn @talmo) : Social LEAP Estimates Animal Poses - storing into .pkg.slp (HDF5). Some more pointers: Where to find description of .pkg.slp file format? talmolab/sleap#2093 (comment) . There are functionalities to convert to https://github.com/rly/ndx-pose (NWB extension)
• DeepLabCut (attn @MMathisLab): DeepLabCut uses HDF5 internally too, and can convert HDF5 to NWB with implementation in https://github.com/DeepLabCut/DLC2NWB and the end using the https://github.com/rly/ndx-pose .

Even though that section was primarily aiming for hardware-based motion tracking of humans (AFAIK; e.g. like point-light and other systems), I think it would be great to review the Motion BIDS data type which was and see how it could/should be extended to be able to capture results of the processing from SLEAP, DeepLabCut, or any other similar tool. We might need to workout a BEP and also see if some common format could be established for tool-agnostic format.

Also attn @bids-standard/bep032

Related issues

• BEP for audio/video capture of behaving subjects #1771

[talmo]

Relevant:

• Unifying library by @niksirbi and colleagues: https://github.com/neuroinformatics-unit/movement
• NWB extension by @rly: https://github.com/rly/ndx-pose

[MMathisLab]

Thanks @yarikoptic! DeepLabCut uses HDF5, and has an export for NWB for saving analyzed data.

• function to convert HDF5 to NWB

[yarikoptic]

Thank you @talmo and @MMathisLab . So both SLEAP and DeepLabCut has export to ndx-pose. I didn't check though if such representation is generalized that e.g. they could potentially read it back (I see there is "load" functionality within https://github.com/DeepLabCut/DLC2NWB)?

@niksirbi 's movement has open issue

• I/O support for the ndx-pose NWB extension neuroinformatics-unit/movement#23
  but otherwise seems indeed to support input/output of various components from/to DeepLabCut (DLC), SLEAP (SLEAP), LightingPose (LP), Anipose (Anipose) .

I wonder if someone could digest for me a summary of how much ndx-pose covers data types across those 4 toolkits. (The situation reminded me of NIfTI in neuroimaging originally envisioned as an "exchange format" among more specialized back then DICOM, AFNI BRIK/HEAD, FreeSurfer, etc)

[talmo]

I think NWB through ndx-pose covers most of it. We wanted to still add support for embedding images with compression, but that's on the backburner.

I believe there might still be a field or two that we're missing for total completeness (and there are some serious performance issues with pynwb), but otherwise it's nearly there.

The movement library is probably the most complete and functional, but it is focused on final tracked inference outputs (i.e., contiguous frames), while ndx-pose supports both the training data (random individual frames, doesn't make sense to organize by video/time) and inference outputs (strictly timeseries per video).

We're still adding proper support for the training data types from NWB into sleap-io, but otherwise it's a good base. We support the roundtrip, but might lose some metadata useful in some cases.

[MMathisLab]

(I see there is "load" functionality within https://github.com/DeepLabCut/DLC2NWB)?

• Yep! :D

But my pref. for a data standard is HDF5 due to interoperability between fields (NWM is very "neuro" focused). We also support human motion data in .json files and COCO formats.

[niksirbi]

Just chiming in to confirm some things that have already been said about movement:

• We currently support loading pose tracks (tracked keypoints over time) from:

  • DeepLabCut (+ LightningPose, which uses the same format)
  • SLEAP (using the sleap-io package under the hood).
  • 3D poses from Anipose

• Support for ndx-pose is underway with this PR. I keep getting distracted by other priorities, but we plan to finalise and merge this within ~1 month.

• Less relevant for this discussion, we also support tracked bounding boxes, currently from only one source format.

The above list is likely to grow, depending on user demand and the availability of suitable contributors. So far, we've been neuro-centric, but that is changing, so we're keen to also support formats that are common in other fields.

As @talmo said, movement solely deals with predicted (inference output) and tracked data, not with the training datasets.

Regarding ndx-pose, I think it indeed covers most neuroscience use-cases, but @MMathisLab is absolutely right about the “neuro” focus. Machine-learning-based motion tracking is used by many fields other than neuroscience, e.g. ethology, conservation biology, biomechanics, animal welfare, veterinary medicine, sports science, etc. Some of these fields do not follow the “subject-session” paradigm that’s prominent in neuroscience, and have probably never heard of NWB or BIDS. Moreover, in some cases it’s not uncommon to track tens or hundreds of individuals at once (e.g. to look at collective behaviour), and it’s hard to imagine how that data would fit into ndx-pose or BIDS.

But perhaps that’s fine; neuro people are likely to prefer ndx-pose, especially if they are planning to eventually analyse the motion tracks in conjunction with neural recordings. BIDS + NWB doesn’t need to (and cannot) cater to everyone. I’m just noting that motion tracking in general (including every field that employs these methodologies) is in need of a standardised format, and ndx-pose probably won't be it.

FYI @roaldarbol's animovement is an R package with similar scope to movement, and it already supports a bunch of formats. The developer, Mikkel, has some nice ideas on formatting motion tracking data as a tidy dataframe that could be saved in a variety of formats (e.g. parquet or csv).

Cross-linking some relevant discussions happening elsewhere:

• Discussion on the Behaviour Forum
• Issue about picking a movement-native data format
• Issue about improving the output format of the freemocap project.

[rly]

I’m just noting that motion tracking in general (including every field that employs these methodologies) is in need of a standardised format, and ndx-pose probably won't be it.

I totally agree. ndx-pose is neuro-focused, but that might be OK for BIDS purposes. It looks like Motion-BIDS was driven by human motion capture data. I believe the required fields in Motion-BIDS are also required in ndx-pose, but many of the recommended fields are not relevant or not captured by the tools, as far as I know.

Indeed, ndx-pose (and NWB) does not support handling multiple subjects in a nice way -- each subject's pose data should be in its own NWB file with its individual subject information, even if the data from multiple subjects were acquired and processed together. But I think BIDS also does not currently support handling multiple subjects in a session nicely, though BIDS 2.0 might change that (ses-X/sub-Y organization instead of sub-Y/ses-X).

neuroconv supports converting outputs from DLC, SLEAP, and LightningPose to ndx-pose. Once movement supports writing ndx-pose NWB files (which I 100% support!), then I think any formats that can be read by movement should also be convertible to ndx-pose NWB files through movement.

If you want to extend Motion-BIDS to support single-subject animal motion data, I think allowing NWB files with ndx-pose data makes sense, and would be better than supporting the unique output formats of N different tools. Note that there is no way to tell whether an NWB file has ndx-pose data just by the file name. You would have to open the file. In addition, ndx-pose NWB files could contain other very large related datasets from other modalities. Are symlinks allowed so that a single NWB file can be used in multiple places in the BIDS structure?

Alternatively, you could not extend BIDS support to allow another file type, and instead someone could write a tool to convert data from ndx-pose, DLC, SLEAP, etc. to the existing Motion-BIDS standard of JSON and TSV files. This could be built into movement or neuroconv. This would avoid complicating the reading of motion data in different formats for BIDS users. However, I think NWB users and tools would prefer storing pose data in NWB with ndx-pose.

[talmo]

relevant

---

I also want to put another plug in to not forget the training data format support. The needs of data used for training these motion capture models are pretty different from the inference outputs.

If our goal is to store enough metadata to make these kinds of data more reusable and reproducible, we can't ignore the fact that they're produced by machine learning systems. These are pieces of software that depend not only on statically accessible code, but also on the data that was used to train them -- which is arguably considerably more important than the specific tool that was used.

Just like you would store all the hyperparameters that a classical piece of software (or data generating hardware) used to produce a dataset, it is essential that we treat the training data that is part and parcel to the operation of a machine learning system just as seriously.

[robertoostenveld]

Just zooming out a bit: I wonder how data used in animal behavioral research is fundamentally different from data when the animal is Homo Sapiens. The consideration is similar to animal MRI versus human MRI.

As before with microscopy, for the micro electrophysiology BEP032 we are recognizing the similarities between micro electrodes in human and other animals. The specific difference that we identified to be accommodated with BEP032 is more between macro (already supported with iEEG) and micro (now being added).

Did anyone already give it a try to represent animal motion capture data in the existing standard specification? I realize that page refers to human motion, but if that were searched-and-replaced by "human and animal", might that not work? Probably not, but I wonder what the hypothetical attempt would reveal...

[MMathisLab]

Looking at the specs, it's extremely comparable with https://github.com/AdaptiveMotorControlLab/DLC2Kinematics and our output H5 files. This package take in the outputs of DeepLabCut and outputs velocity, angles, quaterions, etc :)

And yes -- humans are technically animals! :)

[yarikoptic]

Thanks everyone for sharing information/views/wisdom! It might be great to create a "table" to express what is supported by each of existing standards (in particular relating to BIDS Motion) and tools (which could read/write in what standard).

Meanwhile, just to answer @rly question:

Are symlinks allowed so that a single NWB file can be used in multiple places in the BIDS structure?

They are not since symlinks are not generally available across file systems. FWIW, within BIDS there is a number of locations within metadata (in .json typically) where BIDS URIs are supported to reference other files.

[niksirbi]

Did anyone already give it a try to represent animal motion capture data in the existing standard specification? I realize that page refers to human motion, but if that were searched-and-replaced by "human and animal", might that not work? Probably not, but I wonder what the hypothetical attempt would reveal...

@robertoostenveld I happened to revisit this recently, finally getting around to reading both the Motion-BIDS spec and the associated paper.

I haven’t yet tried to represent animal pose estimation data (optical markerless, like that produced by DeepLabCut, SLEAP, etc.) using this format—but based on my reading of the spec, it should absolutely be possible. We’d simply need to represent the x, y (and optionally z) coordinates of each predicted body part position as separate motion channels. That said, I see two main limitations that could hinder the adoption of this spec in the animal research community:

• Minor: Representing tracking data for a single subject and session currently requires at least four files: _motion.tsv, _motion.json, _channels.tsv, and _channels.json. I understand why this is the case—it’s very much in line with the BIDS philosophy and likely necessary to support the generality of the spec (which spans optical marker-based systems, markerless tracking, VR, IMUs, etc.). However, many animal researchers are used to storing everything in a single file (e.g. one CSV with named columns), and may find this level of "file fragmentation" off-putting. For what it’s worth, I personally like the BIDS approach.

• Major: The spec currently supports only single-subject data, which reflects a broader limitation in BIDS and NWB (where the subject is the primary organising unit). This becomes a real challenge for many animal behaviour researchers—especially those studying social interaction (e.g. animal pairs) or collective behaviour (e.g. groups of tens or hundreds). In those fields, a single video might contain many individuals, tracked using tools like TRex or idtracker.ai. I appreciate this is quite out-of-scope for the core BIDS effort, but it is an important limitation for researchers outside the neuro-centric use cases.

Ways forward

• For the single-subject case, I don’t think the spec itself needs to be amended. It would help a lot just to add an example dataset for animal markerless pose estimation—perhaps re-using a public dataset tracked with DeepLabCut or SLEAP.
• Even with such an example in place, I don’t expect many animal behaviour researchers to immediately adopt the Motion-BIDS format. To encourage uptake, we’d need automated tools to convert data from common formats (like DeepLabCut and SLEAP) into BIDS-Motion. This would definitely fall within the scope of movement. That said—speaking with my maintainer hat on—it may take us some time to get there (considering all other development priorities).