Add ability to add features to existing track without retracking

[freemansw1]

This new function (and its various support functions) is designed to allow the user to append a newly detected feature DataFrame to an existing set of tracks. This (should) work with all existing tobac tracking options, except memory due to an upstream issue I've documented here: soft-matter/trackpy#776 .

I have added docs, but haven't yet added a notebook. I'd like to do this before this function is merged, but wanted to start the discussion now.

• Have you followed our guidelines in CONTRIBUTING.md?
• Have you self-reviewed your code and corrected any misspellings?
• Have you written documentation that is easy to understand?
• Have you written descriptive commit messages?
• Have you added NumPy docstrings for newly added functions?
• Have you formatted your code using black?
• If you have introduced a new functionality, have you added adequate unit tests?
• Have all tests passed in your local clone?
• If you have introduced a new functionality, have you added an example notebook?
• Have you kept your pull request small and limited so that it is easy to review?
• Have the newest changes from this branch been merged?

[freemansw1]

I should have noted that I put in a good bit of effort to refactor the primary tracking function to make it less of a monolith. I'm not sure I succeeded there, but it is a bit cleaner now.

[github-actions]

Linting results by Pylint:

Your code has been rated at 8.36/10 (previous run: 8.37/10, -0.01)
_{The linting score is an indicator that reflects how well your code version follows Pylint’s coding standards and quality metrics with respect to the RC_v1.6.x branch.
A decrease usually indicates your new code does not fully meet style guidelines or has potential errors.}

[w-k-jones]

Nice work! I’ll try and start reviewing this next week. Does this include variable dt capability as well?

[freemansw1]

Nice work! I’ll try and start reviewing this next week. Does this include variable dt capability as well?

Inherently, yes it does. It would be good to add a wrapper that automagically does this, though. Would be good to get as part of 1.6 (or perhaps a 1.5.5). A nice feature for folks.

[freemansw1]

okay I'm looking into the codecov failure. It seems to be a real failure, but only when running coverage, which is just frankly bizarre.

[codecov]

Codecov Report

❌ Patch coverage is 74.83871% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.78%. Comparing base (8639ece) to head (435fb40).
⚠️ Report is 2 commits behind head on RC_v1.6.x.

Files with missing lines	Patch %	Lines
tobac/tracking.py	70.98%	74 Missing ⚠️
tobac/testing.py	92.72%	4 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           RC_v1.6.x     #461      +/-   ##
=============================================
+ Coverage      64.86%   65.78%   +0.92%     
=============================================
  Files             27       27              
  Lines           4047     4270     +223     
=============================================
+ Hits            2625     2809     +184     
- Misses          1422     1461      +39     

Flag	Coverage Δ
unittests	65.78% <74.83%> (+0.92%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[freemansw1]

okay I'm looking into the codecov failure. It seems to be a real failure, but only when running coverage, which is just frankly bizarre.

This seems to be a problem with trackpy<0.6.0, so I have added both a check for newer trackpy versions and have set (our first!) package version requirement of trackpy>=0.6.1.

[freemansw1]

something is wrong with the linting CI; GitHub isn't pulling the newest version.

[w-k-jones]

Ran into this bug while testing:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[52], line 5
      2 with xr.open_dataset(fs.open(abi_files[2]), engine="h5netcdf").CMI.expand_dims("t").rename(t="time") as cmi:
      3     new_features = tobac.feature_detection_multithreshold(cmi, dxy, **parameters_features)
----> 5 tobac.tracking.append_tracks_trackpy(
      6     tracks,
      7     new_features,
      8     dt,
      9     dxy,
     10     **parameters_linking
     11 )

File ~/python/tobac/tobac/tracking.py:835, in append_tracks_trackpy(tracks_orig, new_features, dt, dxy, dz, v_max, d_max, d_min, subnetwork_size, memory, stubs, time_cell_min, order, extrapolate, method_linking, adaptive_step, adaptive_stop, cell_number_start, cell_number_unassigned, vertical_coord, min_h1, max_h1, min_h2, max_h2, PBC_flag)
    826 particle_matches = (
    827     trajectories_unfiltered[["cell", "particle", "hdim_1"]]
    828     .drop_duplicates(["cell", "particle"])
   (...)
    831     .max()
    832 )
    834 if cell_matches + particle_matches != 2:
--> 835     raise ValueError(
    836         "Error in appending tracks. Multiple pairs of cell:particle found. Please report this bug."
    837         " Number of cell matches: {0}, Number of particle matches: {1}".format(
    838             cell_matches, particle_matches
    839         )
    840     )
    841 # dictionary of particle:cell pairs
    842 particle_num_to_cell_num = {b: a for a, b in cell_particle_pairs}

ValueError: Error in appending tracks. Multiple pairs of cell:particle found. Please report this bug. Number of cell matches: 2, Number of particle matches: 2

I'll have a look deeper into what is going on in the morning

[w-k-jones]

Ok, so the issue I'm encountering is that on line 820, the result of

trajectories_unfiltered[["cell", "particle", "hdim_1"]].drop_duplicates(["cell", "particle"]).groupby("cell")["particle"].count()

is 2 for every row. I'll have to dig a bit deeper into why this is occurring. Do you have an example of applying the append_tracks method to real data so I can see if I'm using it correctly?

[freemansw1]

Moving this back to draft as there are a number of open questions and things to work on here.

[freemansw1]

Moved back to ready to review. Looks like I got all the bugs. Seems that the primary bug was that I was giving trackpy the current (time t) velocities but displaced at the previous (time t-1) time.

[freemansw1]

Outstanding Issues:

• Need to deal appropriately with time_cell_min: currently, it could produce different output if time_cell_min>2* dt
• Need to add documentation to the user guide
• Need an example notebook
• Need to add 3D tests