fix: make dask_property picklable - #629
Conversation
d160873 to
9578bb4
Compare
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #629 +/- ##
==========================================
- Coverage 90.05% 90.03% -0.03%
==========================================
Files 24 24
Lines 3610 3601 -9
==========================================
- Hits 3251 3242 -9
Misses 359 359 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
property keeps fget/fset/fdel in C slots and has no reduction of its own, so a behavior class defined in __main__ or a notebook could not be sent to a distributed worker. _DaskMethod already pickled fine; it just gets a test. Assisted-by: ClaudeCode:claude-opus-4.8
9578bb4 to
6c962dd
Compare
|
OK. Looks fine. Wonder how you came onto this? Wonder whether the classes that have the property should be changed to have their own reduce() - but there's no downside to this either. |
Claude found it while working on scikit-hep/coffea#1603. In coffea to make dask-awkward optional dependency we just vendored dask method and dask property. So I thought I'd just upstream the patch here too. |
🤖 AI text below 🤖
A class using
@dask_propertycannot be pickled by value:propertykeepsfget/fset/fdelin C-level slots and provides no reduction of its own, so CPython refuses outright —pickle.dumps(property(f))raises the same error._DaskPropertyinherits that refusal even though it does have an instance__dict__.This matters because cloudpickle only pickles a class by reference when it is reachable as
getattr(module, cls.__qualname__). Anything else — a behavior mixin written in__main__or in a notebook cell, or one whose generated*Record/*Arrayclasses have been shadowed in their module — gets pickled by value, which walks the class dict and hits the property. The result is that such behaviors cannot be sent to a distributed worker at all._DaskPropertynow defines__reduce__, round-trippingfget/fset/fdel/__doc__/_dask_get._DaskMethodneeds no change: it is a plain Python class whose whole state lives in the instance__dict__, so the default reduction already handles it and cloudpickle serializes the contained function and closure by value. It gets a test to keep it that way.Found while debugging the coffea side of this, where the same copy of
_DaskPropertybrokeFCCSchemaunder a distributed client: scikit-hep/coffea#1603.