Florida docket merger and abstraction prototype#7421
Conversation
Adds tests for ``merge_docket`` and ``merge_oci`` covering supreme-court and appellate flows, OCI create/update, unknown court IDs, and the ``date_last_filing`` derivation from entries. Repairs the factories alongside: imports came from ``faker`` rather than ``factory`` so the previous declarations were inert, and originating cases needed a ``case_number`` for OCI tests. The factories now build Pydantic objects via ``model_construct`` so field-name kwargs work despite the upstream ``validation_alias`` paths. Also passes ``docket_source=Docket.SCRAPER`` to ``find_docket_object`` and ``find_and_disaggregate_docket_object``; the default ``Docket.RECAP`` made ``Docket.save()`` reject newly-created Florida dockets for missing ``pacer_case_id``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the private openai is_iterable import, stop clobbering DB data with empty scrapes (new default OverwriteIfPresent strategy), fail gracefully on duplicate natural keys, fix child-collection edge cases, and make atomic cover the whole related tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Type __model__ as the concrete type[DBModel] and use _default_manager so the model's manager and exceptions resolve, match the relationship dataclasses directly so parent_key is typed, and annotate the dynamic InputField traversal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement merge_docket as a FloridaDocketMerger and wire the OCI merge as a one-to-one RelatedMerger. Teach the abstraction's one-to-one path to skip when its transform yields None so OCI can be gated to supreme-court dockets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mypy 1.15 doesn't support PEP 696 type-param defaults referencing a prior param. Drop the unused S param from InputField, collapse Parameter to a single typevar, and tighten _merge_child's iteration/kwargs typing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![semgrep-app[bot]](https://avatars.githubusercontent.com/in/60555?s=60&v=4){kind=small}
Porting changes from party merger branch
Porting changes from party merger
Drop the phantom MergeType and merge TransformType/DefaultType into a single OutputType on MergerSpecification (5 params -> 3). RelatedMerger keeps the child-data type as ChildType (6 -> 5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Loosen transform/strategy typing on the factories so bare lambdas and explicit strategies stop collapsing inference to Never. TransformType is still inferred from the transform return, keeping the default check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude was a bit too aggressive
Last of the moves
Brennan-Chesley-FLP
left a comment
Brennan-Chesley-FLP
left a comment
There was a problem hiding this comment.
Looked over the changes since my last approval and I still like it. Nothing that looks like a blocker to me.
albertisfu
left a comment
albertisfu
left a comment
There was a problem hiding this comment.
Thanks, @MorganBennetDev! This looks almost ready.
I just left a couple of additional comments. Also, could you please add all the new files introduced in this PR to the mypy checks in lint.yml?
It might be better to add the entire state directory:
cl/corpus_importer/state/
| FLORIDA_SUPREME_DN_RE = re.compile(r"SC\d{4}-\d{4,5}") | ||
|
|
||
|
|
||
| def make_docket_number_core( |
There was a problem hiding this comment.
Can we apply the same transformation we’re using for the Texas docket number core by stripping all non-alphanumeric characters and lowercasing it for consistency?
Also, adding tests for the different FL numbers would be helpful to prevent variations in the number format this method generates.
| ) | ||
|
|
||
| @override | ||
| def query(self) -> QuerySet[Docket]: |
There was a problem hiding this comment.
I was testing this and found that two cases with the same UUID but different docket numbers are not matching. We should prioritize the UUID (pacer_case_id) as the primary source of truth when matching existing cases, even if the docket number has changed (for example, due to a correction from the court).
We could use logic similar to find_docket_object, where pacer_case_id is always prioritized, either in combination with the docket number or on its own.
Why wasn't it possible to continue using find_docket_object here? It would be great to keep the matching logic in a single place if possible.
There was a problem hiding this comment.
Why wasn't it possible to continue using
find_docket_objecthere? It would be great to keep the matching logic in a single place if possible.
When I wrote this, I didn't want to update find_docket_object to be able to return a query set, but I should probably just do that. Maybe something like:
- Create
find_docket_object_querywhich will house thefor kwargs in lookupsloop fromfind_docket_object, returning a queryset, and - Update
find_docket_objectto only do the DN core validation and checks, leaving the actual lookup logic infind_docket_object_query.
Then Florida and future Merger subclasses can use find_docket_object_query to keep the same logic as find_docket_object (minus the DN core conversion, which they should handle themselves).
What do you think?
I think they're already covered by |
|
@albertisfu Updated based on your feedback and implemented my proposed solution to your comment about |
3 participants
Fixes
Closes #7361
Progresses #6878
Summary
Adds prototype merger abstraction in
corpus_importer/state/merger.pyAdds
DocketandOriginatingCourtInformationmergers for Florida incorpus_importer/state/floridausing prototype merger abstractioncorpus_importer/state/florida/utils.pyand add it in the appropriate places to ensure correct DN core generationMoves
MergeResultfromcl/corpus_importer/tasks.pytocl/corpus_importer/state/utils.pyI put the Florida merger in its own directory to keep things organized since.
tasks.pyis already quite large and adding more non-task merger methods to it is only going to make it harder to work in. I think going forward, we should limittasks.pyto only tasks and put related methods in other files/packages when possible.Deployment
This PR should:
skip-deploy(skips everything below)skip-web-deployskip-celery-deployskip-cronjob-deployskip-daemon-deploy