feat(corpus_importer): ingest USPTO PTLITIG patent litigation data

[neilrubin]

Hi — I'm a first-time contributor to CourtListener. I've tried to take on the task of addressing Issue #664, importing PACER data from patent cases collected by the USPTO (the PTLITIG dataset), last updated in 2020. For better or worse, this task touches on some other components of the CL system, and I've broken it into 3 PRs as a result.

First, it seems like the PTLITIG dataset should be treated as a new Docket source value, but the current 8-bit field for such values is full. The first PR switches to a nine-bit source field. Others with a better sense of the big picture might prefer a different solution.

Second, the PTLITIG data identifies the patent(s) at issue in each of the cases. Patent numbers are a very important piece of data in analyzing and making use of patent litigation dockets, and there are other similar well-known identifiers (trademark or copyright registrations come to mind, but there are certainly others) that it would make sense for the CourtListener system to handle and to allow searches on. The second PR adds a database table for such identifiers, currently scoped solely to handle the patent values from PTLITIG. While out of scope for this PR, I would suggest scraping patent numbers from complaints and/or AO120 forms of patent cases to populate this table for more recent cases would be a good follow-on project.

The third PR is the actual ingestion process. The data files to be ingested are available at USPTO. As designed, this seeks to merge all PTLITIG docket entries into the current database, without adding duplicates or disturbing existing entries. Since it is PACER data (some from RECAP and some from the USPTO's own crawl), it includes both docket entries with entry numbers and those without entry numbers (e.g., minute entries). To avoid duplication or difficult merge logic on the minute entries, I chose to not import minute entries at all for items without entry numbers, if there were any existing docket entries for the case in the CL database. The thinking is that cases that currently have docket entries in CL are likely to have dockets at least as complete as those in a 2020 crawl and that minute entries for old cases are of relatively low value anyway, such that they are not worth dealing with the merge logic.

This is one of four PRs that I'm submitting today as a first-time contributor to the repo. While I'm knowledgeable concerning the PACER data, RECAP, and the CL data offerings, and I'm a capable Python coder, I am not intimately familiar with the internals of the repo code. As full disclosure, the code was entirely written using Claude Code, as was the entirety of this .md file, aside from the first six paragraphs of this text. While I have attempted to carefully think through the design, make sure it is correct and consistent with the existing code, and make sure that it passes all CI tests, this is the work of a first-timer.

I'm aware that FLP has some sort of policies and guidance on the use of tools such as Claude Code for development, but I have not been able to review them. See Issue #7383. To the extent this PR is inconsistent with those policies and guidance, I am happy to revise once I'm able to see them.

Fixes

Fixes: #664
Refs: #448

PR 3 of 3 for the PTLITIG import; builds on PR 1/3, #7388 (the
USPTO_PTLITIG source bit) and PR 2/3, #7389 (the DocketIdentifier model).
Please merge in order: #7388, then #7389, then this PR.

Note on the diff: this branch is stacked on #7388 and #7389, which aren't
in main yet, so GitHub currently shows their changes here as well. The
change that belongs to this PR is its final commit — the adapter, the
import_uspto_ptlitig command, and its tests. The diff will narrow to just
that automatically as the earlier PRs merge and this branch is rebased onto
main.

Summary

Imports the USPTO Patent Litigation Docket Reports (PTLITIG) — ~97k U.S.
district-court patent cases (1963–2021) the USPTO compiled from PACER docket
reports.

Because PTLITIG is built from the same PACER docket reports RECAP captures, each
case is a flattened sibling of Juriscraper's DocketReport.data. So rather than
write new normalization, this reshapes each case into that dict and feeds it
through the existing cl.recap.mergers primitives (find_docket_object,
update_docket_metadata, add_parties_and_attorneys) — the way the SCOTUS and
Texas importers do. cl/corpus_importer/ptlitig.py holds the adapter and
merge_ptlitig_docket(); the import_uspto_ptlitig management command reads the
PTLITIG CSV download and merges each case in two passes (the small per-case files
in memory, the 2.7 GB documents.csv streamed).

Safety choices, so PTLITIG's older snapshot never overwrites richer RECAP
data:

• Parties/attorneys are added only to dockets that have none (the merge
  primitive deletes and recreates roles).
• Asserted patents become DocketIdentifier rows, always added (idempotent
  via the unique constraint).
• Docket entries are a safe gap-fill: numbered entries are matched on
  (docket, entry_number) so only the missing ones are created and existing
  RECAP entries are left untouched (kept even when undated); unnumbered entries,
  which have no stable key, are added only to dockets that start out empty.
• A single case that fails on a data problem (an unknown court, a constraint
  violation) is logged and skipped; other errors surface.

Tests use FactoryBoy (no fixtures) and cover the source bit, patent typing,
party building, the entry gap-fill, idempotency, and an end-to-end run of the
command over sample CSVs.

Operational note for whoever runs it: this is a slow, one-time job. Per-case
writes dominate and the full documents.csv is streamed, so a complete ~97k-case
run takes hours, and saving the dockets/entries fires the usual ES indexing. It's
meant to be run once, off-peak, by a maintainer — not on a schedule.

Deployment

This PR should:

• skip-deploy (skips everything below)
  • skip-web-deploy
  • skip-celery-deploy
  • skip-cronjob-deploy
  • skip-daemon-deploy

Adds a management command and supporting code; nothing is deployed or scheduled —
the command is run manually against the PTLITIG CSV download.

[mlissner]

Thanks Neil! This is a nice one to get done. A couple quick thoughts from me, and then I think somebody from our team will slot this for review:

1. Do you plan to make patent numbers searchable in the search engine once the data is ingested? If so, we could have search alerts for patent numbers, which I suspect would please a LOT of people.

2. To do that, we'd need to also parse patent numbers from documents as we get them, which is probably something eyecite can do (or maybe just a basic regex).

3. And if we do that, we should backfill the existing ones, I suppose, too!

This is cool, thanks for picking it up!

[neilrubin]

Hi @mlissner, adding that sort of search and alert functionality is an obvious follow-on but seemed like something your team would have strong opinions about the right approach on. I thought about parsing documents. It seems like a pretty simple problem if you actually have full dockets with all PDFs, but it gets tricky if you want to try to extract the information when you only have partial dockets.

Regex logic that matches on (\d+,\d{3},\d{3}|(RE|D|PP)\d+,\d+) and that also requires a match on [\'‘’]\d{3}\ [Pp]atent in the document, with the same three digits as the last three of the first regex should be quite robust, assuming reliable text extraction and good reassembly of text across line breaks (or appropriate handling of incorrect reassembly in the regex), with the caveat that you won't be able to reliably distinguish between patents that are asserted in the case (what people are probably interested in) and those that are merely mentioned, for example as related patents. (The '123 Patent requirement will actually prevent a lot of false alarms from prior art patent mentions.) Not sure whether requiring a patent-related NOS would improve reliability or reduce it. For cases where you have a copy of a complaint or amended complaint, you might require that the patent be detected there, as asserted patents will always appear in a complaint, and you aren't likely to pick up many prior art patents there.

The gold standard for what patents are actually asserted is the Form AO 120 that is filed with the complaint. The layout of the form has been the same forever, and most filed in PACER should be vector PDFs with the patent numbers typed in, so extracting the patent numbers from the form should be robust. The problem is that people probably don't download these forms often on PACER (to be picked up on the RECAP extension), and recap.email is going miss most of them, because they are filed before most counsel has made an appearance. You folks would know better than me how often you get AO 120 PDFs for patent NOS cases (or how often you get complaints). It would take some extra work, but AO 120 forms are also available for free from the USPTO ODP API (I have a comprehensive collection). The timing of when the USPTO gets them is a little random (whenever court clerks get around to sending them to the PTO), and the copies in ODP are raster scans, which makes extraction a little harder.

[mlissner]

Nice. I think we'd keep this really simple and just use regex to search for patent numbers anywhere, and then make those searchable. If we do it this way, folks can decide for themselves if they want to filter to only certain forms or NOS codes, and we can keep our search engine universal (ie, we can say "if it has a patent number, we find it" instead of "if it has a patent number, and a nos code of XYZ, and, and, and...").

Definitely some details to work out (like, how do you normalize '123 Patent to the correct value?), but this would be a great enhancement.

[neilrubin]

Agreed on simple and on using regexes. My take on the "'123 Patent" issue is that the use of this abbreviated reference format in patent documents is a feature, rather than a bug. A regex on 7- and 8-digit numbers with commas is going to have false positives, and there are a vast number of ways the full patent number will appear. ("U.S. Patent No. 12,345,678" would be canonical, but there are a bunch of other ways.) In contrast, "'123 Patent" is nearly universal as the abbreviated form, and is also going to have virtually zero false positives. (It's like seeing "Id. at NN" in legal writing is virtually 100% going to be a citation to something.) And there are virtually zero patent litigation documents that don't use the abbreviated form somewhere. There are also very, very few documents that use the abbreviated form without somewhere mentioning the full form.

So, the strategy I'd propose is: (1) a very liberal regex on the full patent form (matching any string of digits with commas, plus the RE, D, and PP prefixes) and (2) requiring the presence of a matching short form as a false positive rejecter. If you want to deal with the very rare case where the abbreviated form is not used, you could possibly add a third regex that accepts the canonical form ("U.S. Patent No. 12,345,678").

The only complication that I can see is that the occasional case will have two patents that happen to collide on the last three digits, in which case the most common solution is to expand the abbreviated form to the last four digits. Rare, but it's probably worth handling in the regexes.

The nice thing is that you have a large ground-truth set in the form of the PTLITIG set, so it would be pretty easy to test the regexes at scale.

In case it's useful, here the USPTO paper describing their methodology: USPTO Economic Working Paper No. 2019-05. Looks like they used NOS codes to identify the cases, and they manually coded the patent numbers (for 97k cases—not my idea of a good time).

[mlissner]

So, the strategy I'd propose is: (1) a very liberal regex on the full patent form (matching any string of digits with commas, plus the RE, D, and PP prefixes) and (2) requiring the presence of a matching short form as a false positive rejecter. If you want to deal with the very rare case where the abbreviated form is not used, you could possibly add a third regex that accepts the canonical form ("U.S. Patent No. 12,345,678").

Yeah, that sounds exactly right to me (including doing the third regex).

Thanks Neil!