feat(recap_rss): Add command to import a historical PACER RSS archive

[neilrubin]

Fixes

Addresses the historical-RSS portion of #448 (incorporate all possible bulk
PACER data). #448 is an umbrella checklist, so this references rather than
closes it.

Summary

I have a collection of PACER RSS feed captures made over about 9 weeks in 2015-2016 that I would like to donate to CourtListener. This collection predates CourtListener's own RSS feed captures, and based on a comparison with .docket.json files at the Internet Archive, I believe this will add ~3 million non-duplicative PACER docket entries to the collection. The actual RSS captures are in a ~500 MB tarball that I can provide to FLP or anyone else interested on request.

This is one of four PRs that I'm submitting today as a first-time contributor to the repo. While I'm knowledgeable concerning the PACER data, RECAP, and the CL data offerings, and I'm a capable Python coder, I am not intimately familiar with the internals of the repo code. As full disclosure, the code was entirely written using Claude Code, as was the entirety of this .md file, aside from the first three paragraphs of this summary. I have attempted to carefully think through the design, make sure it is correct and consistent with the existing code, and make sure that it passes all CI tests, this is the work of a first-timer.

I'm aware that FLP has some sort of policies and guidance on the use of tools such as Claude Code for development, but I have not been able to review them. See Issue 7383. To the extent this PR is inconsistent with those policies and guidance, I am happy to revise once I'm able to see them.

Adds a single management command, import_rss_archive, that ingests a donated
archive of historical PACER RSS feed captures (487 MB, 37,225 .rss files
across 171 courts, 2015‑11‑22 → 2016‑01‑24). This window predates
CourtListener's own RSS scraping (~2018), so it is gap data, not a re-import of
feeds we already have.

The archive is laid out as one subdirectory per court, named by PACER court
code:

<root>/<court_id>/<court_id>-<epoch_ms>.rss

The command walks each court's files oldest first (by the epoch_ms in the
filename) and, for each file, does exactly what RssFeedData.reprocess_item()
already does for a stored feed — parse with the existing PacerRssFeed and
merge with the existing merge_rss_feed_contents():

rss_feed = PacerRssFeed(map_cl_to_pacer_id(court_id))
rss_feed._parse_text(path.read_bytes().decode())
merge_rss_feed_contents(rss_feed.data, court_id)

So it reuses the live parse-and-merge pipeline (find_docket_object →
update_docket_metadata → add_recap_source → add_docket_entries) with no
changes to it.

Because the directory names are PACER court codes, they're run through the
existing map_pacer_to_cl_id() to get the CourtListener court id. For 168/171
courts these are identical; the three that differ (cofc→uscfc, azb→arb,
neb→nebraskab) are routed to the correct CL court rather than skipped.

What's deliberately not here, to keep the change small and avoid
decisions that affect other code:

• No model or migration change. Records come from PACER RSS, so the
  pipeline's built-in add_recap_source() is accurate; I did not add a new
  source bit or tag. (Happy to add provenance tagging if you'd prefer it —
  Docket.source is out of free bits.)
• No raw-feed retention. The structured docket/entry/document data is
  merged regardless; I left out persisting the raw .rss files as
  RssFeedData/S3 rows since that's an infra choice. Easy to add behind a flag.
• Runs synchronously, like reprocess_item(). A single pass over the
  archive is tractable and avoids cross-worker cache races. Could be switched to
  merge_rss_feed_contents.delay() behind a CeleryThrottle if parallelism is
  needed.

Notes / gotchas:

• No spurious alerts. Calling merge_rss_feed_contents() directly enqueues
  no docket alerts: the alert-sending step (send_alerts_and_webhooks) is a
  separate task in the live Celery chain that this command never invokes, and
  enqueue_docket_alert only sets a short-lived Redis semaphore. Same behavior
  as reprocess_item().
• Re-running is safe. Within a run, identical items are skipped via the
  Redis item-hash cache; across runs (after the 2‑day TTL), numbered entries
  dedup durably on (docket, entry_number). A test covers re-ingesting with the
  cache cleared.
• Only directories that name a court in
  Court.federal_courts.all_pacer_courts() (the same set scrape_rss uses) are
  ingested; unknown directories, empty captures, and unparseable files are
  skipped with a log line.

Verification:

• Parsed the entire archive with the current Juriscraper: 0 parse errors
  across all 37,225 files (33,716 with data, 3,509 empty/header-only). Every
  file is pure ASCII despite an ISO‑8859‑1 declaration, so a plain UTF‑8 decode
  is safe.
• cl.recap_rss.tests.ImportRssArchiveTest covers ingestion, idempotent
  re-ingest (item-hash cache cleared), unknown-court skip, malformed-file skip,
  empty-feed skip, and PACER→CL court-id mapping. Run alongside the existing
  RecapMinuteEntriesTest: 17 tests pass, serial and --parallel; mypy
  and ruff are clean.

Estimated impact (measured against a local snapshot of the CL database and of IA .docket.json files): the
archive holds ~3.4M unique docket entries, of which ~93% (~3.0M) are not yet
in CL (≈2.45M numbered + ≈0.58M unnumbered) — consistent with this being
pre‑2018 gap data. ~99.9% of cases map to dockets CL already has, so the command
mostly adds entries to existing dockets rather than creating new ones.

Deployment

This PR should:

• skip-deploy (skips everything below)
  • skip-web-deploy
  • skip-celery-deploy
  • skip-cronjob-deploy
  • skip-daemon-deploy

This is a management command run manually against a donated archive; it changes
no web, task, cron, or daemon code. Running it is a manual, one-time step:

1. manage.py import_rss_archive --root /path/to/RSS [--courts cand nysd ...]

[CLAassistant]

All committers have signed the CLA.

[mlissner]

Thanks Neil. We did a huge RSS ingestion at some point that may include this data, but @albertisfu will know or have the details (I hope?). Alberto, when you get a sec, can you see if this is duplicative, please?

I'm not sure this is entirely worth us doing, honestly. It's a rather small window of fairly old data, and getting mergers right is hard. It might be better to pass on this if the code looks rough. OTOH, maybe our existing RSS import scripts could be refashioned to work here?

[albertisfu]

Thanks @neilrubin

As Mike mentioned, we imported around 3M RSS feeds from 180 courts, both before and after 2018. Most of that data belongs to bankruptcy courts, but it also includes district and appellate courts.

There's a possibility that we already have some or all of the content you have.

Would it be possible for you to provide a sample of your data so we can quickly check whether those cases were already imported or not?

[neilrubin]

Hi @mlissner. Certainly no hurt feelings if you decide the effort and risk to ingest aren't worth the trouble, in view of the age and size.

@albertisfu, the collection is of RSS feeds from 172 courts (meant to be all Federal courts with RSS feeds at the time), with most of the district and bankruptcy courts, plus four circuit courts and some special courts. It's the raw RSS feed downloads, captured once every four hours, with one capture per file. (~37.4k files). I'll make the tarball available here for a few weeks, if you want to take a look: PACER_RSS_2015-2016.tar.bz2