Skip to content

Add frequent thin scrape for actively touring artists#72

Draft
switz wants to merge 1 commit into
masterfrom
feature/thin-scrape-active-artists
Draft

Add frequent thin scrape for actively touring artists#72
switz wants to merge 1 commit into
masterfrom
feature/thin-scrape-active-artists

Conversation

@switz

@switz switz commented Jul 11, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds a recurring Hangfire job (RefreshActiveArtists) that runs every ~2 hours (0, 2, 4, 12, 14, 16, 18, 20, 22 UTC — avoiding the 6-11 AM daily import window)
  • Finds artists with a show in the past 7 days and runs a lightweight "thin scrape" limited to the current year
  • Introduces ImportOptions to thread thin-scrape behavior through the importer pipeline without changing abstract method signatures

Per-importer optimizations

  • Archive.org: Appends year:<current> to the Lucene search query so only current-year recordings are fetched; skips source-deletion logic (since results are a filtered subset)
  • Phish.in: Paginates from the last page backward (newest shows first), skipping pre-cutoff shows and stopping once an entire page has no in-range shows. Eras/tours/songs/venues still sync to avoid KeyNotFoundException on new entities.
  • PhishNet: Filters source list to current-year display_date only, skipping per-show rating scrapes for historical shows

Test plan

  • Build succeeds, all 52 existing tests pass
  • Trigger RefreshActiveArtists via Hangfire dashboard with allowedInDev=true — verify it identifies artists with recent shows
  • Verify thin scrape for Phish processes only current-year shows (check logs for page count and early stop message)
  • Verify archive.org thin scrape URL includes year filter
  • Verify no historical sources are deleted during thin scrape
  • Verify artistsCurrentlySyncing guard prevents overlap between thin scrape and full daily import

Every ~2 hours (avoiding the daily import window), check for artists with
a show in the past 7 days and run a lightweight current-year-only import.

- Archive.org: filter search to current year, skip source deletion
- Phish.in: paginate from newest page backward, stop at year boundary
- PhishNet: only scrape ratings for current-year sources
- Thread ImportOptions through ImporterService to importers
@switz switz force-pushed the feature/thin-scrape-active-artists branch from 3c01d46 to 53ffbc4 Compare July 11, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant