Index reharvest XML files (S3 key ↔ DOI ↔ publication date) to speed up reharvesting

[PascalEgn]

Description

For some reharvest DAGs (e.g., Elsevier), we currently need to read/parse large numbers of XML files to discover metadata like DOI and publication date. This is slow and expensive, and makes it difficult to quickly decide which records to reharvest.

Introduce an indexing mechanism so we can efficiently answer:

• which S3 key corresponds to a given DOI
• what publication date (and/or other relevant dates) the record has
• which records fall in a date range

Proposed approach (to be decided)

Consider one of the following (or a hybrid):

1. Index file(s) in the publisher’s S3 bucket

   • e.g. daily/weekly manifests (JSON/CSV/Parquet) mapping {doi, published_date, s3_key, checksum, updated_at}
   • pros: simple, cheap, easy to backfill
   • cons: queryability depends on how we read it (Athena/Glue/etc.)

2. Central searchable index (e.g., OpenSearch)

   • store mapping {doi, published_date, s3_key, publisher, ingestion_date, ...}
   • pros: fast ad-hoc queries, supports UI/tools, good for operational usage
   • cons: needs infra + update strategy + retention policy

3. Hybrid

   • keep authoritative manifests in S3 + load into OpenSearch for fast querying.

Work involved

• Define the minimal schema for the index (DOI, S3 key, publication date; optionally publisher, file hash, record type, etc.).
• Decide indexing storage strategy (S3 manifests vs OpenSearch vs hybrid).
• Implement a job/process to build the index:
  • initial backfill for existing XMLs
  • incremental updates for new XMLs
• Update reharvest tooling/DAGs to use the index when selecting candidate files to reprocess.

Acceptance criteria

• Given a DOI, we can retrieve the corresponding S3 key(s) without parsing XMLs.
• Given a date range, we can list candidate records/S3 keys without parsing all XMLs.
• Reharvest selection runs significantly faster and uses less compute than the current “scan + parse everything” approach.
• Indexing process supports incremental updates and does not require full rebuild each time (unless explicitly triggered).