Proposal

The wikidata table we use to cache wikidata objects is a useful dataset in its own right. Currently it'll contain a mostly complete image of authors on the site. This is useful for doing bot/sync work, as otherwise you have to either perform a lot of wikidata queries, or read the veeery large wikidata dumps.

This is the psql command:

psql -d openlibrary -c "\COPY wikidata TO STDOUT WITH (FORMAT csv, DELIMITER E'\t')"

Justification

Breakdown

Requirements Checklist

• [ ]

Related files

• openlibrary/scripts/oldump.sh

  Lines 128 to 135 in f5e32ce

  	log "=== Step 3 ==="
  	if [[ ! -f $(compgen -G "ol_dump_ratings_$yyyymm*.txt.gz") ]]
  	then
  	log "generating ratings table: ol_dump_ratings_$yyyymmdd.txt.gz"
  	time psql $PSQL_PARAMS --set=upto="$yyyymmdd" -f $SCRIPTS/dump-ratings.sql | gzip -c > ol_dump_ratings_$yyyymmdd.txt.gz
  	else
  	log "Skipping: $(compgen -G "ol_dump_ratings_$yyyymm*.txt.gz")"
  	fi

• https://github.com/internetarchive/openlibrary/blob/f532982e7737020f6f6e7ba3a120f2d4cc4238ac/scripts/dump-ratings.sql

Stakeholders

@RayBB @pidgezero-one

Instructions for Contributors

Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.