Right now, every time the scraper is run, the entire database is wiped, and recreated from scratch.
In the unlikely chance that the scraper misses a user for some reason, that user is completely deleted from the database.
Scraper should be more intelligent about this.
- If a user is added, add the user.
- If a user is deleted—and not simply missing because of an HTTP or parsing error—remove them.
- Same with individual user data points.
Right now, every time the scraper is run, the entire database is wiped, and recreated from scratch.
In the unlikely chance that the scraper misses a user for some reason, that user is completely deleted from the database.
Scraper should be more intelligent about this.