Set default older_than to tomorrow

[krivard]

Summary:

We currently wait until a day is over before pulling covid_hosp input files posted that day. Unfortunately, the new healthdata.gov posting schedule for covid_hosp data is Fridays and Mondays at ~12:10pm. Forecasting would like to use the fresh Monday data for the Monday forecasts they submit at 3pm (Slack convo link). This means we should change the default pipeline behavior to permit same-day healthdata.gov files to be included in each update.

This PR updates the default selection criteria for which healthdata.gov files to import so that it includes data posted today.

WARNING: if healthdata.gov posts a file after 12:30 on a Monday or a Friday, that file will never be picked up by the regular pipeline and must be handled with a manual invocation of the pipeline that selects for the day the late file was posted. 7/17 edit: a fix for this edge case is in progress, thanks george!

This is a semi-hotfix. The covid_hosp acquisition materials are in the midst of a massive overhaul, but we'd like to implement this behavior change before the new code will be ready, so this PR modifies the old code.

Prerequisites:

• Unless it is a documentation hotfix it should be merged against the dev branch
• Branch is up-to-date with the branch to be merged with, i.e. dev
• Build is successful
• Code is cleaned up and formatted

[melange396]

this is untested in any way, and will probably need unit/integration test changes too.

with the commit i just pushed, it "should" now look back to the same day as the last issue to see if there were any other new data revisions issued, instead of just looking back to the next day after the last issue. any revisions from that last day that were already collected are skipped.

the new usage of the with database.connect() could maybe be widened in its scope (ie, pulled outisde of some number of the nested blocks), but i didnt want to keep the connection open while datasets are fetched and merged (which could potentially take a long time).

i think this also means, in normal usage, we should never again see the "no new issues; nothing to do" log message, but will now see the new log message "already collected revision:" on every run.

[krivard]

oh noooooo this needs an ON DUPLICATE KEY UPDATE since we're no longer guaranteed to only insert once for each reference date + issue pair

[krivard]

Okay @melange396 I've pushed up what I have and left a TODO where the ON DUPLICATE KEY should go

[melange396]

i initially had a different change for the ON DUPLICATE KEY stuff where i rewrote more of the surrounding code for clarity and efficiency, but decided against that because of how it will complicate the integration with #1203 down the road.

[dshemetov]

EDIT: Updated after fixing.

Simplified the metadata files so they're easier to read:

# metadata.csv
Update Date,Days Since Update,User,Rows,Row Change,Columns,Column Change,Metadata Published,Metadata Updates,Column Level Metadata,Column Level Metadata Updates,Archive Link
03/13/2021 00:00:00 AM,0,0,0,0,0,0,0,0,0,0,https://test0.csv
03/15/2021 00:00:00 AM,0,0,0,0,0,0,0,0,0,0,https://test1.csv

# metadata2.csv
Update Date,Days Since Update,User,Rows,Row Change,Columns,Column Change,Metadata Published,Metadata Updates,Column Level Metadata,Column Level Metadata Updates,Archive Link
03/13/2021 00:00:00 AM,0,0,0,0,0,0,0,0,0,0,https://test0.csv
03/15/2021 00:00:00 AM,0,0,0,0,0,0,0,0,0,0,https://test1.csv
03/15/2021 00:00:01 AM,0,0,0,0,0,0,0,0,0,0,https://test2.csv
03/15/2021 00:00:02 AM,0,0,0,0,0,0,0,0,0,0,https://test3.csv
03/16/2021 00:00:00 AM,0,0,0,0,0,0,0,0,0,0,https://test4.csv
03/16/2021 00:00:01 AM,0,0,0,0,0,0,0,0,0,0,https://test5.csv

Some commentary on the tests test_scenarios.py:

• on line 73, we initialize the db with data from the default dataset dataset.csv and using the two issues from metadata.csv
• on line 106, we test a second acquisition which is a noop because we use metadata.csv again
• on line 123, we test a third acquisition with metadata2.csv; it shares two issues with metadata.csv, but contains 4 new issues; each of these 4 updates contains an update to a row for WY data
  • the first two (duplicate) issues are ignored correctly
  • 03/15/2021 00:00:01 and 03/15/2021 00:00:02 check that we correctly handle two updates for the same row for the same issue for an already existing row
  • 03/16/2021 00:00:00 and 03/16/2021 00:00:01 check that we correctly handle two updates for the same row for the same issue for a brand new row

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

[melange396]

Looks like we arent even testing ON DUPLICATE KEY UPDATE (at least not in the place you're pointing out)... For WY, dataset0.csv has value 5 and date 2020/12/09 (the date is under column "reporting_cutoff_start" which eventually gets renamed/mapped to "date" in the db), dataset1.csv has value 8 and date 2020/12/10... So even with matching "issue"s, theres no key collision because the "date"s are different, and the value isn't overwritten. Line 125 is correct in failing, the value should be 5.

[dshemetov]

Good call @melange396, I fixed the test thanks to your pointer. I also simplified some things specific to this test (see updated comment above).

[melange396]

Nice, thanks @dshemetov ! I especially appreciate you getting rid of those extra (and confusing) csv files.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

[melange396]

LGTM, i think this is ready for merge and release!