Skip to content

Add rvdss indicator/data source #1542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 67 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
62b9070
Create rvdss_historic.py
cchuong Sep 13, 2024
073aac9
Create rvdss_update.py
cchuong Sep 13, 2024
01af95f
create utils.py for common functions
cchuong Sep 14, 2024
6a002e0
create constants.py and update utils
cchuong Sep 16, 2024
714455c
Update rvdss_historic.py
cchuong Sep 16, 2024
6ee8bb7
Update rvdss_update.py
cchuong Sep 16, 2024
8814554
fix typo and add missing abbreviation to constants
cchuong Sep 17, 2024
d7905c8
fix typo
cchuong Sep 18, 2024
08f908a
add missing geo
cchuong Sep 18, 2024
07ed998
Update constants.py
cchuong Sep 18, 2024
fd5bf15
Revert "Update constants.py"
cchuong Sep 18, 2024
678b468
Revert "add missing geo"
cchuong Sep 18, 2024
4bfc933
fix geo and virus abbreviation
cchuong Sep 20, 2024
e8957c3
remove "province of" from geo_values
cchuong Sep 20, 2024
7720a24
construct urls automatically
nmdefries Sep 23, 2024
59f79bf
comment constants
nmdefries Sep 23, 2024
e70b0e9
note historic urls don't need to be updated
nmdefries Sep 23, 2024
72d1906
be stricter about importing local fns
nmdefries Sep 23, 2024
bf51bd3
move dashboard file names to constants
nmdefries Sep 23, 2024
ee3cadf
move run-the-whole-pipeline code into main()
nmdefries Sep 23, 2024
180e67f
add code to calculate number of positive tests back in
cchuong Sep 24, 2024
6bd6e24
update abbreviate_geo to remove periods and other spelling
cchuong Sep 24, 2024
a7666b8
fix lab name missing province
cchuong Sep 24, 2024
503165e
comment historic script
nmdefries Sep 24, 2024
256e697
move output file names to constants
nmdefries Sep 24, 2024
cd83087
replace boolean comparisons with pythonic "not"
nmdefries Sep 24, 2024
969295b
actually put csv names in constants
nmdefries Sep 25, 2024
00f3f9a
break more helper functions and add doctsrings
cchuong Oct 2, 2024
ecca542
add more comments
cchuong Oct 4, 2024
31ec961
calculate update dates in a new function
cchuong Oct 10, 2024
0be5f08
combine different spellings of labs
cchuong Oct 13, 2024
5696636
change slash to underscore in constants and move more processing code…
cchuong Oct 13, 2024
30f3df6
rvdss interface and new fn layout so current/historical data can be e…
nmdefries Nov 22, 2024
49e67a9
test outline
nmdefries Nov 22, 2024
b0fa747
add regex as dependency
nmdefries Nov 22, 2024
e6d9053
add unidecode as dependency
nmdefries Nov 22, 2024
c7a4203
import relative to delphi.epidata
nmdefries Nov 22, 2024
8c6b555
switch all rvdss tests from unittest to pytest; basic abbr virus tests
nmdefries Nov 22, 2024
c0742fc
Update test_utils.py
cchuong Jan 24, 2025
b2e5013
Add tests and testdata
cchuong Mar 7, 2025
6353b18
Add utils tests and move test data
cchuong Apr 2, 2025
e503e97
Add testdata of historic reports
cchuong Apr 4, 2025
5662a79
Add additional tests and testdata
cchuong Apr 24, 2025
22c8589
Update sql table definitions and add extra na values to historic data…
cchuong Apr 25, 2025
9d22911
Add extra values that should be read as NA and counts with spaces in …
cchuong Apr 25, 2025
901b2fe
update sql keys
cchuong Apr 25, 2025
1214e70
Update pull_historic.py
cchuong Apr 25, 2025
4e5528b
Update rvdss.sql
cchuong May 8, 2025
908df83
remove unused code and table definition
cchuong May 13, 2025
02de2eb
add extra edge cases, and duplicate checking
cchuong May 15, 2025
dc14c71
remove saving to csv
cchuong May 15, 2025
4d283a6
remove scripts for manual testing
cchuong May 15, 2025
58a1478
add extra duplication checks and check if tables exists
cchuong May 16, 2025
f96d926
add logger
cchuong May 25, 2025
7cf0116
add basic integration tests
cchuong May 30, 2025
38bdd50
Combine multiple tables into one
cchuong Jul 7, 2025
5ba4fd5
stop scraping unused table
cchuong Jul 8, 2025
277c44b
skeleton integration tests
cchuong Jul 18, 2025
4e5a051
Merge pull request #1561 from cmu-delphi/ndefries/rvdss-tests
cchuong Jul 18, 2025
f1fde56
Check for duplicate indexes and use merge to combine tables
cchuong Jul 19, 2025
914862f
test extra edge case
cchuong Jul 19, 2025
ffba810
Clean up old comments
cchuong Jul 19, 2025
6932653
cleanup old code to do with unused tables
cchuong Jul 19, 2025
a482227
update tests for new table structure
cchuong Jul 19, 2025
48a1b5a
Update unit tests and remove unused functions/tests
cchuong Aug 1, 2025
9641f02
Update functions to combine tables
cchuong Aug 1, 2025
d757ed5
Update unit tests to check dtype and column order
cchuong Aug 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions src/acquisition/rvdss/constants.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# The dataset calls the same viruses, provinces, regions (province groups),
# and country by multiple names. Map each of those to a common abbreviation.
VIRUSES = {
"parainfluenza": "hpiv",
"piv": "hpiv",
"para": "hpiv",
"adenovirus": "adv",
"adeno": "adv",
"human metapneumovirus": "hmpv",
"enterovirus/rhinovirus": "evrv",
"rhinovirus": "evrv",
"rhv": "evrv",
"entero/rhino": "evrv",
"rhino":"evrv",
"ev/rv":"evrv",
"coronavirus":"hcov",
"coron":"hcov",
"coro":"hcov",
"respiratory syncytial virus":"rsv",
"influenza":"flu",
"sars-cov-2":"sarscov2",
}

GEOS = {
"newfoundland": "nl",
"newfoundland and labrador": "nl",
"prince edward island":"pe",
"nova scotia":"ns",
"new brunswick":"nb",
"québec":"qc",
"quebec":"qc",
"ontario":"on",
"manitoba" : "mb",
"saskatchewan":"sk",
"alberta": "ab",
"british columbia" :"bc",
"yukon" : "yk",
"northwest territories" : "nt",
"nunavut" : "nu",
"canada":"ca",
"can":"ca" ,
"at":"atlantic",
"atl":"atlantic",
"pr" :"prairies" ,
"terr" :"territories",
}

# Regions are groups of provinces that are geographically close together. Some single provinces are reported as their own region (e.g. Québec, Ontario).
REGIONS = ['atlantic','atl','at','province of québec','québec','qc','province of ontario','ontario','on',
'prairies', 'pr', "british columbia",'bc',"territories",'terr',]
NATION = ["canada","can",'ca',]

# Construct dashboard and data report URLS.
DASHBOARD_BASE_URL_2023 = "https://health-infobase.canada.ca/src/data/respiratory-virus-detections/archive/{date}/"
DASHBOARD_BASE_URLS_2023 = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Please describe a bit more about what these URLs are and if we will need to update them (add new dates in) ever.

issue: also, are these actually for 2023? All the dates are in 2024

DASHBOARD_BASE_URL_2023.format(date = date) for date in
(
"2024-06-20",
"2024-06-27",
"2024-07-04",
"2024-07-11",
"2024-07-18",
"2024-08-01",
"2024-08-08",
"2024-08-15",
"2024-08-22",
"2024-08-29",
"2024-09-05"
)
)

SEASON_BASE_URL = "https://www.canada.ca"
ALTERNATIVE_SEASON_BASE_URL = "www.phac-aspc.gc.ca/bid-bmi/dsd-dsm/rvdi-divr/"
HISTORIC_SEASON_REPORTS_URL + "/en/public-health/services/surveillance/respiratory-virus-detections-canada/{year_range}.html"

# Each URL created here points to a list of all data reports made during that
# season, e.g.
# https://www.canada.ca/en/public-health/services/surveillance/respiratory-virus-detections-canada/2014-2015.html.
# The Public Health Agency of Canada site switched in 2024 to reporting
# disease data in a dashboard with a static URL. Therefore, this collection
# of URLs does _NOT_ need to be updated. It is used for fetching historical
# data (for dates on or before June 8, 2024) only.
HISTORIC_SEASON_URL = (HISTORIC_SEASON_REPORTS_URL.format(year_range = year_range) for year_range in
(
"2013-2014",
"2014-2015",
"2015-2016",
"2016-2017",
"2017-2018",
"2018-2019",
"2019-2020",
"2020-2021",
"2021-2022",
"2022-2023",
"2023-2024"
)
)

LAST_WEEK_OF_YEAR = 35
Loading
Loading