Merge pull request #856 from cmu-delphi/release/delphi-epidata-0.3.11

Release Delphi Epidata 0.3.11

Author: krivard

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.10
+current_version = 0.3.11
 commit = False
 tag = False
 

docs/api/covidcast-signals/dsew-cpr.md

@@ -17,14 +17,20 @@ grand_parent: COVIDcast Epidata API
 
 The Community Profile Report (CPR) is published by the Data Strategy and Execution Workgroup (DSEW) of the White House COVID-19 Team. For more information, see the [official description and data dictionary at healthdata.gov](https://healthdata.gov/Health/COVID-19-Community-Profile-Report/gqxm-d9w9) for "COVID-19 Community Profile Report".
 
-This data source provides various COVID-19 related metrics, of which we report hospital admissions. Other sources of hospital admissions data in COVIDcast include [HHS](hhs.md) and [medical insurance claims](hospital-admissions.md). The CPR differs from these sources in that it is part of the public health surveillance stream (like HHS, unlike claims) but is available at a daily-county level (like claims, unlike HHS). CPR hospital admissions figures at the state level and above are meant to match those from HHS, but are known to differ. See the Limitations section for details.
+This data source provides various COVID-19 related metrics, of which we report hospital admissions and vaccinations. 
 
-County, MSA, state, and HHS-level values are pulled directly from CPR; nation-level values are aggregated up from the state level.
+For hospital admissions, other sources of data in COVIDcast include [HHS](hhs.md) and [medical insurance claims](hospital-admissions.md).  The CPR differs from these sources in that it is part of the public health surveillance stream (like HHS, unlike claims) but is available at a daily-county level (like claims, unlike HHS). CPR hospital admissions figures at the state level and above are meant to match those from HHS, but are known to differ. See the Limitations section for details.
+
+County, MSA, state, and HHS-level values are pulled directly from CPR when available; nation-level values are aggregated up from the state level.
 
 | Signal | Description |
 | --- | --- |
 | `confirmed_admissions_covid_1d_7dav` | Number of adult and pediatric confirmed COVID-19 hospital admissions occurring each day. Smoothed using a 7-day average.  <br/> **Earliest date available:** 2019-12-16 for state, HHS, and nation; 2021-01-06 for MSA and county |
 | `confirmed_admissions_covid_1d_prop_7dav` | Number of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. Smoothed using a 7-day average. <br/> **Earliest date available:** 2019-12-16 for state, HHS, and nation; 2021-01-06 for MSA and county |
+| `people_full_vaccinated` | "People fully vaccinated includes those who have received two doses of the Pfizer-BioNTech or Moderna vaccine and those who have received one dose of the J&J/Janssen vaccine" - from the CPR data dictionary.  <br/> **Earliest date available:** 2021-01-15 at any geo level except MSA and 2021-04-01 at the MSA level.|
+| `people_booster_doses` |"The count of people who received a booster dose includes anyone who is fully vaccinated and has received another dose of COVID-19 vaccine since 2021-08-13. This includes people who received booster doses and people who received additional doses." - from the CPR data dictionary. <br/> **Earliest date available:** 2021-11-01 for state, HHS, and nation. Not available below state level. |
+| `doses_admin_7dav` | "Doses administered shown by date of report, not date of administration. ... [S]ubmitting entities will have the ability to update or delete previously submitted records using new functionality available in CDC’s Data Clearinghouse. Use of this new functionality may result in fluctuations across metrics as historical data are updated or deleted" - from the CPR data dictionary.  Smoothed using a 7-day average. <br/> **Earliest date available:** 2021-04-29 for state, HHS, and nation. Not available below state level. |
+| `booster_doses_admin_7dav` | "Doses administered shown by date of report, not date of administration. ... [S]ubmitting entities will have the ability to update or delete previously submitted records using new functionality available in CDC’s Data Clearinghouse. Use of this new functionality may result in fluctuations across metrics as historical data are updated or deleted" - from the CPR data dictionary. "[A] booster dose includes anyone who is fully vaccinated and has received another dose of COVID-19 vaccine since August 13, 2021. This includes people who received booster doses and people who received additional doses." - from the CPR data dictionary. Smoothed using a 7-day average.<br/> **Earliest date available:** 2021-11-01 for state, HHS, and nation. Not available below state level. |
 
 ## Table of contents
 {: .no_toc .text-delta}
@@ -36,14 +42,27 @@ County, MSA, state, and HHS-level values are pulled directly from CPR; nation-le
 
 For counts-based fields like hospital admissions, CPR reports rolling sums for the preceding 7 days. The 7-day average signals are computed by Delphi by dividing each sum by 7 and assigning it to the last date in the included range, so e.g. the signal for June 7 is the average of the underlying data for June 1 through 7, inclusive.
 
-The `confirmed_admissions_covid_1d_7dav` signal mirrors the `Confirmed COVID-19 admissions - last 7 days` CPR field for all geographic resolutions except nation. Nation-level admissions is calculated by summing state-level values.
+The `confirmed_admissions_covid_1d_7dav` signal mirrors the `Confirmed COVID-19 admissions - last 7 days` CPR field for all geographic resolutions except nation. Nation-level admissions is calculated by summing state-level values. 
+
+The `doses_admin_7dav` and `booster_doses_admin_7dav` signals mirror the `Doses administered - last 7 days` and `Booster doses administered - last 7 days` CPR fields for all geographic resolutions except nation. Nation-level doses are calculated by summing state-level values.
 
 ## Limitations
 
 Nation-level estimates may be inaccurate since aggregations are done using state-level smoothed values instead of raw values. Ideally we would aggregate raw values before smoothing, but the raw values are not accessible in this case.
 
 Because DSEW does not provide updates on weekends, estimates are not available for all dates.
 
+Currently, of all the vaccination signals, county-level data is only available for `people_full_vaccinated`. Until 2021-11-15, several states reported vaccinated people not allocated to any individual county. These unallocated counts were reported using a FIPS code ending with `000` for that state, which is never a FIPS code for a real county.
+
+This data source is susceptible to large corrections that can create strange data effects such as negative counts and sudden changes of 1M+ counts from one day to the next. Many of these corrections are documented in the "High-Visibility Data Notes" section in the first tab of the CPR spreadsheet for that day. To locate the correct spreadsheet for some `time_value` R, consult the following table:
+
+| Signal type | CPR date |
+| - | - |
+| Hospital Admissions | usually R+2, sometimes R+1 |
+| Vaccinations | usually R+1, sometimes R+2 |
+
+Not all CPRs have the same lag between the CPR date (listed in the filename) and the date for a particular signal.
+
 ### Differences with HHS reports
 
 An analysis comparing the 
@@ -56,7 +75,7 @@ An analysis comparing the
 
 The report is currently updated daily, excluding weekends. However, this is subject to change; DSEW previously issued updates on a twice-weekly schedule. We check for updates daily.
 
-Hospital admissions are reported with a lag of 2 days, but since the CPR is not updated on weekends, lag effectively varies from 2-4 days.
+The CPR is prepared with an internal lag of 1-2 days for most signals. The file is usually posted to healthdata.gov the day after the date listed in the filename, excluding weekends and federal holidays. This results in an effective lag in COVIDcast of 2-4 days, or 5 days when Monday is a holiday.
 
 ## Source and Licensing
 

docs/api/covidcast-signals/quidel.md

@@ -20,7 +20,7 @@ grand_parent: COVIDcast Epidata API
 * **Earliest issue available:** July 29, 2020 
 * **Number of data revisions since May 19, 2020:** 1
 * **Date of last change:** October 22, 2020
-* **Available for:** hrr, msa, state (see [geography coding docs](../covidcast_geography.md))
+* **Available for:** county, hrr, msa, state, HHS, nation (see [geography coding docs](../covidcast_geography.md))
 * **Time type:** day (see [date format docs](../covidcast_times.md))
 * **License:** [CC BY](../covidcast_licensing.md#creative-commons-attribution)
 
@@ -68,60 +68,66 @@ $$
 p = \frac{100 x}{n}
 $$
 
-We estimate p across 3 temporal-spatial aggregation schemes:
+We estimate p across 6 temporal-spatial aggregation schemes:
+- daily, at the county level;
 - daily, at the MSA (metropolitan statistical area) level;
 - daily, at the HRR (hospital referral region) level;
-- daily, at the state level.
+- daily, at the state level;
+- daily, at the HHS level;
+- daily, at the US national level.
 
-**MSA and HRR levels**: In a given MSA or HRR, suppose $$N$$ COVID tests are taken
-in a certain time period, $$X$$ is the number of tests taken with positive
-results. 
+#### Standard Error
 
-For raw signals:
-- if $$N \geq 50$$, we simply use:
+We assume the estimates for each time point follow a binomial distribution. The
+estimated standard error then is:
 
 $$
-p = \frac{100 X}{N}
+\text{se} = 100 \sqrt{ \frac{\frac{p}{100}(1- \frac{p}{100})}{N} } 
 $$
 
-For smoothed signals, before taking the temporal pooling average,
-- if $$N \geq 50$$, we also use:
+#### Smoothing
+
+We add two kinds of smoothing to the smoothed signals:
+
+##### Temporal Smoothing
+Smoothed estimates are formed by pooling data over time. That is, daily, for
+each location, we first pool all data available in that location over the last 7
+days, and we then recompute everything described in the two subsections above. 
+
+Pooling in this way makes estimates available in more geographic areas, as many areas 
+report very few tests per day, but have enough data to report when 7 days are considered.
+
+##### Geographical Smoothing
+
+**County, MSA and HRR levels**: In a given County, MSA or HRR, suppose $$N$$ COVID tests 
+are taken in a certain time period, $$X$$ is the number of tests taken with positive
+results. 
+
+
+For smoothed signals, after taking the temporal pooling,
+- if $$N \geq 50$$, we still use:
 $$
 p = \frac{100 X}{N}
 $$
-- if $$25 \leq N < 50$$, we lend $$50 - N$$ fake samples from its home state to shrink the
+- if $$25 \leq N < 50$$, we lend $$50 - N$$ fake samples from its parent state to shrink the
 estimate to the state's mean, which means:
 $$
 p = 100 \left( \frac{N}{50} \frac{X}{N} + \frac{50 - N}{50}  \frac{X_s}{N_s} \right) 
 $$
 where $$N_s, X_s$$ are the number of COVID tests and the number of COVID tests
-taken with positive results taken in its home state in the same time period.
+taken with positive results taken in its parent state in the same time period.
+A parent state is defined as the state with the largest proportion of the population 
+in this county/MSA/HRR.
 
-**State level**: the states with fewer than 50 tests are discarded. For the
-rest of the states with sufficient samples,
+Counties with sample sizes smaller than 50 are merged into megacounties for 
+the raw signals; counties with sample sizes smaller than 25 are merged into megacounties for
+the smoothed signals.
 
+**State level, HHS level, National level**: locations with fewer than 50 tests are discarded. For the remaining locations,
 $$
 p = \frac{100 X}{N}
 $$
 
-#### Standard Error
-
-We assume the estimates for each time point follow a binomial distribution. The
-estimated standard error then is:
-
-$$
-\text{se} = 100 \sqrt{ \frac{\frac{p}{100}(1- \frac{p}{100})}{N} } 
-$$
-
-#### Smoothing
-
-Smoothed estimates are formed by pooling data over time. That is, daily, for
-each location, we first pool all data available in that location over the last 7
-days, and we then recompute everything described in the last two
-subsections. Pooling in this way makes estimates available in more geographic
-areas, as many areas report very few tests per day, but have enough data to
-report when 7 days are considered.
-
 ### Lag and Backfill
 
 Because testing centers may report their data to Quidel several days after they
@@ -142,13 +148,13 @@ This data source is based on data provided to us by a lab testing company. They
 
 ### Missingness
 
-When fewer than 50 tests are reported in a state on a specific day, no data is
+When fewer than 50 tests are reported in a state/a HHS region/US on a specific day, no data is
 reported for that area on that day; an API query for all reported states on that
 day will not include it.
 
-When fewer than 50 tests are reported in an HRR or MSA on a specific day, and
-not enough samples can be filled in from the parent state, no data is reported
-for that area on that day; an API query for all reported geographic areas on
+When fewer than 50 tests are reported in a county, HRR or MSA on a specific day, and
+not enough samples can be filled in from the parent state for smoothed signals specifically, 
+no data is reported for that area on that day; an API query for all reported geographic areas on
 that day will not include it.
 
 ## Flu Tests