+The following summary is based on resampling the [MIMIC-IV Clinical Database
+Demo](https://physionet.org/content/mimic-iv-demo/2.2/) data set provided by
+PhysioNet.^1,2^ Using the MIMIC-IV Clinical Database Demo set will allow others to benchmark
+`medicalcoder` on their system with the same data as used here.
-```{r outtable, include = FALSE}
-outtable <- readRDS("outtable.rds")
-setDT(outtable)
-outtable_orig <- copy(outtable)
-outtable[, memory := memory / (1024^2)] # From KiB to GiB
-
-outtable <-
- dcast(
- data = outtable,
- data_class + encounters + method + subconditions ~ flag.method,
- value.var = list("time_seconds", "relative_time", "memory")
- )
+[^1^] Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023). MIMIC-IV Clinical Database Demo (version 2.2). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/dp1f-ex47
-neworder <-
- c("method",
- "subconditions",
- "encounters",
- "data_class",
- "time_seconds_current",
- "relative_time_current",
- "memory_current",
- "time_seconds_cumulative",
- "relative_time_cumulative",
- "memory_cumulative"
- )
+[^2^] Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., ... & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220. RRID:SCR_007345.
-setcolorder(outtable, neworder = neworder)
-outtable[, data_class := factor(data_class, levels = c("data.frame", "data.table", "tibble"))]
-setkey(outtable, method, subconditions, encounters, data_class)
+```{r mimic-iv-import, include = FALSE}
+mimicivdata <- readRDS(file.path("mimic-iv", "mimicivdata.rds"))
+data.table::setDT(mimicivdata)
+```
-outtable[, encounters := qwraps2::frmt(as.integer(encounters))]
+The MIMIC-IV demo data consists of
+`r data.table::uniqueN(mimicivdata, by = c("subject_id"))` subjects and a total of
+`r data.table::uniqueN(mimicivdata, by = c("subject_id", "hadm_id"))`
+encounters. A mix of ICD-9 and ICD-10 diagnostic and procedure codes are in the
+data. The subjects were resampled to generate data sets with upto 1,000,000
+subjects. Each generated data set was used to benchmark by
+
+* `method`:
+ * charlson_quan2005
+ * elixhauser_quan2005
+ * pccc_v3.1 without subconditions
+ * pccc_v3.1 with subconditions
+* data class:
+ * data.frame
+ * tibble
+ * data.table
+* `flag.method`
+ * current
+ * cumulative
+
+
-outtable[, method := fifelse(subconditions, paste(method, "(with subconditions)"), method)]
-outtable[, subconditions := NULL]
+```{r outtable, include = FALSE}
+mimicivbenchmark <-readRDS(file.path("mimic-iv", "benchmark.rds"))
+data.table::setDT(mimicivbenchmark)
```
In general, the expected time to apply a comorbidity method is lower for
`tibble`s than for base `data.frame`s, and lower still for `data.table`s. Best
observed case: a `data.table` took
-`r outtable_orig[data_class == "data.table", min(relative_time)]`
-the time of a `data.frame`; `tibble`s sit between the two.
-
-```{r show-outtable, echo = FALSE, results = "asis"}
-foo <- function(x) {
- kbl(
- x = x,
- format = "html",
- escape = FALSE,
- digits = 2,
- align = "ccrrrrrr",
- col.names = c("Encounters", "Data Class", rep(c("Time (seconds)", "Relative time", "Memory (GB)"), times = 2)),
- caption = "Expected time (seconds), relative time (with respect to data.frame), and expected memory use, by flagging method (current or cumulative), number of encounters, and input data storage format."
- ) |>
- collapse_rows(columns = 1, valign = "middle") |>
- row_spec(row = c(seq(1, nrow(x), by = 6)), background = "#D9D9D9") |>
- row_spec(row = c(seq(2, nrow(x), by = 6)), background = "#D9D9D9") |>
- row_spec(row = c(seq(3, nrow(x), by = 6)), background = "#D9D9D9") |>
- add_header_above(c("", "", "flag.method = 'current'" = 3, "flag.method = 'cumulative'" = 3))
-}
-
-cat("\n### Benchmarking Charlson (Quan 2005)\n\n")
-foo(x = outtable[method == "charlson_quan2005", -"method"])
-
-cat("\n### Benchmarking Elixhauser (Quan 2005)\n\n")
-foo(x = outtable[method == "elixhauser_quan2005", -"method"])
-
-cat("\n### Benchmarking PCCC v3.1 (without subconditions)\n\n")
-foo(x = outtable[method == "pccc_v3.1", -"method"])
-
-cat("\n### Benchmarking PCCC v3.1 (with subconditions)\n\n")
-foo(x = outtable[method == "pccc_v3.1 (with subconditions)", -"method"])
-```
+`r mimicivbenchmark[data_class == "data.table", min(relative_time)]`
+the time of a `data.frame`. Best case for `tibble`s was
+`r mimicivbenchmark[data_class == "tibble", min(relative_time)]`
+the time of a `data.frame`.
diff --git a/benchmarking/README.md b/benchmarking/README.md
index b708c1f9..6405a0de 100644
--- a/benchmarking/README.md
+++ b/benchmarking/README.md
@@ -7,1331 +7,55 @@
The major factors impacting the expected computation time for applying a
comorbidity algorithm to a data set are:
-1. Data size: number of subjects/encounters.
+1. Data size: number of encounters.
2. Data storage class: `medicalcoder` has been built such that no imports of
- other namespaces is required. That said, when a `data.table` is passed to
+ other namespaces is required. That said, when a `data.table` is passed to
`comorbidities()` and the `data.table` namespace is available, then S3
dispatch for `merge` is used, along with some other methods, to reduce memory
- use and reduce computation time.
+ use and reduce computation time. When a `tibble` is passed and tidyverse
+ namespaces are available, the tibble-aware path reduces time relative to a
+ base `data.frame`, though `data.table` remains the fastest option.
3. flag.method: "current" will take less time than the "cumulative" method.
-
+The following summary is based on resampling the [MIMIC-IV Clinical Database
+Demo](https://physionet.org/content/mimic-iv-demo/2.2/) data set provided by
+PhysioNet.^1,2^ Using the MIMIC-IV Clinical Database Demo set will allow others to benchmark
+`medicalcoder` on their system with the same data as used here.
+[^1^] Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023). MIMIC-IV Clinical Database Demo (version 2.2). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/dp1f-ex47
-
-In general, the expected time to apply a comorbidity method is the same between
-`data.frame`s and `tibble`s. There is a notable decrease in time required when
-`data.table`s are passed to `comorbidities()`. Best observed case: a
-`data.table` took
-0.3106419
-the time of a `data.frame`.
-
-
-### Benchmarking Charlson (Quan 2005)
-
-| Encounters | -Data Class | -Time (seconds) | -Relative time | -Memory (GB) | -Time (seconds) | -Relative time | -Memory (GB) | -
|---|---|---|---|---|---|---|---|
| 1,000 | -data.frame | -0.19 | -1.00 | -0.31 | -0.48 | -1.00 | -0.29 | -
| data.table | -0.17 | -0.89 | -0.31 | -0.39 | -0.81 | -0.29 | -|
| tibble | -0.23 | -1.20 | -0.31 | -0.54 | -1.20 | -0.29 | -|
| 2,000 | -data.frame | -0.28 | -1.00 | -0.31 | -0.85 | -1.00 | -0.31 | -
| data.table | -0.23 | -0.83 | -0.31 | -0.60 | -0.72 | -0.30 | -|
| tibble | -0.30 | -1.07 | -0.31 | -0.79 | -0.98 | -0.30 | -|
| 5,000 | -data.frame | -0.53 | -1.00 | -0.32 | -1.93 | -1.00 | -0.38 | -
| data.table | -0.39 | -0.75 | -0.32 | -1.20 | -0.62 | -0.36 | -|
| tibble | -0.49 | -0.93 | -0.32 | -1.47 | -0.77 | -0.36 | -|
| 10,000 | -data.frame | -0.99 | -1.00 | -0.34 | -4.00 | -1.00 | -0.48 | -
| data.table | -0.66 | -0.68 | -0.34 | -2.26 | -0.57 | -0.44 | -|
| tibble | -0.83 | -0.84 | -0.34 | -2.70 | -0.68 | -0.44 | -|
| 20,000 | -data.frame | -1.93 | -1.00 | -0.39 | -8.17 | -1.00 | -0.68 | -
| data.table | -1.18 | -0.62 | -0.39 | -4.26 | -0.53 | -0.61 | -|
| tibble | -1.50 | -0.78 | -0.39 | -5.13 | -0.63 | -0.62 | -|
| 50,000 | -data.frame | -4.72 | -1.00 | -0.54 | -20.42 | -1.00 | -1.26 | -
| data.table | -2.61 | -0.56 | -0.53 | -9.66 | -0.48 | -1.12 | -|
| tibble | -3.33 | -0.72 | -0.53 | -11.96 | -0.59 | -1.16 | -|
| 100,000 | -data.frame | -8.76 | -1.00 | -0.80 | -38.94 | -1.00 | -2.33 | -
| data.table | -4.50 | -0.52 | -0.79 | -17.37 | -0.45 | -2.02 | -|
| tibble | -5.60 | -0.66 | -0.80 | -21.50 | -0.56 | -2.20 | -|
| 200,000 | -data.frame | -16.55 | -1.00 | -1.32 | -75.52 | -1.00 | -4.43 | -
| data.table | -8.00 | -0.49 | -1.27 | -32.04 | -0.43 | -3.84 | -|
| tibble | -9.75 | -0.60 | -1.29 | -39.68 | -0.53 | -4.21 | -|
| 500,000 | -data.frame | -40.19 | -1.00 | -2.89 | -189.54 | -1.00 | -10.35 | -
| data.table | -18.37 | -0.46 | -2.72 | -76.28 | -0.41 | -9.39 | -|
| tibble | -22.18 | -0.56 | -2.71 | -94.83 | -0.51 | -9.75 | -|
| 1,000,000 | -data.frame | -80.76 | -1.00 | -5.71 | -389.11 | -1.00 | -20.00 | -
| data.table | -35.76 | -0.45 | -5.33 | -150.93 | -0.39 | -19.53 | -|
| tibble | -43.54 | -0.55 | -5.15 | -189.79 | -0.49 | -18.45 | -
| Encounters | -Data Class | -Time (seconds) | -Relative time | -Memory (GB) | -Time (seconds) | -Relative time | -Memory (GB) | -
|---|---|---|---|---|---|---|---|
| 1,000 | -data.frame | -0.20 | -1.00 | -0.31 | -0.67 | -1.00 | -0.29 | -
| data.table | -0.18 | -0.89 | -0.30 | -0.54 | -0.81 | -0.29 | -|
| tibble | -0.24 | -1.22 | -0.31 | -0.74 | -1.15 | -0.29 | -|
| 2,000 | -data.frame | -0.29 | -1.00 | -0.31 | -1.19 | -1.00 | -0.31 | -
| data.table | -0.25 | -0.84 | -0.30 | -0.85 | -0.73 | -0.32 | -|
| tibble | -0.32 | -1.11 | -0.31 | -1.12 | -0.97 | -0.31 | -|
| 5,000 | -data.frame | -0.56 | -1.00 | -0.33 | -2.72 | -1.00 | -0.39 | -
| data.table | -0.42 | -0.76 | -0.32 | -1.72 | -0.64 | -0.41 | -|
| tibble | -0.54 | -0.97 | -0.32 | -2.15 | -0.79 | -0.40 | -|
| 10,000 | -data.frame | -1.06 | -1.00 | -0.35 | -5.55 | -1.00 | -0.52 | -
| data.table | -0.72 | -0.69 | -0.34 | -3.23 | -0.58 | -0.56 | -|
| tibble | -0.92 | -0.88 | -0.34 | -3.97 | -0.72 | -0.54 | -|
| 20,000 | -data.frame | -2.10 | -1.00 | -0.39 | -11.14 | -1.00 | -0.81 | -
| data.table | -1.31 | -0.63 | -0.39 | -6.04 | -0.54 | -0.86 | -|
| tibble | -1.66 | -0.80 | -0.39 | -7.40 | -0.66 | -0.82 | -|
| 50,000 | -data.frame | -5.19 | -1.00 | -0.54 | -27.39 | -1.00 | -1.69 | -
| data.table | -2.92 | -0.57 | -0.55 | -13.76 | -0.51 | -1.72 | -|
| tibble | -3.66 | -0.71 | -0.55 | -16.85 | -0.62 | -1.62 | -|
| 100,000 | -data.frame | -9.49 | -1.00 | -0.81 | -52.73 | -1.00 | -3.17 | -
| data.table | -5.01 | -0.54 | -0.85 | -25.57 | -0.49 | -3.07 | -|
| tibble | -6.10 | -0.66 | -0.82 | -30.70 | -0.59 | -2.94 | -|
| 200,000 | -data.frame | -17.66 | -1.00 | -1.33 | -102.68 | -1.00 | -6.15 | -
| data.table | -8.78 | -0.51 | -1.40 | -48.30 | -0.48 | -5.73 | -|
| tibble | -10.56 | -0.62 | -1.35 | -57.25 | -0.57 | -5.59 | -|
| 500,000 | -data.frame | -42.42 | -1.00 | -2.92 | -255.09 | -1.00 | -15.10 | -
| data.table | -19.67 | -0.47 | -2.99 | -115.50 | -0.46 | -13.88 | -|
| tibble | -24.13 | -0.58 | -2.83 | -137.08 | -0.54 | -13.88 | -|
| 1,000,000 | -data.frame | -84.97 | -1.00 | -5.89 | -517.47 | -1.00 | -30.59 | -
| data.table | -37.75 | -0.45 | -5.64 | -228.19 | -0.45 | -28.42 | -|
| tibble | -47.85 | -0.58 | -5.32 | -273.80 | -0.53 | -28.96 | -
-| Encounters | -Data Class | -Time (seconds) | -Relative time | -Memory (GB) | -Time (seconds) | -Relative time | -Memory (GB) | -
|---|---|---|---|---|---|---|---|
| 1,000 | -data.frame | -0.25 | -1.00 | -0.30 | -1.39 | -1.00 | -0.34 | -
| data.table | -0.19 | -0.75 | -0.30 | -1.05 | -0.78 | -0.33 | -|
| tibble | -0.29 | -1.15 | -0.30 | -1.52 | -1.15 | -0.32 | -|
| 2,000 | -data.frame | -0.38 | -1.00 | -0.31 | -2.60 | -1.00 | -0.41 | -
| data.table | -0.26 | -0.68 | -0.31 | -1.73 | -0.68 | -0.40 | -|
| tibble | -0.39 | -1.03 | -0.31 | -2.40 | -0.96 | -0.39 | -|
| 5,000 | -data.frame | -0.77 | -1.00 | -0.33 | -6.18 | -1.00 | -0.63 | -
| data.table | -0.44 | -0.58 | -0.33 | -3.56 | -0.58 | -0.59 | -|
| tibble | -0.68 | -0.89 | -0.34 | -4.76 | -0.77 | -0.58 | -|
| 10,000 | -data.frame | -1.51 | -1.00 | -0.37 | -12.50 | -1.00 | -1.02 | -
| data.table | -0.77 | -0.51 | -0.35 | -6.61 | -0.53 | -0.92 | -|
| tibble | -1.20 | -0.79 | -0.38 | -8.65 | -0.69 | -0.94 | -|
| 20,000 | -data.frame | -3.02 | -1.00 | -0.44 | -24.77 | -1.00 | -1.81 | -
| data.table | -1.39 | -0.46 | -0.42 | -12.24 | -0.49 | -1.58 | -|
| tibble | -2.18 | -0.73 | -0.47 | -15.96 | -0.65 | -1.65 | -|
| 50,000 | -data.frame | -7.51 | -1.00 | -0.67 | -60.78 | -1.00 | -4.11 | -
| data.table | -3.11 | -0.42 | -0.60 | -27.76 | -0.46 | -3.59 | -|
| tibble | -4.84 | -0.65 | -0.73 | -36.71 | -0.61 | -3.78 | -|
| 100,000 | -data.frame | -14.21 | -1.00 | -1.08 | -118.87 | -1.00 | -7.83 | -
| data.table | -5.30 | -0.38 | -0.91 | -51.09 | -0.44 | -7.09 | -|
| tibble | -8.45 | -0.61 | -1.16 | -69.71 | -0.59 | -7.16 | -|
| 200,000 | -data.frame | -27.31 | -1.00 | -1.87 | -235.78 | -1.00 | -15.24 | -
| data.table | -9.34 | -0.35 | -1.49 | -97.30 | -0.42 | -14.05 | -|
| tibble | -15.28 | -0.57 | -1.98 | -135.19 | -0.58 | -13.80 | -|
| 500,000 | -data.frame | -67.70 | -1.00 | -4.14 | -601.65 | -1.00 | -37.74 | -
| data.table | -21.48 | -0.32 | -3.20 | -243.94 | -0.41 | -34.18 | -|
| tibble | -36.19 | -0.54 | -4.31 | -336.73 | -0.56 | -33.70 | -|
| 1,000,000 | -data.frame | -137.87 | -1.00 | -8.04 | -1247.81 | -1.00 | -76.08 | -
| data.table | -42.50 | -0.31 | -6.19 | -510.88 | -0.41 | -66.85 | -|
| tibble | -72.45 | -0.53 | -8.44 | -690.52 | -0.56 | -67.83 | -
| Encounters | -Data Class | -Time (seconds) | -Relative time | -Memory (GB) | -Time (seconds) | -Relative time | -Memory (GB) | -
|---|---|---|---|---|---|---|---|
| 1,000 | -data.frame | -0.27 | -1.00 | -0.30 | -1.51 | -1.00 | -0.34 | -
| data.table | -0.22 | -0.83 | -0.31 | -1.17 | -0.80 | -0.34 | -|
| tibble | -0.40 | -1.53 | -0.31 | -1.74 | -1.24 | -0.33 | -|
| 2,000 | -data.frame | -0.42 | -1.00 | -0.30 | -2.84 | -1.00 | -0.41 | -
| data.table | -0.30 | -0.73 | -0.31 | -1.93 | -0.70 | -0.40 | -|
| tibble | -0.53 | -1.32 | -0.31 | -2.73 | -1.01 | -0.40 | -|
| 5,000 | -data.frame | -0.84 | -1.00 | -0.33 | -6.77 | -1.00 | -0.64 | -
| data.table | -0.51 | -0.62 | -0.33 | -3.97 | -0.59 | -0.60 | -|
| tibble | -0.89 | -1.07 | -0.34 | -5.38 | -0.79 | -0.61 | -|
| 10,000 | -data.frame | -1.62 | -1.00 | -0.36 | -13.61 | -1.00 | -1.06 | -
| data.table | -0.88 | -0.55 | -0.36 | -7.31 | -0.54 | -0.95 | -|
| tibble | -1.50 | -0.92 | -0.37 | -9.79 | -0.71 | -0.97 | -|
| 20,000 | -data.frame | -3.24 | -1.00 | -0.44 | -26.82 | -1.00 | -1.87 | -
| data.table | -1.60 | -0.50 | -0.42 | -13.54 | -0.51 | -1.65 | -|
| tibble | -2.69 | -0.83 | -0.45 | -17.90 | -0.67 | -1.71 | -|
| 50,000 | -data.frame | -8.09 | -1.00 | -0.67 | -65.35 | -1.00 | -4.26 | -
| data.table | -3.59 | -0.45 | -0.60 | -31.18 | -0.48 | -3.78 | -|
| tibble | -5.95 | -0.75 | -0.67 | -40.48 | -0.63 | -3.93 | -|
| 100,000 | -data.frame | -15.56 | -1.00 | -1.07 | -127.35 | -1.00 | -8.20 | -
| data.table | -6.26 | -0.41 | -0.92 | -59.44 | -0.47 | -7.40 | -|
| tibble | -10.21 | -0.67 | -1.04 | -75.43 | -0.60 | -7.63 | -|
| 200,000 | -data.frame | -30.28 | -1.00 | -1.84 | -252.49 | -1.00 | -16.10 | -
| data.table | -11.32 | -0.38 | -1.53 | -115.89 | -0.46 | -14.68 | -|
| tibble | -18.18 | -0.61 | -1.74 | -145.13 | -0.58 | -14.86 | -|
| 500,000 | -data.frame | -75.12 | -1.00 | -4.06 | -647.57 | -1.00 | -40.13 | -
| data.table | -26.77 | -0.36 | -3.39 | -291.30 | -0.45 | -36.34 | -|
| tibble | -42.37 | -0.57 | -3.82 | -365.95 | -0.57 | -35.56 | -|
| 1,000,000 | -data.frame | -151.87 | -1.00 | -7.90 | -1349.61 | -1.00 | -81.79 | -
| data.table | -53.90 | -0.36 | -6.74 | -602.01 | -0.45 | -72.67 | -|
| tibble | -84.35 | -0.56 | -7.57 | -764.11 | -0.57 | -69.49 | -