Best practices for summarizing multiple models?

[mbjoseph]

In the case of multiple Stan programs model1.stan, model2.stan, .... passed to tar_stan_mcmc(), what's the ideal way to define downstream targets for each of these models? For instance, let's say I have a function my_summary_fn() that I want to apply across all MCMC objects (for concreteness, this function could do some posterior predictive checks or generate some figures).

So far I have three ideas, but I suspect there's a better way.

Idea 1: naively define targets using the generated target names

In the two model case this would look something like this:

# _targets.R
library(targets)
library(stantargets)

generate_data <- function(n) {
  true_beta <- stats::rnorm(n = 1, mean = 0, sd = 1)
  x <- seq(from = -1, to = 1, length.out = n)
  y <- stats::rnorm(n, x * true_beta, 1)
  list(n = n, x = x, y = y, true_beta = true_beta)
}

list(
  tar_stan_mcmc(
    example,
    c("model1.stan", "model2.stan"),
    data = generate_data(20),
  ), 
  tar_target(
    summary1,
    my_summary_fn(example_mcmc_model1),
  ), 
  tar_target(
    summary2,
    my_summary_fn(example_mcmc_model2),
  )
)

This works but is not DRY and would be hard to scale for many models.

Idea 2: combine MCMC targets manually based on generated names

Another approach would define a new downstream target that combines the MCMC targets from each model:

# _targets.R
library(targets)
library(stantargets)

generate_data <- function(n) {
  true_beta <- stats::rnorm(n = 1, mean = 0, sd = 1)
  x <- seq(from = -1, to = 1, length.out = n)
  y <- stats::rnorm(n, x * true_beta, 1)
  list(n = n, x = x, y = y, true_beta = true_beta)
}

list(
  tar_stan_mcmc(
    example,
    c("model1.stan", "model2.stan"),
    data = generate_data(20),
  ), 
  tar_target(
    mcmc_list, 
    list(example_mcmc_model1, example_mcmc_model2)
  ),
  tar_target(
    summaries, 
    my_summary_fn(mcmc_list), 
    pattern = map(mcmc_list), 
    iteration = "list"
  )
)

This also works, but requires manually specifying the names of the generated mcmc objects (example_mcmc_model1, example_mcmc_model2, ...), which doesn't seem ideal. I suspect there's a more programmatic way to do this that I'm missing.

Idea 3: create a target factory

I haven't tried this yet, but maybe this is a use case for a custom target factory that adds downstream targets to the factory that tar_stan_mcmc() produces.

Is there a smarter way to refer to the set of mcmc targets that result from using multiple models in tar_stan_mcmc()?

[wlandau]

If you can define custom summaries with everything you need from the models, it would probably be best to go with tar_stan_mcmc_rep_summary(). (The summaries argument in tar_stan_mcmc_rep_summary() lets you define custom summaries just like in posterior::summarize_draws().) As described in https://docs.ropensci.org/stantargets/articles/simulation.html, you can have multiple models, and the results across models are aggregated into a single data frame by default. The .file and .name columns distinguish among models, and you could use dplyr to post-process all those models together.

[wlandau]

Alternatively, you can use tarchetypes::tar_map() as described at https://books.ropensci.org/targets/static.html#map. If you do, it is best to frequently check visNetwork() to verify that your targets are connected properly. Sketch:

# _targets.R
library(rlang)
library(targets)
library(tarchetypes)
library(stantargets)

generate_data <- function(n) {
  # ...
}
summarize_mcmc <- function(mcmc) {
  # ...
}

list(
  tar_stan_mcmc(
    example,
    c("model1.stan", "model2.stan"),
    data = generate_data(20)
  ),
  tar_map(
    values = list(models = syms(c("example_mcmc_model1", "example_mcmc_model2"))),
    tar_target(summary, summarize_mcmc(models))
  )
)

[mbjoseph]

Great - thank you! 🙏