Resources: syntax

[HenrikBengtsson]

How to support the different cases of globals:

resources = globals:FALSE
resources = globals:all() - x - y
resources = globals:all() - x + y
resources = globals:all() + {{ vars }}
resources = globals:(all() + {{ vars }} - x)

[HenrikBengtsson]

future(..., resources = . ~ gpu & !fork & ram:32*GiB & file:/path/to/data.csv & time:00:30:25)

Flags:

• localhost
• fork
• gpu

Filesystem:

• file
• folder
• file permissions
• file identifiers, e.g. size, md5
• TMPDIR size

Networking:

• internet
• URL

Needs:

• packages
• package version
• R version (sorry for 'base' package version)
• R capabilities()
• RNG

Consumables:

• ram
• runtime

Genetic consumables (GREs):

• scratch
• licenses

Preferables:

• globals cache
• file cache
• localhost

[HenrikBengtsson]

> foo <- function(resources) resources
> res <- foo(resources= ~ gpu && !fork && r >= 3.2 && file("foo.txt") && ram <= 32*GiB && time <= 00:30:25 && scratch <= 256*GiB && matlab && capabilities == (jpeg || png) && profmem)
> res
~gpu && !fork && r >= 3.2 && file("foo.txt") && ram <= 32 * GiB && 
    time <= 0:30:25 && scratch <= 256 * GiB && matlab && capabilities == 
    (jpeg || png) && profmem

Or maybe:

> res <- foo(resources= ~ isTRUE({ getRversion() >= "3.2" && file_test("-f", "Makefile") && any(capabilities(c("jpeg", "png"))) && capabilities("profmem") }))
> res
~isTRUE({
    getRversion() >= "3.2" && file_test("-f", "Makefile") && 
        any(capabilities(c("jpeg", "png"))) && capabilities("profmem")
})

OTH, that requires R to evaluate, which means we need to launch R in order know if the resources are met, but can't always do that before the future is assigned to a worker, e.g. a HPC scheduler or a remote machine.

[HenrikBengtsson]

There are different types of "resources":

1. required - the R worker needs to support these resource specifications in order for the future to be successfully resolved
2. preferred - the future can be resolved also without these resources, but it will be more efficient if the R worker supported them

Syntax ideas:

• resources ~ R >= 4.0 & ?gpu - requires R (>= 4.0) and prefer a host with a GPU

[HenrikBengtsson]

resources= ~ gpu && !fork && r >= 3.2 && file("foo.txt") && ram <= 32*GiB && time <= 00:30:25 && scratch <= 256*GiB && matlab && capabilities == (jpeg || png) && profmem

resources= ~ gpu() && !fork() && r >= 3.2 && file("foo.txt") && ram(32*GiB) && time(00:30:25) && scratch(256*GiB) && matlab && capabilities(jpeg || png) && profmem()

gpu(nvidia, ram(64*GiB))
os(windows, version(10))
packages(MASS >= 3.2, future)

One problem with NSE is that it'll produce lots of false notes. R CMD check --as-cran will not complain about variables in formulas, e.g.

hello <- function() {
  model <- y ~ x + 1
  message("Hello world!")
  invisible(model)
}

passes with all OK, cf. https://github.com/HenrikBengtsson/teeny/blob/R_CMD_check/formula/R/hello.R.

[HenrikBengtsson]

Are all resources static? Or can a machine have a resource that comes and goes? I guess it could, e.g. a GPU might come and go based on other processes using it

[HenrikBengtsson]

RNG:

~rng
~rng(kind="random.org")