Skip to content

Commit

Permalink
VIGNETTE: Give example of 'bigmemory' crashing parallel workers
Browse files Browse the repository at this point in the history
  • Loading branch information
HenrikBengtsson committed Mar 23, 2024
1 parent 67c7906 commit 4e8afb7
Show file tree
Hide file tree
Showing 3 changed files with 75 additions and 3 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Package: future
Version: 1.33.1-9009
Version: 1.33.1-9010
Title: Unified Parallel and Distributed Processing in R for Everyone
Imports:
digest,
Expand Down
4 changes: 2 additions & 2 deletions R/options.R
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,9 @@
#' environment variable \env{R_FUTURE_*} _when the \pkg{future} package is
#' loaded_. This means that those environment variables must be set before
#' the \pkg{future} package is loaded in order to have an effect.
#' For example, if `R_FUTURE_RNG_ONMISUSE = "ignore"`, then option
#' For example, if `R_FUTURE_RNG_ONMISUSE="ignore"`, then option
#' \option{future.rng.onMisuse} is set to `"ignore"` (character string).
#' Similarly, if `R_FUTURE_GLOBALS_MAXSIZE = "50000000"`, then option
#' Similarly, if `R_FUTURE_GLOBALS_MAXSIZE="50000000"`, then option
#' \option{future.globals.maxSize} is set to `50000000` (numeric).
#'
#' @examples
Expand Down
72 changes: 72 additions & 0 deletions vignettes/future-4-non-exportable-objects.md.rsp
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ _If you identify other cases, please consider [reporting](https://github.com/Hen
:---------------|:-------------------------------------------
**arrow** | Table (`externalptr`)
**base** | connection (`externalptr`)
**bigmemory** | big.matrix (`externalptr`)
**cpp11** | E.g. functions created by `cpp_source()`
**DBI** | DBIConnection (`externalptr`)
**inline** | CFunc (`externalptr` of class DLLHandle)
Expand Down Expand Up @@ -191,6 +192,77 @@ print(v)
```


#### Package: bigmemory

The **[bigmemory]** package provides mechanisms for working with very
large matrices that can be updated in-place, which helps save
memory. For example,

```r
library(bigmemory)

g <- function(x) {
x[1,1] <- 42L
x
}

x <- big.matrix(nrow = 3, ncol = 2, type = "integer")
print(x[1,1])
#> [1] NA

void <- g(x)
print(x[1,1])
#> [1] 42
```

Note how `x` was updated in-place. This is achived by `big.matrix`
objects holds an external pointer to where the matrix data is stored;

```r
str(x)
#> Formal class 'big.matrix' [package "bigmemory"] with 1 slot
#> ..@ address:<externalptr>
```

If we would try to use `x` in a parallel worker, then the parallel
worker crashes due to a bug in **bigmemory**, e.g.

```r
library(bigmemory)

library(future)
plan(multisession, workers = 2)

x <- big.matrix(nrow = 3, ncol = 2, type = "integer")
f <- future(dim(x), packages = "bigmemory")
value(f)
#> Error in unserialize(node$con) :
#> MultisessionFuture (<none>) failed to receive message results from
#> cluster RichSOCKnode #1 (PID 1746676 on localhost 'localhost'). The
#> reason reported was 'error reading from connection'. Post-mortem
#> diagnostic: No process exists with this PID, i.e. the localhost worker
#> is no longer alive. Detected a non-exportable reference
#> ('externalptr') in one of the globals ('x' of class 'big.matrix') used
#> in the future expression. The total size of the 1 globals exported is
#> 696 bytes. There is one global: 'x' (696 bytes of class 'S4')
```

We can protected against this setting:

```r
options(future.globals.onReference = "error")
```

which gives:

```r
f <- future(dim(x), packages = "bigmemory")
#> Error: Detected a non-exportable reference ('externalptr') in one
#> of the globals ('x' of class 'big.matrix') used in the future
#> expression
```


#### Package: cpp11

Another example is **[cpp11]**, which allows us to easily create R functions that are implemented in C++, e.g.
Expand Down

0 comments on commit 4e8afb7

Please sign in to comment.