Trouble changing plans

[wlandau]

This issue is related to #263 (comment) and ropensci/targets#198 (comment), and it interferes with the ability of targets to create heterogeneous transient workers (cc @mattwarkentin).

When I stick to a multisession plan before running any futures, things seem to work fine.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
resolved(out)
#> [1] TRUE

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

However, when I try to change to a sequential plan before calling resolved(), the connection gets corrupted.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

Same thing if I try to restore the original multisession plan.

library(future)
plan_sequential <- plan(multisession, workers = 4)
plan_multisession <- plan()
out <- future(1)
plan(plan_sequential)
plan(plan_multisession)
resolved(out)
#> Error: Cannot resolve MultisessionFuture (<none>), because the connection to the worker is corrupt: Connection (connection: index=4, description="NA", class="NA", mode="NA", text="NA", opened="NA", can read="NA", can write="NA", id=210, raw_id="<pointer: 0xd2>") is no longer valid. There is currently no registered R connection with that index 4

^{Created on 2020-10-19 by the reprex package (v0.3.0)}

[wlandau]

This could affect cloud computing strategies such as https://gist.github.com/DavisVaughan/865d95cf0101c24df27b37f4047dd2e5. On the bright side, though, future.callr seems unaffected.

[HenrikBengtsson]

It's a hack, but you can avoid the shutdown of the cluster by using:

plan(sequential, .cleanup = FALSE)

[wlandau]

Thanks, Henrik! I just committed ropensci/targets@1ede0fb, which uses .cleanup to preserve clusters until the very end of the pipeline. Seems to work. But what exactly makes .cleanup = FALSE a hack?

[HenrikBengtsson]

Only that it exists in the first place for internal purposes and I don't want to commit to long term support for it at this time - despite it's been the for a long time already

[wlandau]

Will there eventually be a different officially supported way to preserve clusters to allow more heterogeneity among workers?