setting the finalize option in tweak [options future.delete = FALSE]

[henrikbengtsson]

setting the finalize option in tweak [options future.delete = FALSE]

[HenrikBengtsson]

I just had a look, and yes, using option future.delete = FALSE will only skip the deletion of the batchtools registry if there's a failure. If there's no failure, then the registry (folder) is always removed:

future.batchtools/R/BatchtoolsFuture-class.R

Lines 632 to 660 in ede01c2

	if (!getOption("future.delete", interactive())) {
	status <- status(future)
	res <- future$result
	if (inherits(res, "FutureResult")) {
	if (result_has_errors(res)) status <- unique(c("error", status))
	}
	mdebug("delete(): status(<future>) = %s",
	paste(sQuote(status), collapse = ", "))
	if (any(c("error", "expired") %in% status)) {
	msg <- sprintf("Will not remove batchtools registry, because the status of the batchtools was %s and option 'future.delete' is FALSE or running in an interactive session: %s", paste(sQuote(status), collapse = ", "), sQuote(path)) #nolint
	mdebug("delete(): %s", msg)
	warning(msg)
	return(invisible(FALSE))
	}
	}
	## Control batchtools info output
	oopts <- options(batchtools.verbose = debug)
	on.exit(options(oopts))
	## Try to delete registry
	interval <- delta
	for (kk in seq_len(times)) {
	try(clearRegistry(reg = reg), silent = TRUE)
	try(removeRegistry(wait = 0.0, reg = reg), silent = FALSE)
	if (!file_test("-d", path)) break
	Sys.sleep(interval)
	interval <- alpha * interval
	}

The background for the future.delete option here was for troubleshooting purposes of failed futures/jobs; I never considered the need to keep it otherwise. Shouldn't be hard to add support for this though, so it's now on the todo list.

About your attempt to configure the backend via the plan() function: This is a direction the Future API is moving toward. This far the focus has been on the "frontend" (what future(), value(), ... should support), but more and more use cases show that there is need to better control backend-specific properties. This is in line with where the future package is heading/aiming, cf. HenrikBengtsson/future#172

[yonicd]

Thank you for the explanation! glad to hear the enhancements are in the works :)

[yonicd]

so the action to remove the contents is here in practice?

https://github.com/HenrikBengtsson/future.batchtools/blob/ede01c270487ee1371ed2b65968bd238bdf8547c/R/BatchtoolsFuture-class.R#L653_L660

which propagates to this https://github.com/mllg/batchtools/blob/3b0b1a9a59e377bb4d827e355d6955d66849c9e6/R/clearRegistry.R

[HenrikBengtsson]

Correct, plus some tweaks of assertions following that.

[HenrikBengtsson]

I've implemented this in the develop branch. After installing:

remotes::install_github("HenrikBengtsson/future.batchtools@develop")

The following now works:

library(future.batchtools)
plan(batchtools_local)

options(future.delete = FALSE)  ## <== DISABLE CLEANUP

f <- future(42)
v <- value(f)

print(f)
# BatchtoolsFuture:
# Label: '<none>'
# Expression:
# [1] 42
# Lazy evaluation: FALSE
# Asynchronous evaluation: TRUE
# Local evaluation: TRUE
# Environment: <environment: R_GlobalEnv>
# Capture standard output: TRUE
# Capture condition classes: 'message', 'warning'
# Globals: <none>
# Packages: <none>
# L'Ecuyer-CMRG RNG seed: <none>
# Resolved: TRUE
# Value: 56 bytes of class 'numeric'
# Early signalling: FALSE
# Owner process: a7e7b546-3d74-3d40-2c40-8de865c540e6
# Class: 'BatchtoolsFuture', 'Future', 'environment'
# batchtools cluster functions: 'Interactive'
# batchtools status: 'defined', 'finished', 'started', 'submitted'
# batchtools Registry:
#   Job Registry
#   Backend  : Interactive
#   File dir : /home/alice/.future/20190502_141813-obkGJW/batchtools_549386655
#   Work dir : /home/alice
#   Jobs     : 1
#   Seed     : 9954
#   Writeable: TRUE
#   File dir exists: TRUE
#   Work dir exists: TRUE

dir("/home/alice/.future/20190502_141813-obkGJW/batchtools_549386655")
# [1] "exports"           "external"          "jobs"             
# [4] "logs"              "more.args.rds"     "registry.rds"     
# [7] "results"           "updates"           "user.function.rds"

[yonicd]

Thank you!