WISH: Try to report on scheduler-specific errors (e.g. 'PBS: job killed')

[henrikbengtsson]

WISH: Try to report on scheduler-specific errors (e.g. 'PBS: job killed')

[HenrikBengtsson]

Implemented. Example,

> library("future.batchtools")
> plan(batchtools_torque, resources = list(vmem = "200mb"))
> f <- future({ system("zcat large-file.txt.gz | sort -u") })
> print(f)
BatchtoolsFuture:
Label: 'batchtools'
Expression:
{
    system("zcat large-file.txt.gz | sort -u")
}
Lazy evaluation: FALSE
Asynchronous evaluation: TRUE
Local evaluation: TRUE
Environment: <environment: R_GlobalEnv>
Globals: <none>
Packages: <none>
L'Ecuyer-CMRG RNG seed: <none>
Resolved: FALSE
Value: <not collected>
Early signalling: FALSE
Owner process: 9fd2af8e-505b-a1e9-db42-9cc3831b890b
Class: 'BatchtoolsFuture', 'Future', 'environment'
batchtools cluster functions: 'TORQUE'
batchtools status: 'defined', 'queued', 'submitted', 'system'
batchtools Registry:
  Job Registry
  Name    : TORQUE
  File dir: /cbc/henrik/repositories/future.batchtools/.make/.future/20170718_233629-rRQ4dI/batchtools_336703133
  Work dir: /cbc/henrik/repositories/future.batchtools/.make
  Jobs    : 1
  Seed    : 1021282606

> v <- value(f)
Error: BatchtoolsExpiration: Future ('batchtools') expired (registry path /cbc/henrik/repositories/future.batchtools/.make/.future/20170718_233629-rRQ4dI/batchtools_336703133).. The last few lines of the logged output:
### [bt 2017-07-18 23:36:43]: Starting calculation of 1 jobs
### [bt 2017-07-18 23:36:43]: Setting working directory to '/cbc/henrik/repositories/future.batchtools/.make'
Loading required package: methods
### [bt 2017-07-18 23:36:43]: Memory measurement disabled
### [bt 2017-07-18 23:36:43]: Starting job [batchtools job.id=1]
=>> PBS: job killed: vmem 284557312 exceeded limit 209715200
In addition: Warning message:
In waitForJobs(ids = jobid, timeout = timeout, sleep = delta, stop.on.error = FALSE,  :
  Some jobs disappeared from the system
> 

[HenrikBengtsson]

A simple "expired" example without a scheduler:

> library("future.batchtools")
> plan(batchtools_local)
> x %<-% { cat("hello\n"); quit(save = "no") }
> x
Error: BatchtoolsExpiration: Future ('batchtools') expired (registry path /cbc/henrik/repositories/future.batchtools/.make/.future/20170718_234726-jbB3s0/batchtools_1168066799).. The last few lines of the logged output:
### [bt 2017-07-18 23:49:26]: Starting calculation of 1 jobs
### [bt 2017-07-18 23:49:26]: Setting working directory to '/cbc/henrik/repositories/future.batchtools/.make'
Loading required package: methods
### [bt 2017-07-18 23:49:26]: Memory measurement disabled
### [bt 2017-07-18 23:49:26]: Starting job [batchtools job.id=1]
hello
In addition: Warning message:
In waitForJobs(ids = jobid, timeout = timeout, sleep = delta, stop.on.error = FALSE,  :
  Some jobs disappeared from the system