sorhawell/fastrditijuu

seamlessClusterDTUfuncitons

fastrditijuu's Issues

improve clean up

some requests

job.delay
dont ask torque too often, if at all
access to all qsub pars
remove echo cpu line in qsub template

files should be transferred thorugh transfer.gbar.dtu.dk only

rare error observed

#try run 48 models
library("fastRditijuu")
library(randomForest)
X = data.frame(replicate(24,rnorm(500)))
y = with(X,X1*X2+X3)

myFunc = function(mtry) tail(randomForest(x=X,y=y,mtry=mtry)$rsq,1)

#run 10 times local
out = lapply(X=rep(1:5,2),fun=myFunc)
plot(rep(1:5,2),out)

#run 1000 times on cluster
out = lply(X=rep(1:24,50),fun=myFunc,
#that extra stuff you need to mention
globalVar=list(X=X,y=y), #gotta mention global variables
packages=c("randomForest"), #mention packages to be installed and/or loaded
user="sowe", #mention user name, remember to set up private/public key
max.nodes = 80) #optional limit to certain number of nodes
# ... do not set higher than 80.

[1] "call server... make temp dir,"
frontend Tmepdir /tmp/4ESkdGd4wsY3
backend Tmepdir /zhome/c7/0/66069/tmp/5a3CSqZUKlwb
save variables, transfer variables, transfer executable, execute!
[1] "Printing from R backend on server..."
set backend work directory to: /zhome/c7/0/66069/tmp/5a3CSqZUKlwb
[1] "Loading exported variables ..."
[1] "Check, install and load packages..."
Loading required package: randomForest
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
Loading required package: BatchJobs
Loading required package: BBmisc
Loading required package: methods
Sourcing configuration file: '/zhome/c7/0/66069/R/x86_64-pc-linux-gnu-library/3.2/BatchJobs/etc/BatchJobs_global_config.R'
Sourcing configuration file: '/zhome/c7/0/66069/.BatchJobs.R'
BatchJobs configuration:
cluster functions: Torque
mail.from:
mail.to:
mail.start: none
mail.done: none
mail.error: none
default.resources:
debug: FALSE
raise.warnings: FALSE
staged.queries: TRUE
max.concurrent.jobs: Inf
fs.timeout: NA

save file with these globalVariables Tempdir.backend aPackage args export list.of.packages new.packages
[1] "following packages loaded on master"
[1] ".GlobalEnv" "package:BatchJobs" "package:methods"
[4] "package:BBmisc" "package:randomForest" "package:stats"
[7] "package:graphics" "package:grDevices" "package:utils"
[10] "package:datasets" "Autoloads" "package:base"
[1] "following global variables exported to server side:"
[1] "X" "y"
[1] "calling the function"
Creating dir: /zhome/c7/0/66069/tmp/5a3CSqZUKlwb/testBatchJobs-files
Saving registry: /zhome/c7/0/66069/tmp/5a3CSqZUKlwb/testBatchJobs-files/registry.RData
Adding 80 jobs to DB.
Saving conf: /zhome/c7/0/66069/tmp/5a3CSqZUKlwb/testBatchJobs-files/conf.RData
Submitting 80 chunks / 80 jobs.
Cluster functions: Torque.
Auto-mailer settings: start=none, done=none, error=none.
Writing 80 R scripts...
SubmitJobs |+ | 0% (00:00:00)SubmitJobs |+ | 1% (00:00:00)SubmitJobs |+ | 2% (00:00:00)SubmitJobs |++ | 4% (00:00:00)SubmitJobs |++ | 5% (00:00:00)SubmitJobs |+++ | 6% (00:00:00)SubmitJobs |++++ | 8% (00:00:00)SubmitJobs |++++ | 9% (00:00:10)SubmitJobs |+++++ | 10% (00:00:09)SubmitJobs |++++++ | 11% (00:00:07)SubmitJobs |++++++ | 12% (00:00:07)SubmitJobs |+++++++ | 14% (00:00:06)SubmitJobs |+++++++ | 15% (00:00:05)SubmitJobs |++++++++ | 16% (00:00:05)SubmitJobs |+++++++++ | 18% (00:00:04)SubmitJobs |+++++++++ | 19% (00:00:04)SubmitJobs |++++++++++ | 20% (00:00:04)SubmitJobs |++++++++++ | 21% (00:00:07)SubmitJobs |+++++++++++ | 22% (00:00:06)SubmitJobs |++++++++++++ | 24% (00:00:06)SubmitJobs |++++++++++++ | 25% (00:00:06)SubmitJobs |+++++++++++++ | 26% (00:00:05)SubmitJobs |+++++++++++++ | 28% (00:00:05)SubmitJobs |++++++++++++++ | 29% (00:00:04)SubmitJobs |+++++++++++++++ | 30% (00:00:04)SubmitJobs |+++++++++++++++ | 31% (00:00:04)SubmitJobs |++++++++++++++++ | 32% (00:00:06)SubmitJobs |+++++++++++++++++ | 34% (00:00:05)SubmitJobs |+++++++++++++++++ | 35% (00:00:05)SubmitJobs |++++++++++++++++++ | 36% (00:00:05)SubmitJobs |++++++++++++++++++ | 38% (00:00:05)SubmitJobs |+++++++++++++++++++ | 39% (00:00:04)SubmitJobs |++++++++++++++++++++ | 40% (00:00:04)SubmitJobs |++++++++++++++++++++ | 41% (00:00:04)SubmitJobs |+++++++++++++++++++++ | 42% (00:00:04)SubmitJobs |+++++++++++++++++++++ | 44% (00:00:03)SubmitJobs |++++++++++++++++++++++ | 45% (00:00:03)SubmitJobs |+++++++++++++++++++++++ | 46% (00:00:04)SubmitJobs |+++++++++++++++++++++++ | 48% (00:00:04)SubmitJobs |++++++++++++++++++++++++ | 49% (00:00:04)SubmitJobs |++++++++++++++++++++++++ | 50% (00:00:04)SubmitJobs |+++++++++++++++++++++++++ | 51% (00:00:03)SubmitJobs |++++++++++++++++++++++++++ | 52% (00:00:03)SubmitJobs |++++++++++++++++++++++++++ | 54% (00:00:03)SubmitJobs |+++++++++++++++++++++++++++ | 55% (00:00:03)SubmitJobs |++++++++++++++++++++++++++++ | 56% (00:00:03)SubmitJobs |++++++++++++++++++++++++++++ | 57% (00:00:02)SubmitJobs |+++++++++++++++++++++++++++++ | 59% (00:00:03)SubmitJobs |+++++++++++++++++++++++++++++ | 60% (00:00:03)SubmitJobs |++++++++++++++++++++++++++++++ | 61% (00:00:03)SubmitJobs |+++++++++++++++++++++++++++++++ | 62% (00:00:03)SubmitJobs |+++++++++++++++++++++++++++++++ | 64% (00:00:02)SubmitJobs |++++++++++++++++++++++++++++++++ | 65% (00:00:03)SubmitJobs |++++++++++++++++++++++++++++++++ | 66% (00:00:03)SubmitJobs |+++++++++++++++++++++++++++++++++ | 68% (00:00:02)SubmitJobs |++++++++++++++++++++++++++++++++++ | 69% (00:00:02)SubmitJobs |++++++++++++++++++++++++++++++++++ | 70% (00:00:02)SubmitJobs |+++++++++++++++++++++++++++++++++++ | 71% (00:00:02)SubmitJobs |++++++++++++++++++++++++++++++++++++ | 72% (00:00:02)SubmitJobs |++++++++++++++++++++++++++++++++++++ | 74% (00:00:02)SubmitJobs |+++++++++++++++++++++++++++++++++++++ | 75% (00:00:02)SubmitJobs |+++++++++++++++++++++++++++++++++++++ | 76% (00:00:01)SubmitJobs |++++++++++++++++++++++++++++++++++++++ | 78% (00:00:02)SubmitJobs |+++++++++++++++++++++++++++++++++++++++ | 79% (00:00:01)SubmitJobs |+++++++++++++++++++++++++++++++++++++++ | 80% (00:00:01)SubmitJobs |++++++++++++++++++++++++++++++++++++++++ | 81% (00:00:01)SubmitJobs |++++++++++++++++++++++++++++++++++++++++ | 82% (00:00:01)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++ | 84% (00:00:01)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++ | 85% (00:00:01)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++ | 86% (00:00:01)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++ | 88% (00:00:01)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++ | 89% (00:00:00)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++ | 90% (00:00:00)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++ | 91% (00:00:00)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++ | 92% (00:00:00)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++ | 94% (00:00:00)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++ | 95% (00:00:00)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++ | 96% (00:00:00)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++ | 98% (00:00:00)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++ | 99% (00:00:00)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++| 100% (00:00:00)

Sending 80 submit messages...
Might take some time, do not interrupt this!
Error in sqliteSendQuery(con, statement, bind.data) :
rsqlite_query_send: could not execute1: database is locked
Error in value(get(), ...) :
Error in dbSendMessages: Error in sqliteSendQuery(con, statement, bind.data) :
rsqlite_query_send: could not execute1: database is locked
Calls: with ... -> submitJobs -> -> value -> stopf
Execution halted
returning from server... fetch results from server,
scp: /zhome/c7/0/66069/tmp/5a3CSqZUKlwb/Tempout.rda: No such file or directory
read results,
Show Traceback

Rerun with Debug
Error in gzfile(file, "rb") : cannot open the connection In addition: Warning message:
In gzfile(file, "rb") :
cannot open compressed file '/tmp/4ESkdGd4wsY3/Tempout.rda', probable reason 'No such file or directory'

seed handling

include seed handling for non-deterministic processes