Comments (9)
A couple options I use:
future::plan(future.batchtools::batchtools_sge, template = "sge-simple.tmpl")
seems to work this way.
from future.batchtools.
How do you provide defaults for the variables in the template file? I'm using a template that includes activating a conda environment:
$ cat ~/.batchtools.sge.tmpl
#!/bin/bash
## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>
## Combining output/error messages into one file
#$ -j y
## Giving the name of the output log file
#$ -o <%= log.file %>
## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd
## Use environment variables
#$ -V
## time
#$ -l h_rt=<%= resources$h_rt %>
## memory
#$ -l h_vmem=<%= resources$h_vmem %>
export PATH=<%= resources$conda.path %>:$PATH
source activate <%= resources$conda.env %>
## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
...and I'd like to set defaults for resources$conda.path
and resources$conda.env
. When just using batchtools, setting resources can be done with a config file:
$ cat ~/.batchtools.conf.R
default.resources = list(h_rt = '00:59:00',
h_vmem = '4G',
conda.env = "py3",
conda.path = "/ebio/abt3_projects/software/miniconda3/bin")
cluster.functions = makeClusterFunctionsSGE(template = "~/.batchtools.tmpl")
temp.dir = "/ebio/abt3_projects/temp_data/"
from future.batchtools.
From revisiting this section of the README, I think I understand a little more. I am also trying to use the resources
argument on SGE. This is my template file sge_batchtools.tmpl
:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -o <%= log.file %>
#$ -V
#$ -N <%= job.name %>
#$ -pe smp <%= resources[["slots"]] %>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
and my script run.R
:
library(future.batchtools)
future::plan(batchtools_sge(template = "sge_batchtools.tmpl"))
future(system2("hostname"))
which gives an error:
$ Rscript run.R
Loading required package: future
Error: Fatal error occurred: 101. Command 'qsub' produced exit code 2. Output: 'Unable to read script file because of error: ERROR! -pe option must have range as 2nd argument'
Execution halted
But when I replace <%= resources[["slots"]] %>
with 2
in sge_batchtools.tmpl
, Rscript run.R
submits one job with two slots as desired.
Related: HenrikBengtsson/future#181, HenrikBengtsson/future#263, ropensci/drake#169.
from future.batchtools.
Don't know SGE well enough, so I could be wrong, but I think you wanna specify parallel environment "smp" (symmetric multiprocessing) as in -pe smp 2
.
from future.batchtools.
My bad - I somehow missed that you do indeed specify smp
- I should go will to sleep now.
from future.batchtools.
Found the problem in #26 (comment): my run.R
script did not actually set the slots
element of resources
. This worked for me:
library(future.batchtools)
future::plan(batchtools_sge(template = "sge_batchtools.tmpl"))
future(system2("hostname"), resources = list(slots = 2))
As desired, I saw a short-lived job with 2 slots on the cluster.
from future.batchtools.
At least with the configuration that I have list above, I get no output from failed jobs. Moreover, it's not clear where the qsub job log file is, given that it's just set as <%= log.file %>
in the *.tmpl file. I also haven't found any documentation about how best to troubleshoot failed qsub jobs (eg., AFAK, there's no getLog()
for future.batchtools and batchtools::getLog()
doesn't work with future.batchtools jobs).
Is there a good way to troubleshoot failed jobs? Preferably, I would like a function to print the stderr/stdout from each job and the qacct -j JOBID
info. I really like using future.batchtools + future.apply, but it's always a pain to troubleshoot failed jobs.
from future.batchtools.
I also haven't found any documentation about how best to troubleshoot failed qsub jobs (eg., AFAK, there's no getLog() for future.batchtools and batchtools::getLog() doesn't work with future.batchtools jobs).
Still a problem.
Also, it's not clear what variables are available in the template. I know of job.name
, log.file
, and resources
, but are there any others? If so, is there documentation on this?
from future.batchtools.
I'd like to redirect this question/ask/request to the batchtools package. I agree that {future.batchtools} might be able to improve it's documentation on this but I want to minimize any type of redundacy here and thereby the risk of falling out of sync with {batchtools}; {batchtools} is in charge on how things work below the future layer.
from future.batchtools.
Related Issues (20)
- huge results file with 'conditions' - performance bottleneck HOT 4
- use batchtools directly for scheduling
- Eqw on SGE cluster while R code finishes without error
- Simple chunking with nested parallelism HOT 1
- Slurm readLog() Error - Option to change fs.latency & scheduler.latency from batchtools_slurm or future::tweak HOT 10
- Proposed bugfix for batchtools reveals bug in future.batchtools? HOT 4
- Problem forwarding batchtools resources to individual futures HOT 4
- problem with running example parallel futures using batchtools_lsf
- Add batchtools template for SGE
- batchtools templates: `resources[["asis"]]` for as-is declarations HOT 1
- print() for BatchtoolsFuture should report on the template file used HOT 1
- Compatibility with promises package HOT 2
- TESTS: Error: identical(Sys.getenv(), oenvs0) is not TRUE on MS Windows CRAN HOT 1
- run(), resolved()[?], and result(): the RNG state is updated - from where? HOT 3
- Add support for plan(batchtools_multicore, workers = I(1))
- All batchtools_nnn() functions should return the future invisibly
- Template tools: add option to ShellCheck rendered template
- Template tools: export functions to find and render template
- PERFORMANCE: status() to memoize "finalized" state? HOT 2
- Error: Log file for job with id 1 not available HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from future.batchtools.