Comments (9)
wlandau
commented on June 11, 2024
1
tar_map() acts on existing target objects, not the arguments of tar_render(). In other words, tar_render() runs first, and then tar_map() tries to act on the output.
In this situation, you actually do not need tar_map() at all. You loop over calls to tar_render_raw(), which I just added in dc9f8aa.
# _targets.R
library(targets)
library(tarchetypes)
tar_pipeline(
# upstream targets...
purrr::map2(
c("report1", "report2"),
c("report1.Rmd", "report2.Rmd"),
~tar_render_raw(.x, .y)
)
)from tarchetypes.
yonicd
commented on June 11, 2024
I am trying to run all reports in parallel using clustermq but am getting an odd error. does assert_package need to be exported for this to run?
reports <- c('periods','labs','survival',"model")
report_paths <- file.path('script/analysis',glue::glue('{reports}.Rmd'))
pipe_report <- tar_pipeline(
Q(tarchetypes::tar_render_raw,
name = reports, path = report_paths,
const = list(
render_arguments = quote(list(output_dir = 'deliv/reports',quiet = TRUE))
),
pkgs = c('tarchetypes'),
n_jobs = 4
)
)Starting 4 processes ...
Running 4 calculations (1 objs/0 Mb common; 1 calls/chunk) ...
Master: [9.6s 7.4% CPU]; Worker: [avg 75.7% CPU, max 372.2 Mb]
Error in summarize_result(job_result, n_errors, n_warnings, cond_msgs, :
2/2 jobs failed (0 warnings). Stopping.
(Error #1) could not find function "assert_package"
(Error #2) could not find function "assert_package"
In addition: Warning message:
package 'clustermq' was built under R version 4.0.3
Error: callr subprocess failed: 2/2 jobs failed (0 warnings). Stopping.
(Error #1) could not find function "assert_package"
(Error #2) could not find function "assert_package"
Type .Last.error.trace to see where the error occuredfrom tarchetypes.
wlandau
commented on June 11, 2024
The correct usage is discussed at https://wlandau.github.io/targets-manual/hpc.html#clustermq. Your tar_pipeline() does not need any parallel code at all for this, and it should not call Q(). tar_pipeline() simply defines a pipeline without running it. So in the _targets.R file in #18 (comment), all you need to do is add the appropriate clustermq options.
# _targets.R
library(targets)
library(tarchetypes)
options(clustermq.scheduler = "multicore") # More at https://mschubert.github.io/clustermq/articles/userguide.html#configuration-1
tar_pipeline(
# upstream targets...
purrr::map2(
c("report1", "report2"),
c("report1.Rmd", "report2.Rmd"),
~tar_render_raw(.x, .y)
)
)Then, tar_make_clustermq(workers = 4) will start 4 persistent workers and distribute the reports among them.
from tarchetypes.
yonicd
commented on June 11, 2024
oh my mistake. I wasn't thinking that the map will create 4 targets. I thought it was running 1 target and i had to parallelize it internally.
Thanks for the explanation!
from tarchetypes.
yonicd
commented on June 11, 2024
tar_make_clustermq errors out on winos
> tar_make_clustermq()
Error in self$run_clustermq() : attempt to apply non-function
In addition: Warning message:
package 'clustermq' was built under R version 4.0.3
Error: 'mccollect' is not an exported object from 'namespace:parallel'
Error: callr subprocess failed: attempt to apply non-function
Type .Last.error.trace to see where the error occured> .Last.error.trace
Stack trace:
Process 23556:
1. targets:::tar_make_clustermq()
2. targets:::callr_outer(targets_function = tar_make_clustermq_inner, ...
3. targets:::trn(is.null(callr_function), callr_inner(target_script_path(), ...
4. base:::do.call(callr_function, prepare_callr_arguments(callr_function, ...
5. (function (func, args = list(), libpath = .libPaths(), repos = default_repo ...
6. callr:::get_result(output = out, options)
7. throw(newerr, parent = remerr[[2]])
x callr subprocess failed: attempt to apply non-function
Process 8280:
19. (function (targets_script, targets_function, targets_arguments) ...
20. base:::do.call(targets_function, targets_arguments)
21. (function (pipeline, names_quosure, reporter, workers, log_worker) ...
22. clustermq_init(pipeline = pipeline, names = names, queue = "parallel", ...
23. self$run_clustermq()
24. base:::.handleSimpleError(function (e) ...
25. h(simpleError(msg, call))
x attempt to apply non-function session info
- Session info -------------------------------------------------------------------------------------------------------------------------------------
setting value
version R version 4.0.2 (2020-06-22)
os Windows 10 x64
system x86_64, mingw32
ui RStudio
language (EN)
collate English_United States.1252
ctype English_United States.1252
tz America/New_York
date 2020-11-12
- Packages -----------------------------------------------------------------------------------------------------------------------------------------
! package * version date lib source
P assertthat 0.2.1 2019-03-21 [?] CRAN (R 4.0.2)
backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2)
P base64enc 0.1-3 2015-07-28 [?] CRAN (R 4.0.0)
P bit 4.0.4 2020-08-04 [?] CRAN (R 4.0.3)
P bit64 4.0.5 2020-08-30 [?] CRAN (R 4.0.3)
P blastula 0.3.2 2020-05-19 [?] CRAN (R 4.0.2)
P callr 3.4.3 2020-03-28 [?] CRAN (R 4.0.2)
P class 7.3-17 2020-04-26 [?] CRAN (R 4.0.2)
P cli 2.0.2 2020-02-28 [?] CRAN (R 4.0.2)
P clipr 0.7.0 2019-07-23 [?] CRAN (R 4.0.2)
P clustermq 0.8.95.1 2020-07-13 [?] CRAN (R 4.0.3)
P codetools 0.2-16 2018-12-24 [?] CRAN (R 4.0.2)
P colorspace 1.4-1 2019-03-18 [?] CRAN (R 4.0.2)
P crayon 1.3.4 2017-09-16 [?] CRAN (R 4.0.2)
P curl 4.3 2019-12-02 [?] CRAN (R 4.0.2)
P data.table 1.13.0 2020-07-24 [?] CRAN (R 4.0.2)
P desc 1.2.0 2018-05-01 [?] CRAN (R 4.0.2)
P details 0.2.1 2020-09-08 [?] local
digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.3)
P dplyr 1.0.2 2020-08-18 [?] CRAN (R 4.0.2)
P ellipsis 0.3.1 2020-05-15 [?] CRAN (R 4.0.2)
P evaluate 0.14 2019-05-28 [?] CRAN (R 4.0.2)
P fansi 0.4.1 2020-01-08 [?] CRAN (R 4.0.2)
P farver 2.0.3 2020-01-16 [?] CRAN (R 4.0.2)
P forcats 0.5.0 2020-03-01 [?] CRAN (R 4.0.2)
P fs 1.5.0 2020-07-31 [?] CRAN (R 4.0.2)
P furrr 0.1.0 2018-05-16 [?] CRAN (R 4.0.2)
P future 1.18.0 2020-07-09 [?] CRAN (R 4.0.2)
P generics 0.0.2 2018-11-29 [?] CRAN (R 4.0.2)
P ggplot2 * 3.3.2 2020-06-19 [?] CRAN (R 4.0.2)
P globals 0.12.5 2019-12-07 [?] CRAN (R 4.0.0)
P glue 1.4.2 2020-08-27 [?] CRAN (R 4.0.2)
P gower 0.2.2 2020-06-23 [?] CRAN (R 4.0.2)
P gt 0.2.2 2020-08-05 [?] CRAN (R 4.0.2)
P gtable 0.3.0 2019-03-25 [?] CRAN (R 4.0.2)
P hardhat 0.1.4 2020-07-02 [?] CRAN (R 4.0.2)
P haven 2.3.1 2020-06-01 [?] CRAN (R 4.0.2)
P here 0.1 2017-05-28 [?] CRAN (R 4.0.2)
P hms 0.5.3 2020-01-08 [?] CRAN (R 4.0.2)
P htmltools 0.5.0 2020-06-16 [?] CRAN (R 4.0.2)
P htmlwidgets 1.5.1 2019-10-08 [?] CRAN (R 4.0.2)
P httr 1.4.2 2020-07-20 [?] CRAN (R 4.0.2)
P igraph 1.2.5 2020-03-19 [?] CRAN (R 4.0.2)
P ipred 0.9-9 2019-04-28 [?] CRAN (R 4.0.2)
P jsonlite 1.7.1 2020-09-07 [?] CRAN (R 4.0.2)
P keyring 1.1.0 2018-07-16 [?] CRAN (R 4.0.3)
P knitr 1.30 2020-09-22 [?] CRAN (R 4.0.3)
P labeling 0.3 2014-08-23 [?] CRAN (R 4.0.0)
P labelled 2.7.0 2020-09-21 [?] CRAN (R 4.0.3)
P lattice 0.20-41 2020-04-02 [?] CRAN (R 4.0.2)
P lava 1.6.8 2020-09-26 [?] CRAN (R 4.0.2)
P lifecycle 0.2.0 2020-03-06 [?] CRAN (R 4.0.2)
P listenv 0.8.0 2019-12-05 [?] CRAN (R 4.0.2)
P lubridate 1.7.9 2020-06-08 [?] CRAN (R 4.0.2)
P magrittr * 1.5 2014-11-22 [?] CRAN (R 4.0.2)
P MASS 7.3-51.6 2020-04-26 [?] CRAN (R 4.0.2)
P Matrix 1.2-18 2019-11-27 [?] CRAN (R 4.0.2)
P munsell 0.5.0 2018-06-12 [?] CRAN (R 4.0.2)
P nnet 7.3-14 2020-04-26 [?] CRAN (R 4.0.2)
P parsnip 0.1.3 2020-08-04 [?] CRAN (R 4.0.2)
P patchwork * 1.0.1 2020-06-22 [?] CRAN (R 4.0.2)
P pillar 1.4.6 2020-07-10 [?] CRAN (R 4.0.2)
P pkgconfig 2.0.3 2019-09-22 [?] CRAN (R 4.0.2)
P plyr 1.8.6 2020-03-03 [?] CRAN (R 4.0.2)
P png 0.1-7 2013-12-03 [?] CRAN (R 4.0.0)
P pROC 1.16.2 2020-03-19 [?] CRAN (R 4.0.2)
P processx 3.4.3 2020-07-05 [?] CRAN (R 4.0.2)
P prodlim 2019.11.13 2019-11-17 [?] CRAN (R 4.0.2)
P ps 1.3.4 2020-08-11 [?] CRAN (R 4.0.2)
P purrr 0.3.4 2020-04-17 [?] CRAN (R 4.0.2)
P qs 0.23.3 2020-10-05 [?] CRAN (R 4.0.3)
P R6 2.4.1 2019-11-12 [?] CRAN (R 4.0.2)
P RApiSerialize 0.1.0 2014-04-19 [?] CRAN (R 4.0.0)
P Rcpp 1.0.5 2020-07-06 [?] CRAN (R 4.0.2)
PD RcppParallel 5.0.2 2020-06-24 [?] CRAN (R 4.0.2)
P recipes 0.1.13 2020-06-23 [?] CRAN (R 4.0.2)
P rematch2 2.1.2 2020-05-01 [?] CRAN (R 4.0.2)
P remedy 0.1.1 2020-08-26 [?] github (thinkr-open/remedy@eb8fbdd)
P renv 0.12.0 2020-08-28 [?] CRAN (R 4.0.2)
rlang 0.4.8 2020-10-08 [1] CRAN (R 4.0.3)
P rmarkdown 2.5 2020-10-21 [?] CRAN (R 4.0.2)
P rpart 4.1-15 2019-04-12 [?] CRAN (R 4.0.2)
P rprojroot 1.3-2 2018-01-03 [?] CRAN (R 4.0.2)
P rsample 0.0.8 2020-09-23 [?] CRAN (R 4.0.2)
P rsconnect 0.8.16 2019-12-13 [?] CRAN (R 4.0.2)
P rstudioapi 0.11 2020-02-07 [?] CRAN (R 4.0.2)
sasdrive 0.0.1 2020-10-15 [1] Github (sagerx/sasdrive@a788cda)
P scales 1.1.1 2020-05-11 [?] CRAN (R 4.0.2)
P sessioninfo 1.1.1 2018-11-05 [?] CRAN (R 4.0.2)
P stringfish 0.14.2 2020-09-05 [?] CRAN (R 4.0.3)
P stringi 1.4.6 2020-02-17 [?] CRAN (R 4.0.0)
P stringr 1.4.0 2019-02-10 [?] CRAN (R 4.0.2)
P survival 3.1-12 2020-04-10 [?] CRAN (R 4.0.2)
targets * 0.0.0.9002 2020-10-27 [1] Github (wlandau/targets@fd9b7a4)
P tibble 3.0.3 2020-07-10 [?] CRAN (R 4.0.2)
P tidyr 1.1.2 2020-08-27 [?] CRAN (R 4.0.2)
P tidyselect 1.1.0 2020-05-11 [?] CRAN (R 4.0.2)
P timeDate 3043.102 2018-02-21 [?] CRAN (R 4.0.2)
P utf8 1.1.4 2018-05-24 [?] CRAN (R 4.0.2)
P uuid 0.1-4 2020-02-26 [?] CRAN (R 4.0.0)
P vctrs 0.3.4 2020-08-29 [?] CRAN (R 4.0.2)
P visNetwork 2.0.9 2019-12-06 [?] CRAN (R 4.0.2)
withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2)
P xfun 0.18 2020-09-29 [?] CRAN (R 4.0.3)
P xml2 1.3.2 2020-04-23 [?] CRAN (R 4.0.2)
P yaml 2.2.1 2020-02-01 [?] CRAN (R 4.0.2)
P yardstick 0.0.7 2020-07-13 [?] CRAN (R 4.0.2)
[1] C:/Users/jonathan.sidi/AppData/Local/Temp/RtmpshEHp7/renv-system-library
P -- Loaded and on-disk path mismatch.
D -- DLL MD5 mismatch, broken installation.
from tarchetypes.
yonicd
commented on June 11, 2024
sorry.... i was using
options(
tidyverse.quiet = TRUE,
clustermq.scheduler = "multicore"
)where on winOS it must be
options(
tidyverse.quiet = TRUE,
clustermq.scheduler = "multiprocess"
)from tarchetypes.
wlandau
commented on June 11, 2024
Yup, Windows requires multiprocess.
from tarchetypes.
wlandau
commented on June 11, 2024
As of 88389e2, #18 (comment) should condense down to this:
# _targets.R
library(targets)
library(tarchetypes)
values <- list(
name = rlang::syms(c("report1", "report2"),
file = c("report1.Rmd", "report2.Rmd")
)
tar_pipeline(
tar_eval(tar_render(name, file), values = values)
)from tarchetypes.
yonicd
commented on June 11, 2024
very nice. thanks!
from tarchetypes.
Related Issues (20)
- `tar_quarto()` always ends normally for quarto project even if there is error HOT 3
- `tar_quarto()` ignores `output_dir` in _quarto.yml when passed an individual file
- combine tar_cue_age with a conditional statement HOT 4
- Rep-specific seeds in tar_rep(), tar_map_rep(), etc. HOT 5
- optional garbage collection between reps of the `tar_rep*()` functions HOT 1
- tar_change repository not considered for change part
- Branches not in metadata: branches out of range
- GitHub interactions are temporarily limited because the maintainer is out of office.
- tar_cross() HOT 2
- Bug: `tar_quarto_rep()` throws an error if used together with `future::plan()` from _targets.R template HOT 1
- Support Quarto profiles? HOT 10
- Expose `tar_render()`, `tar_quarto()` and similar functions to the `deps` argument of `tar_target_raw()` HOT 8
- Errors and warnings with Quarto
- tar_quarto_rep doesn't work on reports in subdirectories HOT 2
- `retrieval = "none"` in quarto target factories HOT 2
- [general] Use `tar_rep()` and `tar_rep2()` inside of `tar_map()` HOT 2
- Allow trailing comma in `tar_map()` HOT 1
- Let `tar_map()` substitute more fields, e.g., `priority` HOT 2
- Safely allow tar_quarto() etc. to run the report from a custom working directory HOT 2
- Export `tar_format_alias()` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tarchetypes.