Giter Site home page Giter Site logo

ropensci / tarchetypes Goto Github PK

View Code? Open in Web Editor NEW
119.0 7.0 15.0 1.51 MB

Archetypes for targets and pipelines

Home Page: https://docs.ropensci.org/tarchetypes

License: Other

R 100.00%
reproducibility high-performance-computing r data-science rstats pipeline r-package workflow targets r-targetopia

tarchetypes's Introduction

tarchetypes

ropensci zenodo R Targetopia CRAN status check codecov lint

The tarchetypes R package is a collection of target and pipeline archetypes for the targets package. These archetypes express complicated pipelines with concise syntax, which enhances readability and thus reproducibility. Archetypes are possible because of the flexible metaprogramming capabilities of targets. In targets, one can define a target as an object outside the central pipeline, and the tar_target_raw() function completely avoids non-standard evaluation. That means anyone can write their own niche interfaces for specialized projects. tarchetypes aims to include the most common and versatile archetypes and usage patterns.

Grouped data frames

tarchetypes has functions for easy dynamic branching over subsets of data frames:

  • tar_group_by(): define row groups using dplyr::group_by() semantics.
  • tar_group_select(): define row groups using tidyselect semantics.
  • tar_group_count(): define a given number row groups.
  • tar_group_size(): define row groups of a given size.

If you define a target with one of these functions, all downstream dynamic targets will automatically branch over the row groups.

# _targets.R file:
library(targets)
library(tarchetypes)
produce_data <- function() {
  expand.grid(var1 = c("a", "b"), var2 = c("c", "d"), rep = c(1, 2, 3))
}
list(
  tar_group_by(data, produce_data(), var1, var2),
  tar_target(group, data, pattern = map(data))
)
# R console:
library(targets)
tar_make()
#> ▶ dispatched target data
#> ● completed target data [0.007 seconds]
#> ▶ dispatched branch group_b3d7d010
#> ● completed branch group_b3d7d010 [0 seconds]
#> ▶ dispatched branch group_6a76c5c0
#> ● completed branch group_6a76c5c0 [0 seconds]
#> ▶ dispatched branch group_164b16bf
#> ● completed branch group_164b16bf [0 seconds]
#> ▶ dispatched branch group_f5aae602
#> ● completed branch group_f5aae602 [0 seconds]
#> ● completed pattern group
#> ▶ completed pipeline [0.104 seconds]

# First row group:
tar_read(group, branches = 1)
#> # A tibble: 3 × 4
#>   var1  var2    rep tar_group
#>   <fct> <fct> <dbl>     <int>
#> 1 a     c         1         1
#> 2 a     c         2         1
#> 3 a     c         3         1

# Second row group:
tar_read(group, branches = 2)
#> # A tibble: 3 × 4
#>   var1  var2    rep tar_group
#>   <fct> <fct> <dbl>     <int>
#> 1 a     d         1         2
#> 2 a     d         2         2
#> 3 a     d         3         2

Literate programming

Consider the following R Markdown report.

---
title: report
output: html_document
---

```{r}
library(targets)
tar_read(dataset)
```

We want to define a target to render the report. And because the report calls tar_read(dataset), this target needs to depend on dataset. Without tarchetypes, it is cumbersome to set up the pipeline correctly.

# _targets.R
library(targets)
list(
  tar_target(dataset, data.frame(x = letters)),
  tar_target(
    report, {
      # Explicitly mention the symbol `dataset`.
      list(dataset)
      # Return relative paths to keep the project portable.
      fs::path_rel(
        # Need to return/track all input/output files.
        c( 
          rmarkdown::render(
            input = "report.Rmd",
            # Always run from the project root
            # so the report can find _targets/.
            knit_root_dir = getwd(),
            quiet = TRUE
          ),
          "report.Rmd"
        )
      )
    },
    # Track the input and output files.
    format = "file",
    # Avoid building small reports on HPC.
    deployment = "main"
  )
)

With tarchetypes, we can simplify the pipeline with the tar_render() archetype.

# _targets.R
library(targets)
library(tarchetypes)
list(
  tar_target(dataset, data.frame(x = letters)),
  tar_render(report, "report.Rmd")
)

Above, tar_render() scans code chunks for mentions of targets in tar_load() and tar_read(), and it enforces the dependency relationships it finds. In our case, it reads report.Rmd and then forces report to depend on dataset. That way, tar_make() always processes dataset before report, and it automatically reruns report.Rmd whenever dataset changes.

Alternative pipeline syntax

tar_plan() is a drop-in replacement for drake_plan() in the targets ecosystem. It lets users write targets as name/command pairs without having to call tar_target().

tar_plan(
  tar_file(raw_data_file, "data/raw_data.csv", format = "file"),
  # Simple drake-like syntax:
  raw_data = read_csv(raw_data_file, col_types = cols()),
  data =raw_data %>%
    mutate(Ozone = replace_na(Ozone, mean(Ozone, na.rm = TRUE))),
  hist = create_plot(data),
  fit = biglm(Ozone ~ Wind + Temp, data),
  # Needs tar_render() because it is a target archetype:
  tar_render(report, "report.Rmd")
)

Installation

Type Source Command
Release CRAN install.packages("tarchetypes")
Development GitHub remotes::install_github("ropensci/tarchetypes")
Development rOpenSci install.packages("tarchetypes", repos = "https://dev.ropensci.org")

Documentation

For specific documentation on tarchetypes, including the help files of all user-side functions, please visit the reference website. For documentation on targets in general, please visit the targets reference website. Many of the linked resources use tarchetypes functions such as tar_render().

Help

Please read the help guide to learn how best to ask for help using targets and tarchetypes.

Code of conduct

Please note that this package is released with a Contributor Code of Conduct.

Citation

citation("tarchetypes")
#> To cite tarchetypes in publications use:
#> 
#>   William Michael Landau (2021). tarchetypes: Archetypes for Targets.
#>   https://docs.ropensci.org/tarchetypes/,
#>   https://github.com/ropensci/tarchetypes.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {tarchetypes: Archetypes for Targets},
#>     author = {William Michael Landau},
#>     year = {2021},
#>     note = {{https://docs.ropensci.org/tarchetypes/, https://github.com/ropensci/tarchetypes}},
#>   }

tarchetypes's People

Contributors

fkohrt avatar mikemahoney218 avatar mutlusun avatar noamross avatar wlandau avatar wlandau-lilly avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

tarchetypes's Issues

using tar_render in a tar_map

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

What trouble are you having?

Is it possible to tar_map over a set of rmds?

Reproducible example

reports <- tibble::tibble(outputs = c('report_a','report_b'))%>%
  dplyr::mutate(
    paths =  sprintf("script/analysis/%s.Rmd",outputs)
  )

  tar_map(
    values = reports,
    names = 'outputs',
    tar_render(period_report, paths,output_dir = 'deliv/reports')
    )
> tar_make()
Error in assert_scalar(path, "tar_render() only takes one file at a time.") : 
  object 'paths' not found
Error: callr subprocess failed: object 'paths' not found
Type .Last.error.trace to see where the error occured
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Desired result

What output or behavior do you want to see? Please be as specific as you can.

same behavior as writing each one out.

  tarchetypes::tar_render(period_report,
                          "script/analysis/periods.Rmd",
                          output_dir = 'deliv/reports'),
  tarchetypes::tar_render(survival_report,
                          "script/analysis/survival.Rmd",
                          output_dir = 'deliv/reports'),
  tarchetypes::tar_render(labs_report,
                          "script/analysis/labs.Rmd",
                          output_dir = 'deliv/reports'),
  tarchetypes::tar_render(model,
                          "script/analysis/period_pred.Rmd",
                          output_dir = 'deliv/reports')

Diagnostic information

session info
- Session info -------------------------------------------------------------------
 setting  value                       
 version  R version 4.0.2 (2020-06-22)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/New_York            
 date     2020-10-28                  

- Packages -----------------------------------------------------------------------
 ! package      * version    date       lib source                              
 P abind          1.4-5      2016-07-21 [?] CRAN (R 4.0.0)                      
 P assertthat     0.2.1      2019-03-21 [?] CRAN (R 4.0.2)                      
   backports      1.1.10     2020-09-15 [1] CRAN (R 4.0.2)                      
 P base64enc      0.1-3      2015-07-28 [?] CRAN (R 4.0.0)                      
 P broom          0.7.0      2020-07-09 [?] CRAN (R 4.0.2)                      
 P callr          3.4.3      2020-03-28 [?] CRAN (R 4.0.2)                      
 P car            3.0-9      2020-08-11 [?] CRAN (R 4.0.2)                      
 P carData        3.0-4      2020-05-22 [?] CRAN (R 4.0.0)                      
 P cellranger     1.1.0      2016-07-27 [?] CRAN (R 4.0.2)                      
 P checkmate      2.0.0      2020-02-06 [?] CRAN (R 4.0.2)                      
 P cli            2.0.2      2020-02-28 [?] CRAN (R 4.0.2)                      
 P clipr          0.7.0      2019-07-23 [?] CRAN (R 4.0.2)                      
 P codetools      0.2-16     2018-12-24 [?] CRAN (R 4.0.2)                      
 P colorspace     1.4-1      2019-03-18 [?] CRAN (R 4.0.2)                      
 P crayon         1.3.4      2017-09-16 [?] CRAN (R 4.0.2)                      
 P curl           4.3        2019-12-02 [?] CRAN (R 4.0.2)                      
 P data.table     1.13.0     2020-07-24 [?] CRAN (R 4.0.2)                      
 P desc           1.2.0      2018-05-01 [?] CRAN (R 4.0.2)                      
 P details        0.2.1      2020-09-08 [?] local                               
   digest         0.6.27     2020-10-24 [1] CRAN (R 4.0.3)                      
 P doParallel     1.0.15     2019-08-02 [?] CRAN (R 4.0.2)                      
 P dplyr          1.0.2      2020-08-18 [?] CRAN (R 4.0.2)                      
 P ellipsis       0.3.1      2020-05-15 [?] CRAN (R 4.0.2)                      
 P evaluate       0.14       2019-05-28 [?] CRAN (R 4.0.2)                      
 P fansi          0.4.1      2020-01-08 [?] CRAN (R 4.0.2)                      
 P farver         2.0.3      2020-01-16 [?] CRAN (R 4.0.2)                      
 P flextable      0.5.11     2020-09-09 [?] CRAN (R 4.0.2)                      
 P forcats        0.5.0      2020-03-01 [?] CRAN (R 4.0.2)                      
 P foreach        1.5.0      2020-03-30 [?] CRAN (R 4.0.2)                      
 P foreign        0.8-80     2020-05-24 [?] CRAN (R 4.0.2)                      
 P fs             1.5.0      2020-07-31 [?] CRAN (R 4.0.2)                      
 P furrr          0.1.0      2018-05-16 [?] CRAN (R 4.0.2)                      
 P future         1.18.0     2020-07-09 [?] CRAN (R 4.0.2)                      
 P gdtools        0.2.2      2020-04-03 [?] CRAN (R 4.0.2)                      
 P generics       0.0.2      2018-11-29 [?] CRAN (R 4.0.2)                      
 P ggplot2        3.3.2      2020-06-19 [?] CRAN (R 4.0.2)                      
 P ggpubr         0.4.0      2020-06-27 [?] CRAN (R 4.0.2)                      
 P ggrepel        0.8.2      2020-03-08 [?] CRAN (R 4.0.2)                      
 P ggsignif       0.6.0      2019-08-08 [?] CRAN (R 4.0.2)                      
 P globals        0.12.5     2019-12-07 [?] CRAN (R 4.0.0)                      
 P glue           1.4.2      2020-08-27 [?] CRAN (R 4.0.2)                      
 P gridExtra      2.3        2017-09-09 [?] CRAN (R 4.0.2)                      
 P gt             0.2.2      2020-08-05 [?] CRAN (R 4.0.2)                      
 P gtable         0.3.0      2019-03-25 [?] CRAN (R 4.0.2)                      
 P gtsummary      1.3.5      2020-09-29 [?] CRAN (R 4.0.2)                      
 P haven          2.3.1      2020-06-01 [?] CRAN (R 4.0.2)                      
 P here           0.1        2017-05-28 [?] CRAN (R 4.0.2)                      
 P hms            0.5.3      2020-01-08 [?] CRAN (R 4.0.2)                      
 P htmltools      0.5.0      2020-06-16 [?] CRAN (R 4.0.2)                      
 P htmlwidgets    1.5.1      2019-10-08 [?] CRAN (R 4.0.2)                      
 P httr           1.4.2      2020-07-20 [?] CRAN (R 4.0.2)                      
 P igraph         1.2.5      2020-03-19 [?] CRAN (R 4.0.2)                      
 P iterators      1.0.12     2019-07-26 [?] CRAN (R 4.0.2)                      
 P jsonlite       1.7.1      2020-09-07 [?] CRAN (R 4.0.2)                      
 P km.ci          0.5-2      2009-08-30 [?] CRAN (R 4.0.2)                      
 P KMsurv         0.1-5      2012-12-03 [?] CRAN (R 4.0.0)                      
 P knitr          1.30       2020-09-22 [?] CRAN (R 4.0.3)                      
 P labelled       2.7.0      2020-09-21 [?] CRAN (R 4.0.3)                      
 P lattice        0.20-41    2020-04-02 [?] CRAN (R 4.0.2)                      
 P lifecycle      0.2.0      2020-03-06 [?] CRAN (R 4.0.2)                      
 P listenv        0.8.0      2019-12-05 [?] CRAN (R 4.0.2)                      
 P lubridate      1.7.9      2020-06-08 [?] CRAN (R 4.0.2)                      
 P magrittr       1.5        2014-11-22 [?] CRAN (R 4.0.2)                      
 P Matrix         1.2-18     2019-11-27 [?] CRAN (R 4.0.2)                      
 P munsell        0.5.0      2018-06-12 [?] CRAN (R 4.0.2)                      
 P officer        0.3.14     2020-09-07 [?] CRAN (R 4.0.2)                      
 P openxlsx       4.1.5      2020-05-06 [?] CRAN (R 4.0.2)                      
 P pillar         1.4.6      2020-07-10 [?] CRAN (R 4.0.2)                      
 P pkgconfig      2.0.3      2019-09-22 [?] CRAN (R 4.0.2)                      
 P png            0.1-7      2013-12-03 [?] CRAN (R 4.0.0)                      
 P prismatic      0.2.0      2019-12-01 [?] CRAN (R 4.0.2)                      
 P processx       3.4.3      2020-07-05 [?] CRAN (R 4.0.2)                      
 P ps             1.3.4      2020-08-11 [?] CRAN (R 4.0.2)                      
 P purrr        * 0.3.4      2020-04-17 [?] CRAN (R 4.0.2)                      
 P R6             2.4.1      2019-11-12 [?] CRAN (R 4.0.2)                      
 P RColorBrewer   1.1-2      2014-12-07 [?] CRAN (R 4.0.0)                      
 P Rcpp           1.0.5      2020-07-06 [?] CRAN (R 4.0.2)                      
 P readxl         1.3.1      2019-03-13 [?] CRAN (R 4.0.2)                      
   renv           0.12.0     2020-08-28 [1] CRAN (R 4.0.2)                      
 P rio            0.5.16     2018-11-26 [?] CRAN (R 4.0.2)                      
   rlang          0.4.8      2020-10-08 [1] CRAN (R 4.0.3)                      
   rmarkdown      2.4.4      2020-10-15 [1] Github (rstudio/rmarkdown@64e8307)  
 P rprojroot      1.3-2      2018-01-03 [?] CRAN (R 4.0.2)                      
 P rsample        0.0.8      2020-09-23 [?] CRAN (R 4.0.2)                      
 P rstatix        0.6.0      2020-06-18 [?] CRAN (R 4.0.2)                      
 P rstudioapi     0.11       2020-02-07 [?] CRAN (R 4.0.2)                      
   sasdrive       0.0.1      2020-10-15 [1] Github (sagerx/sasdrive@a788cda)    
 P scales         1.1.1      2020-05-11 [?] CRAN (R 4.0.2)                      
 P sessioninfo    1.1.1      2018-11-05 [?] CRAN (R 4.0.2)                      
 P stringi        1.4.6      2020-02-17 [?] CRAN (R 4.0.0)                      
 P stringr        1.4.0      2019-02-10 [?] CRAN (R 4.0.2)                      
 P survival       3.1-12     2020-04-10 [?] CRAN (R 4.0.2)                      
 P survminer      0.4.8      2020-07-25 [?] CRAN (R 4.0.2)                      
 P survMisc       0.5.5      2018-07-05 [?] CRAN (R 4.0.2)                      
 P systemfonts    0.3.2      2020-09-29 [?] CRAN (R 4.0.2)                      
   tarchetypes  * 0.0.0.9000 2020-10-27 [1] Github (wlandau/tarchetypes@6d72150)
   targets      * 0.0.0.9002 2020-10-27 [1] Github (wlandau/targets@fd9b7a4)    
 P tibble         3.0.3      2020-07-10 [?] CRAN (R 4.0.2)                      
 P tidyr          1.1.2      2020-08-27 [?] CRAN (R 4.0.2)                      
 P tidyselect     1.1.0      2020-05-11 [?] CRAN (R 4.0.2)                      
 P utf8           1.1.4      2018-05-24 [?] CRAN (R 4.0.2)                      
 P uuid           0.1-4      2020-02-26 [?] CRAN (R 4.0.0)                      
 P vctrs          0.3.4      2020-08-29 [?] CRAN (R 4.0.2)                      
 P vip            0.2.2      2020-04-06 [?] CRAN (R 4.0.2)                      
 P visNetwork     2.0.9      2019-12-06 [?] CRAN (R 4.0.2)                      
   withr          2.3.0      2020-09-22 [1] CRAN (R 4.0.2)                      
 P xfun           0.18       2020-09-29 [?] CRAN (R 4.0.3)                      
 P xml2           1.3.2      2020-04-23 [?] CRAN (R 4.0.2)                      
 P xtable         1.8-4      2019-04-21 [?] CRAN (R 4.0.2)                      
 P yaml           2.2.1      2020-02-01 [?] CRAN (R 4.0.2)                      
 P zip            2.1.0      2020-08-10 [?] CRAN (R 4.0.2)                      
 P zoo            1.8-8      2020-05-02 [?] CRAN (R 4.0.2)                      

[1] C:/Users/jonathan.sidi/AppData/Local/Temp/RtmpYXoO9n/renv-system-library

 P -- Loaded and on-disk path mismatch.

traceback
 Stack trace:

 Process 20408:
 1. targets:::tar_make()
 2. targets:::callr_outer(targets_function = tar_make_inner, targets_arguments = ta ...
 3. targets:::trn(is.null(callr_function), callr_inner(target_script_path(),  ...
 4. base:::do.call(callr_function, prepare_callr_arguments(callr_function,  ...
 5. (function (func, args = list(), libpath = .libPaths(), repos = default_repos(), ...
 6. callr:::get_result(output = out, options)
 7. throw(newerr, parent = remerr[[2]])

 x callr subprocess failed: object 'paths' not found 

 Process 3532:
 19. (function (targets_script, targets_function, targets_arguments)  ...
 20. base:::source(targets_script)
 21. base:::withVisible(eval(ei, envir))
 22. base:::eval(ei, envir)
 23. base:::eval(ei, envir)
 24. targets:::tar_pipeline(tar_map(values = values, names = "domains",  ...
 25. targets:::pipeline_init(unlist(list(...), recursive = TRUE))
 26. targets:::pipeline_targets_init(targets)
 27. targets:::`%||%`(targets, list())
 28. base:::unlist(list(...), recursive = TRUE)
 29. tarchetypes:::tar_map(values = reports, names = "outputs", tar_render(period_re ...
 30. base:::unlist(list(...), recursive = TRUE)
 31. tarchetypes:::tar_render(period_report, paths, output_dir = "deliv/reports")
 32. tarchetypes:::assert_scalar(path, "tar_render() only takes one file at a time.")
 33. base:::.handleSimpleError(function (e)  ...
 34. h(simpleError(msg, call))

 x object 'paths' not found 


  • A reproducible example.
  • Session info, available through sessionInfo() or reprex(si = TRUE).
  • A stack trace from traceback() or rlang::trace_back().
  • The SHA-1 hash of the GitHub commit of tarchetypes currently installed. packageDescription("tarchetypes")$GithubSHA1 shows you this.

Use targets' newly-exported metaprogramming and error-handling utiltities

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • [n/a] For any problems you identify, post a minimal reproducible example like this one so the maintainer can troubleshoot. A reproducible example is:
    • [n/a] Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • [n/a] Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • [n/a] Readable: format your code according to the tidyverse style guide.

Description

To clean up some function cloning technical debt (related: https://ieeexplore.ieee.org/document/7069885) I just exported targets' error handling and metaprogramming utilites:
https://docs.ropensci.org/targets/reference/index.html#section-extending-targets. tarchetypes should use them. Need to wait until after version 0.2.1 is accepted on CRAN though.

Support pattern in `tar_map()`

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Currently, tar_map() is a helper function to build static branches, but it does not have a pattern argument. I think this could be added in order for a better combination of static and dynamic branches.

Replicate the change trigger from drake

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

From ropensci/targets#130 and ropensci/targets#131, cc @noamross and @jaredlander.

Archetype

tar_change(dataset, download_data(), change = remote_hash(dep1, dep2))

Equivalent to

list(
  tar_target(dataset_change, remote_hash(dep1, dep2), cue = tar_cue(mode = "always")),
  tar_target(
    dataset, {
      dataset_change # Enforces the dependency relationship
      download_data(),
    }
  )
)

Patterns/branching should be disabled.

Doesn't seem like much of a change to me, but user feedback suggests this is likely to simplify things for others.

using iteration in a tar_render

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

What trouble are you having?

I have a workflow with dynamic branching over endpoints in a model lets call it PARAMCD and have a parameterized Rmd that creates outputs based on params$paramcd. I would like to do something like tar_target(...,pattern = map(PARAM)), but it looks like tar_render isnt set up for this.

Is this correct?

Or am I looking at this function wrong?

here is the Rmd header

---
subtitle: "Predicting Something Amazing"
author: "Jonathan Sidi"
date: "`r Sys.Date()`"
output: 
  bookdown::html_document2: default
editor_options: 
  chunk_output_type: console
params:
  paramcd: 'TOTAL'
title: "`r sprintf('Fancy title: %s',params$paramcd)`"
---

for now i have the data loading set up kind of awkwardly, but if i could iterate over the length of the target PARAMCD i could drop the use of this

```{r}
data_objs <- c('paramcd','model_data','model_train','lab_panels')
logistic_objs <- c('logistic_fit','logistic_best_fit','logistic_last_fit','logistic_best_workflow')
rf_objs <- c('rf_fit','rf_best_fit','rf_last_fit')
binom_obj <- 'binom_ci'

withr::with_dir(
  rprojroot::find_root('_targets.R'),
  targets::tar_load(all_of(c(data_objs,logistic_objs,rf_objs,binom_obj)))
)
```
 
```{r}
this <- which(params$paramcd%in%paramcd)

model_data <- model_data[[this]]
model_train <- model_train[[this]]

logistic_fit <- logistic_fit[[this]]
logistic_best_fit <- logistic_best_fit[[this]]
logistic_last_fit <- logistic_last_fit[[this]]
logistic_best_workflow <- logistic_best_workflow[[this]]

rf_fit <- rf_fit[[this]]
rf_best_fit <- rf_best_fit[[this]]
rf_last_fit <- rf_last_fit[[this]]

binom_ci <- binom_ci[[this]]

```
session info
- Session info -------------------------------------------------------------------------------------
 setting  value                       
 version  R version 4.0.2 (2020-06-22)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/New_York            
 date     2020-12-11                  

- Packages -----------------------------------------------------------------------------------------
 !  package        * version     date       lib source                              
 P  assertthat       0.2.1       2019-03-21 [?] CRAN (R 4.0.2)                      
    backports        1.2.0       2020-11-02 [1] CRAN (R 4.0.3)                      
 P  binom            1.1-1       2014-01-02 [?] CRAN (R 4.0.3)                      
 P  bookdown         0.20        2020-06-23 [?] CRAN (R 4.0.2)                      
 P  broom          * 0.7.0       2020-07-09 [?] CRAN (R 4.0.2)                      
 P  callr            3.4.3       2020-03-28 [?] CRAN (R 4.0.2)                      
 P  checkmate        2.0.0       2020-02-06 [?] CRAN (R 4.0.2)                      
 P  class            7.3-17      2020-04-26 [?] CRAN (R 4.0.2)                      
 P  cli              2.1.0       2020-10-12 [?] CRAN (R 4.0.3)                      
 P  clipr            0.7.0       2019-07-23 [?] CRAN (R 4.0.2)                      
 P  clustermq        0.8.95.1    2020-07-13 [?] CRAN (R 4.0.3)                      
 P  codetools        0.2-16      2018-12-24 [?] CRAN (R 4.0.2)                      
 P  colorspace       1.4-1       2019-03-18 [?] CRAN (R 4.0.2)                      
    commonmark       1.7         2018-12-01 [1] CRAN (R 4.0.2)                      
 P  crayon           1.3.4       2017-09-16 [?] CRAN (R 4.0.2)                      
 P  data.table       1.13.0      2020-07-24 [?] CRAN (R 4.0.2)                      
 P  desc             1.2.0       2018-05-01 [?] CRAN (R 4.0.2)                      
 P  details          0.2.1       2020-09-08 [?] local                               
 P  dials          * 0.0.9       2020-09-16 [?] CRAN (R 4.0.2)                      
 P  DiceDesign       1.8-1       2019-07-31 [?] CRAN (R 4.0.2)                      
    digest           0.6.27      2020-10-24 [1] CRAN (R 4.0.3)                      
 P  distributional   0.2.1       2020-10-06 [?] CRAN (R 4.0.3)                      
 P  dplyr          * 1.0.2       2020-08-18 [?] CRAN (R 4.0.2)                      
 P  ellipsis         0.3.1       2020-05-15 [?] CRAN (R 4.0.2)                      
 P  evaluate         0.14        2019-05-28 [?] CRAN (R 4.0.2)                      
 P  fansi            0.4.1       2020-01-08 [?] CRAN (R 4.0.2)                      
 P  farver           2.0.3       2020-01-16 [?] CRAN (R 4.0.2)                      
 P  forcats          0.5.0       2020-03-01 [?] CRAN (R 4.0.2)                      
 P  foreach          1.5.0       2020-03-30 [?] CRAN (R 4.0.2)                      
 P  fs               1.5.0       2020-07-31 [?] CRAN (R 4.0.2)                      
 P  furrr            0.1.0       2018-05-16 [?] CRAN (R 4.0.2)                      
 P  future           1.18.0      2020-07-09 [?] CRAN (R 4.0.2)                      
 P  generics         0.1.0       2020-10-31 [?] CRAN (R 4.0.3)                      
 P  ggdist           2.3.0       2020-10-31 [?] CRAN (R 4.0.3)                      
 P  ggplot2        * 3.3.2       2020-06-19 [?] CRAN (R 4.0.2)                      
 P  glmnet           4.0-2       2020-06-16 [?] CRAN (R 4.0.2)                      
 P  globals          0.12.5      2019-12-07 [?] CRAN (R 4.0.0)                      
 P  glue             1.4.2       2020-08-27 [?] CRAN (R 4.0.2)                      
 P  gower            0.2.2       2020-06-23 [?] CRAN (R 4.0.2)                      
 P  GPfit            1.0-8       2019-02-08 [?] CRAN (R 4.0.2)                      
 P  gridExtra        2.3         2017-09-09 [?] CRAN (R 4.0.2)                      
 P  gt               0.2.2       2020-08-05 [?] CRAN (R 4.0.2)                      
 P  gtable           0.3.0       2019-03-25 [?] CRAN (R 4.0.2)                      
 P  gtsummary        1.3.5       2020-09-29 [?] CRAN (R 4.0.2)                      
 P  hardhat          0.1.4       2020-07-02 [?] CRAN (R 4.0.2)                      
 P  haven            2.3.1       2020-06-01 [?] CRAN (R 4.0.2)                      
 P  here             0.1         2017-05-28 [?] CRAN (R 4.0.2)                      
 P  hms              0.5.3       2020-01-08 [?] CRAN (R 4.0.2)                      
 P  htmltools        0.5.0       2020-06-16 [?] CRAN (R 4.0.2)                      
 P  htmlwidgets      1.5.1       2019-10-08 [?] CRAN (R 4.0.2)                      
 P  httr             1.4.2       2020-07-20 [?] CRAN (R 4.0.2)                      
 P  igraph           1.2.5       2020-03-19 [?] CRAN (R 4.0.2)                      
 P  infer          * 0.5.3       2020-07-14 [?] CRAN (R 4.0.2)                      
 P  ipred            0.9-9       2019-04-28 [?] CRAN (R 4.0.2)                      
 P  iterators        1.0.12      2019-07-26 [?] CRAN (R 4.0.2)                      
 P  janitor          2.0.1       2020-04-12 [?] CRAN (R 4.0.2)                      
 P  jsonlite         1.7.1       2020-09-07 [?] CRAN (R 4.0.2)                      
 P  knitr            1.30        2020-09-22 [?] CRAN (R 4.0.3)                      
 P  labeling         0.3         2014-08-23 [?] CRAN (R 4.0.0)                      
 P  labelled         2.7.0       2020-09-21 [?] CRAN (R 4.0.3)                      
 P  lattice          0.20-41     2020-04-02 [?] CRAN (R 4.0.2)                      
 P  lava             1.6.8       2020-09-26 [?] CRAN (R 4.0.2)                      
 P  lhs              1.1.1       2020-10-05 [?] CRAN (R 4.0.2)                      
 P  lifecycle        0.2.0       2020-03-06 [?] CRAN (R 4.0.2)                      
 P  listenv          0.8.0       2019-12-05 [?] CRAN (R 4.0.2)                      
 P  lubridate        1.7.9       2020-06-08 [?] CRAN (R 4.0.2)                      
 P  magrittr       * 1.5         2014-11-22 [?] CRAN (R 4.0.2)                      
 P  MASS             7.3-51.6    2020-04-26 [?] CRAN (R 4.0.2)                      
 P  Matrix           1.2-18      2019-11-27 [?] CRAN (R 4.0.2)                      
 P  modeldata      * 0.0.2       2020-06-22 [?] CRAN (R 4.0.2)                      
 P  munsell          0.5.0       2018-06-12 [?] CRAN (R 4.0.2)                      
 P  nnet             7.3-14      2020-04-26 [?] CRAN (R 4.0.2)                      
 P  parsnip        * 0.1.3       2020-08-04 [?] CRAN (R 4.0.2)                      
 P  patchwork      * 1.0.1       2020-06-22 [?] CRAN (R 4.0.2)                      
 P  pillar           1.4.6       2020-07-10 [?] CRAN (R 4.0.2)                      
 P  pkgconfig        2.0.3       2019-09-22 [?] CRAN (R 4.0.2)                      
 P  plyr             1.8.6       2020-03-03 [?] CRAN (R 4.0.2)                      
 P  png              0.1-7       2013-12-03 [?] CRAN (R 4.0.0)                      
 P  prettyunits      1.1.1       2020-01-24 [?] CRAN (R 4.0.2)                      
 P  pROC             1.16.2      2020-03-19 [?] CRAN (R 4.0.2)                      
 P  processx         3.4.3       2020-07-05 [?] CRAN (R 4.0.2)                      
 P  prodlim          2019.11.13  2019-11-17 [?] CRAN (R 4.0.2)                      
 P  ps               1.3.4       2020-08-11 [?] CRAN (R 4.0.2)                      
 P  purrr          * 0.3.4       2020-04-17 [?] CRAN (R 4.0.2)                      
 P  qs               0.23.3      2020-10-05 [?] CRAN (R 4.0.3)                      
    R6               2.5.0       2020-10-28 [1] CRAN (R 4.0.3)                      
 P  ranger           0.12.1      2020-01-10 [?] CRAN (R 4.0.2)                      
 P  RApiSerialize    0.1.0       2014-04-19 [?] CRAN (R 4.0.0)                      
 P  Rcpp             1.0.5       2020-07-06 [?] CRAN (R 4.0.2)                      
 PD RcppParallel     5.0.2       2020-06-24 [?] CRAN (R 4.0.2)                      
 P  readr            1.3.1       2018-12-21 [?] CRAN (R 4.0.2)                      
 P  recipes        * 0.1.15.9000 2020-11-13 [?] Github (tidymodels/recipes@591dcbf) 
 P  renv             0.12.2-13   2020-11-13 [?] Github (rstudio/renv@d022cac)       
    rlang            0.4.8       2020-10-08 [1] CRAN (R 4.0.3)                      
 P  rmarkdown        2.5         2020-10-21 [?] CRAN (R 4.0.2)                      
 P  rpart            4.1-15      2019-04-12 [?] CRAN (R 4.0.2)                      
 P  rprojroot        1.3-2       2018-01-03 [?] CRAN (R 4.0.2)                      
 P  rsample        * 0.0.8       2020-09-23 [?] CRAN (R 4.0.2)                      
 P  rsconnect        0.8.16      2019-12-13 [?] CRAN (R 4.0.2)                      
 P  rstudioapi       0.11        2020-02-07 [?] CRAN (R 4.0.2)                      
 P  sass             0.2.0       2020-03-18 [?] CRAN (R 4.0.2)                      
 P  scales         * 1.1.1       2020-05-11 [?] CRAN (R 4.0.2)                      
 P  sessioninfo      1.1.1       2018-11-05 [?] CRAN (R 4.0.2)                      
 P  shape            1.4.5       2020-09-13 [?] CRAN (R 4.0.2)                      
 P  snakecase        0.11.0      2019-05-25 [?] CRAN (R 4.0.2)                      
 P  stringfish       0.14.2      2020-09-05 [?] CRAN (R 4.0.3)                      
 P  stringi          1.4.6       2020-02-17 [?] CRAN (R 4.0.0)                      
 P  stringr          1.4.0       2019-02-10 [?] CRAN (R 4.0.2)                      
 P  survival         3.1-12      2020-04-10 [?] CRAN (R 4.0.2)                      
    tarchetypes      0.0.0.9000  2020-10-29 [1] Github (wlandau/tarchetypes@9f76f8f)
    targets        * 0.0.0.9002  2020-10-27 [1] Github (wlandau/targets@fd9b7a4)    
 P  tibble         * 3.0.4       2020-10-12 [?] CRAN (R 4.0.3)                      
 P  tidymodels     * 0.1.1       2020-07-14 [?] CRAN (R 4.0.2)                      
 P  tidyr          * 1.1.2       2020-08-27 [?] CRAN (R 4.0.2)                      
 P  tidyselect       1.1.0       2020-05-11 [?] CRAN (R 4.0.2)                      
 P  timeDate         3043.102    2018-02-21 [?] CRAN (R 4.0.2)                      
 P  tune           * 0.1.1.9001  2020-11-13 [?] Github (tidymodels/tune@3a9de60)    
 P  utf8             1.1.4       2018-05-24 [?] CRAN (R 4.0.2)                      
 P  vctrs            0.3.4       2020-08-29 [?] CRAN (R 4.0.2)                      
 P  vip              0.2.2       2020-04-06 [?] CRAN (R 4.0.2)                      
 P  viridisLite      0.3.0       2018-02-01 [?] CRAN (R 4.0.2)                      
 P  visNetwork       2.0.9       2019-12-06 [?] CRAN (R 4.0.2)                      
    withr            2.3.0       2020-09-22 [1] CRAN (R 4.0.2)                      
 P  workflows      * 0.2.1       2020-10-08 [?] CRAN (R 4.0.3)                      
 P  xfun             0.18        2020-09-29 [?] CRAN (R 4.0.3)                      
 P  xml2             1.3.2       2020-04-23 [?] CRAN (R 4.0.2)                      
 P  yaml             2.2.1       2020-02-01 [?] CRAN (R 4.0.2)                      
 P  yardstick      * 0.0.7       2020-07-13 [?] CRAN (R 4.0.2)                      

[1] C:/Users/jonathan.sidi/AppData/Local/Temp/Rtmp4qdVa2/renv-system-library

 P -- Loaded and on-disk path mismatch.
 D -- DLL MD5 mismatch, broken installation.

Thanks

Reproducible example

  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Desired result

What output or behavior do you want to see? Please be as specific as you can.

Diagnostic information

  • A reproducible example.
  • Session info, available through sessionInfo() or reprex(si = TRUE).
  • A stack trace from traceback() or rlang::trace_back().
  • The SHA-1 hash of the GitHub commit of tarchetypes currently installed. packageDescription("tarchetypes")$GithubSHA1 shows you this.

`tar_map()` does not interpret <integer64> values correctly

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is a genuine bug in tarchetypes and not a known limitation, usage error, or issue from another package that tarchetypes depends on. If you are unsure, please submit a discussion thread instead.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

When converting an class value, tar_map() does not handle correctly. See the reprex

Reproducible example

  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.
withr::local_package("targets")
tar_dir({ # tar_dir() runs code from a temporary directory.
    tar_script({
        tarchetypes::tar_map(
            values = data.frame(game_id = bit64::as.integer64(1)),
            list(
                tar_target(
                    name,
                    game_id
                )
            )
        )
    }, ask = FALSE)
    tar_manifest()
})
#> # A tibble: 1 × 3
#>   name                       command               pattern
#>   <chr>                      <chr>                 <chr>  
#> 1 name_4.94065645841247e.324 4.94065645841247e-324 <NA>

This is correct value

bit64::as.integer64(1) |>
    as.character()
#> [1] "1"

Created on 2022-05-19 by the reprex package (v2.0.1)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.0 (2022-04-22 ucrt)
#>  os       Windows 10 x64 (build 22000)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language en
#>  collate  Chinese (Simplified)_China.utf8
#>  ctype    Chinese (Simplified)_China.utf8
#>  tz       Asia/Taipei
#>  date     2022-05-19
#>  pandoc   2.17.1.1 @ C:/Program Files/RStudio/bin/quarto/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  backports     1.4.1   2021-12-13 [1] CRAN (R 4.2.0)
#>  base64url     1.4     2018-05-14 [1] CRAN (R 4.2.0)
#>  bit           4.0.4   2020-08-04 [1] CRAN (R 4.2.0)
#>  bit64         4.0.5   2020-08-30 [1] CRAN (R 4.2.0)
#>  callr         3.7.0   2021-04-20 [1] CRAN (R 4.2.0)
#>  cli           3.3.0   2022-04-25 [1] CRAN (R 4.2.0)
#>  codetools     0.2-18  2020-11-04 [2] CRAN (R 4.2.0)
#>  crayon        1.5.1   2022-03-26 [1] CRAN (R 4.2.0)
#>  data.table    1.14.2  2021-09-27 [1] CRAN (R 4.2.0)
#>  digest        0.6.29  2021-12-01 [1] CRAN (R 4.2.0)
#>  ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.2.0)
#>  evaluate      0.15    2022-02-18 [1] CRAN (R 4.2.0)
#>  fansi         1.0.3   2022-03-24 [1] CRAN (R 4.2.0)
#>  fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.2.0)
#>  fs            1.5.2   2021-12-08 [1] CRAN (R 4.2.0)
#>  glue          1.6.2   2022-02-24 [1] CRAN (R 4.2.0)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.2.0)
#>  htmltools     0.5.2   2021-08-25 [1] CRAN (R 4.2.0)
#>  igraph        1.3.1   2022-04-20 [1] CRAN (R 4.2.0)
#>  knitr         1.39    2022-04-26 [1] CRAN (R 4.2.0)
#>  lifecycle     1.0.1   2021-09-24 [1] CRAN (R 4.2.0)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
#>  pillar        1.7.0   2022-02-01 [1] CRAN (R 4.2.0)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.2.0)
#>  processx      3.5.3   2022-03-25 [1] CRAN (R 4.2.0)
#>  ps            1.7.0   2022-04-23 [1] CRAN (R 4.2.0)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.2.0)
#>  R.cache       0.15.0  2021-04-30 [1] CRAN (R 4.2.0)
#>  R.methodsS3   1.8.1   2020-08-26 [1] CRAN (R 4.2.0)
#>  R.oo          1.24.0  2020-08-26 [1] CRAN (R 4.2.0)
#>  R.utils       2.11.0  2021-09-26 [1] CRAN (R 4.2.0)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
#>  reprex        2.0.1   2021-08-05 [1] CRAN (R 4.2.0)
#>  rlang         1.0.2   2022-03-04 [1] CRAN (R 4.2.0)
#>  rmarkdown     2.14    2022-04-25 [1] CRAN (R 4.2.0)
#>  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.2.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.2.0)
#>  stringi       1.7.6   2021-11-29 [1] CRAN (R 4.2.0)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.2.0)
#>  styler        1.7.0   2022-03-13 [1] CRAN (R 4.2.0)
#>  targets     * 0.12.0  2022-04-19 [1] CRAN (R 4.2.0)
#>  tibble        3.1.7   2022-05-03 [1] CRAN (R 4.2.0)
#>  tidyselect    1.1.2   2022-02-21 [1] CRAN (R 4.2.0)
#>  utf8          1.2.2   2021-07-24 [1] CRAN (R 4.2.0)
#>  vctrs         0.4.1   2022-04-13 [1] CRAN (R 4.2.0)
#>  withr         2.5.0   2022-03-03 [1] CRAN (R 4.2.0)
#>  xfun          0.31    2022-05-10 [1] CRAN (R 4.2.0)
#>  yaml          2.3.5   2022-02-21 [1] CRAN (R 4.2.0)
#> 
#>  [1] C:/Users/liang/AppData/Local/R/win-library/4.2
#>  [2] C:/Program Files/R/R-4.2.0/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

New shorthand for branching over file stems

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

From ropensci/targets#136, it takes two targets to branch over a file stem.

list(
  tar_target(paths, c("a.csv", "b.csv")),
  tar_target(files, paths, pattern = map(paths), format = "file")
)

We should define a new target archetype to condense this down to one line.

tarchetypes::tar_files(files, c("a.csv", "b.csv"))

Batched replication

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Here in the targets-stan example, we batch simulated datasets in order to avoid egregiously large numbers of targets.

tar_pipeline(
  tar_target(
    index_batch,
    seq_len(2), # Change the number of simulation batches here.
    deployment = "local"
  ),
  tar_target(
    index_sim,
    seq_len(2), # Change the number of simulations per batch here.
    deployment = "local"
  ),
  tar_target(
    data_continuous,
    map_dfr(index_sim, ~simulate_data_continuous()),
    pattern = map(index_batch),
    format = "fst_tbl"
  ),
  ...
)

This technique is good practice in simulation studies, and I want to make it easier to use. I propose new target archetypes to help. The above could simplify down to this:

tar_pipeline(
  tar_rep_tbl(data_continuous, simulate_data_continuous(), batches = 2, reps_per_batch = 2)
)

Proposed archetypes:

Non-standard evaluation Standard evaluation
tar_rep_list() tar_rep_list_raw()
tar_rep_df() tar_rep_df_raw()
tar_rep_tbl() tar_rep_tbl_raw()

Target archetype that stores to S3

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Let's take @mdneuzerling's blog post on drake + S3 and turn it into a tar_s3() archeytpe. Related: #8.

`tar_render()` does not recognize non-ASCII character in `.Rmd` when using `options(encoding = "UTF-8")` on Windows

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation or usage error.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

It is very useful to set encoding as "UTF-8" in .Rprofile to deal with non-ASCII character on Windows. Currently it works well with targets functions but not tar_render(). tar_render() will not recognize non-ASCII characters in this case.

Reproducible example

  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.
withr::with_dir(
    tempdir(), {
        library(targets)
        library(tarchetypes)
        readr::write_lines(
            '---\ntitle: "标题"\noutput: html_document\n---\n\n这是有内容。',
            file = "test.Rmd"
        )
        readr::write_lines("options(encoding = \"UTF-8\")", file = ".Rprofile")
        tar_script(
            {
                library(targets)
                library(tarchetypes)
                list(tar_render(test, "test.Rmd"))
            }
        )
        tar_make()
    }
)
#> * run target test
#> [WARNING] This document format requires a nonempty <title> element.
#>   Defaulting to 'test.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> * end pipeline
#> Warning messages:
#> 1: In knitr::knit(path, output = connection, tangle = TRUE, quiet = TRUE) :
#>   The file "test.Rmd" must be encoded in UTF-8. Please see https://yihui.org/en/2018/11/biggest-regret-knitr/ for more info.
#> 2: The file "test.Rmd" must be encoded in UTF-8. Please see https://yihui.org/en/2018/11/biggest-regret-knitr/ for more info. 
#> 3: 输入链结'test.knit.md'内的输入不对 
#> 4: 输入链结'test.utf8.md'内的输入不对 
#> 5: 1 targets produced warnings. Run tar_meta(fields = warnings) for the messages. 

Created on 2021-02-26 by the reprex package (v1.0.0)

Session info
sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                         
#>  version  R version 4.0.4 (2021-02-15)  
#>  os       Windows 10 x64                
#>  system   x86_64, mingw32               
#>  ui       RTerm                         
#>  language (EN)                          
#>  collate  Chinese (Simplified)_China.936
#>  ctype    Chinese (Simplified)_China.936
#>  tz       Asia/Taipei                   
#>  date     2021-02-26                    
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.0)
#>  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.3)
#>  callr         3.5.1   2020-10-13 [1] CRAN (R 4.0.3)
#>  cli           2.3.0   2021-01-31 [1] CRAN (R 4.0.3)
#>  codetools     0.2-18  2020-11-04 [2] CRAN (R 4.0.4)
#>  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.0.3)
#>  data.table    1.13.6  2020-12-30 [1] CRAN (R 4.0.3)
#>  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.3)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.8     2019-03-20 [1] CRAN (R 4.0.0)
#>  hms           1.0.0   2021-01-13 [1] CRAN (R 4.0.3)
#>  htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.0.3)
#>  igraph        1.2.6   2020-10-06 [1] CRAN (R 4.0.2)
#>  knitr         1.30    2020-09-22 [1] CRAN (R 4.0.4)
#>  lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.0.4)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.3)
#>  pillar        1.4.7   2020-11-20 [1] CRAN (R 4.0.3)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.0)
#>  processx      3.4.5   2020-11-30 [1] CRAN (R 4.0.3)
#>  ps            1.5.0   2020-12-05 [1] CRAN (R 4.0.3)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.0.0)
#>  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.3)
#>  readr         1.4.0   2020-10-05 [1] CRAN (R 4.0.2)
#>  reprex        1.0.0   2021-01-27 [1] CRAN (R 4.0.3)
#>  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.3)
#>  rmarkdown     2.6     2020-12-14 [1] CRAN (R 4.0.4)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  styler        1.3.2   2020-02-23 [1] CRAN (R 4.0.2)
#>  tarchetypes * 0.0.4   2021-02-02 [1] CRAN (R 4.0.3)
#>  targets     * 0.1.0   2021-02-01 [1] CRAN (R 4.0.3)
#>  tibble        3.0.6   2021-01-29 [1] CRAN (R 4.0.3)
#>  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.0)
#>  vctrs         0.3.6   2020-12-17 [1] CRAN (R 4.0.3)
#>  withr         2.4.1   2021-01-26 [1] CRAN (R 4.0.3)
#>  xfun          0.21    2021-02-10 [1] CRAN (R 4.0.3)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] C:/Users/liang/Documents/R/win-library/4.0
#> [2] C:/Program Files/R/R-4.0.4/library

The warnings indicated the non-ASCII characters are not recognized. Is there a method to get rid of it? Or this function tar_render() should be updated?

Quarto targets - indicate code chunk that causes error

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Thanks for adding Quarto targets - I've been eagerly following the feature progress on this one! Ideally, tar_quarto() would produce a similar error message to tar_render() i.e. specify the lines/code chunk that produce an error (without needing to inspect tar_meta() for quick debugging). Currently it just specifies that there is an error in the pipeline. MWE below altering the vignette example with a spelling error in the code chunk.

targets::tar_dir({  # tar_dir() runs code from a temporary directory.
    # Unparameterized Quarto document:
    lines <- c(
        "---",
        "title: report.qmd source file",
        "output_format: html",
        "---",
        "Assume these lines are in report.qmd.",
        "```{r}",
        # Note the incorrect spelling to cause an error
        "targets::tar_red(data)",
        "```"
    )
    writeLines(lines, "report.qmd")
    # Include the report in a pipeline as follows.
    targets::tar_script({
        library(tarchetypes)
        list(
            tar_target(data, data.frame(x = seq_len(26), y = letters)),
            tar_quarto(report, path = "report.qmd")
        )
    }, ask = FALSE)

    # Run pipeline
    targets::tar_make()
})
#> • start target report
#> ✖ error target report
#> • end pipeline: 2.266 seconds
#> Error : ! System command 'quarto' failed
#> ✖ Problem with the pipeline.
#> ℹ Show errors: tar_meta(fields = error, complete_only = TRUE)
#> ℹ Learn more: https://books.ropensci.org/targets/debugging.html
#> Error:
#> ! problem with the pipeline.

Created on 2022-07-01 by the reprex package (v2.0.1)

A target archetype that runs on AWS through Metaflow

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

targets (and drake) are currently tied to traditional HPC (Slurm, SGE, etc.). That's enough for me and my team right now, but not for the increasing number of people like @MilesMcBain who rely on AWS and other cloud platforms. There is not yet much cloud support native to the R ecosystem, and since I don't use AWS for my own work, I am not prepared to do much at a low level.

Metaflow not only runs computation on the cloud and stores the results on S3, it also abstracts away the devops overhead that comes along with that, and it supports a sophisticated versioning system for code and data. I think we will gain a ton of power and convenience if we leverage Metaflow's rich and potentially complementary feature set in situations where targets needs the cloud.

Earlier, I proposed a targets-within-Metaflow approach, which I think would be useful for people with multiple {targets} pipelines in the same project. Here, I would like to explore the reverse: a target archetype that that runs some R code as a single AWS Metaflow step. Sketch:

# Archetype:
tar_metaflow(some_name, some_command(dep1, dep2), cpu = 4, memory = 10000)

# Equivalent to:
tar_target(some_name, {
  metaflow::metaflow("some_name") %>%
    metaflow::step(
      step = "some_name",
      metaflow::decorator("batch", cpu = 4, memory = 1000),
      r_function = function(self) self$some_name <- some_command(dep1, dep2)
    ) %>%
    metaflow::run()
  download_artifact_from_aws("some_name") # needs to be defined
})

cc @savingoyal (for when you return) @jasonge27, @bcgalvin.

R Markdown websites

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Re #46 (reply in thread)

tar_plan(): a drake-plan-like pipeline archetype

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

A plan is a pipeline archetype where most targets are simple and require no extra config. The proposed tar_plan() is like tar_pipeline(), but it will support the simpler target = command syntax in drake plan for named arguments. Unnamed arguments will assume the target is fully already specified with tar_target(). cc @MilesMcBain.

The following are equivalent:

tar_plan(
  tar_target(
    raw_data_file,
    "data/raw_data.csv",
    format = "file"
  ),
  raw_data, read_csv(raw_data_file, col_types = cols()),
  data = raw_data %>%
    mutate(Ozone = replace_na(Ozone, mean(Ozone, na.rm = TRUE))),
  hist = create_plot(data),
  fit = biglm(Ozone ~ Wind + Temp, data)),
  tar_render(report, "report.Rmd")
)
tar_pipeline(
  tar_target(
    raw_data_file,
    "data/raw_data.csv",
    format = "file"
  ),
  tar_target(
    raw_data,
    read_csv(raw_data_file, col_types = cols()),
    deployment = "local"
  ),
  tar_target(
    data,
    raw_data %>%
      mutate(Ozone = replace_na(Ozone, mean(Ozone, na.rm = TRUE)))
  ),
  tar_target(hist, create_plot(data)),
  tar_target(fit, biglm(Ozone ~ Wind + Temp, data)),
  tar_render(report, "report.Rmd")
)

Challenges

  1. Tidy evaluaton may be a little tricky. We might want to delay it until we convert plan-like targets into tar_target() calls.
  2. In what package do we support this DSL? I'm thinking tarchetypes since we don't want to bother targets with it and we don't want to make users load 3 different packages just to get the full interface. But we do need to be careful about how we define tarchetypes' scope. It can support DSLs too, but we need to be clear that it is more about overall shorthand than just target-by-target helpers.

Allow empty argument in tar_plan

targets is amazing and I'm so impressed with what you've done here. Congratulations on such a great package!

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Empty arguments are not allowed in tarchetypes::tar_plan, unlike its predecessor drake_plan.

Reproducible example

  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.
library(tarchetypes)
targets::tar_dir({
    targets::tar_script({
      library(tarchetypes)
      tar_plan(
        this = fails(),
      )
    })
    targets::tar_make()
  })
#> Error in FUN(X[[i]], ...) : argument is missing, with no default
#> Error: callr subprocess failed: argument is missing, with no default
#> Visit https://books.ropensci.org/targets/debugging.html for debugging advice.

Created on 2021-04-16 by the reprex package (v0.3.0)

Session info
devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.5 (2021-03-31)
#>  os       Ubuntu 18.04.5 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language en_US                       
#>  collate  en_NZ.UTF-8                 
#>  ctype    en_NZ.UTF-8                 
#>  tz       Pacific/Auckland            
#>  date     2021-04-16                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                               
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.1)                       
#>  callr         3.6.0      2021-03-28 [1] CRAN (R 4.0.5)                       
#>  cli           2.4.0      2021-04-05 [1] CRAN (R 4.0.5)                       
#>  codetools     0.2-18     2020-11-04 [4] CRAN (R 4.0.3)                       
#>  crayon        1.4.1      2021-02-08 [1] CRAN (R 4.0.5)                       
#>  data.table    1.14.0     2021-02-21 [1] CRAN (R 4.0.5)                       
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 4.0.1)                       
#>  devtools      2.3.2      2020-09-18 [1] CRAN (R 4.0.3)                       
#>  digest        0.6.27     2020-10-24 [1] CRAN (R 4.0.3)                       
#>  ellipsis      0.3.1      2020-05-15 [1] CRAN (R 4.0.1)                       
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 4.0.1)                       
#>  fansi         0.4.2      2021-01-15 [1] CRAN (R 4.0.5)                       
#>  fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.2)                       
#>  glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.3)                       
#>  highr         0.8        2019-03-20 [1] CRAN (R 4.0.1)                       
#>  htmltools     0.5.0      2020-06-16 [1] CRAN (R 4.0.1)                       
#>  igraph        1.2.6      2020-10-06 [1] CRAN (R 4.0.3)                       
#>  knitr         1.30       2020-09-22 [1] CRAN (R 4.0.3)                       
#>  lifecycle     1.0.0      2021-02-15 [1] CRAN (R 4.0.5)                       
#>  magrittr      2.0.1      2020-11-17 [1] CRAN (R 4.0.5)                       
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 4.0.1)                       
#>  pillar        1.6.0      2021-04-13 [1] CRAN (R 4.0.5)                       
#>  pkgbuild      1.1.0      2020-07-13 [1] CRAN (R 4.0.2)                       
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.1)                       
#>  pkgload       1.1.0      2020-05-29 [1] CRAN (R 4.0.1)                       
#>  prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.1)                       
#>  processx      3.5.1      2021-04-04 [1] CRAN (R 4.0.5)                       
#>  ps            1.6.0      2021-02-28 [1] CRAN (R 4.0.5)                       
#>  purrr         0.3.4      2020-04-17 [1] CRAN (R 4.0.1)                       
#>  R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.3)                       
#>  remotes       2.2.0      2020-07-21 [1] CRAN (R 4.0.2)                       
#>  rlang         0.4.10     2020-12-30 [1] CRAN (R 4.0.4)                       
#>  rmarkdown     2.5        2020-10-21 [1] CRAN (R 4.0.3)                       
#>  rprojroot     2.0.2      2020-11-15 [1] CRAN (R 4.0.3)                       
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.1)                       
#>  stringi       1.5.3      2020-09-09 [1] CRAN (R 4.0.3)                       
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 4.0.1)                       
#>  tarchetypes * 0.1.1.9000 2021-04-15 [1] Github (ropensci/tarchetypes@1246ccb)
#>  targets       0.3.1.9001 2021-04-15 [1] Github (ropensci/targets@7a76cf8)    
#>  testthat      3.0.0      2020-10-31 [1] CRAN (R 4.0.3)                       
#>  tibble        3.1.0      2021-02-25 [1] CRAN (R 4.0.5)                       
#>  tidyselect    1.1.0      2020-05-11 [1] CRAN (R 4.0.1)                       
#>  usethis       1.6.3      2020-09-17 [1] CRAN (R 4.0.3)                       
#>  utf8          1.2.1      2021-03-12 [1] CRAN (R 4.0.5)                       
#>  vctrs         0.3.7      2021-03-29 [1] CRAN (R 4.0.5)                       
#>  withr         2.4.1      2021-01-26 [1] CRAN (R 4.0.5)                       
#>  xfun          0.19       2020-10-30 [1] CRAN (R 4.0.3)                       
#>  yaml          2.2.1      2020-02-01 [1] CRAN (R 4.0.1)                       
#> 
#> [1] /home/users/bellk/R/4.0
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library
## Expected result

The result should run and be identical to the output removing the comma in means = colMeans(data),. This worked in drake:

library(drake)
drake_plan(this = worked(),)
#> # A tibble: 1 x 2
#>   target command   
#>   <chr>  <expr_lst>
#> 1 this   worked()

Created on 2021-04-16 by the reprex package (v0.3.0)

Diagnostic information

si included

tar_map() targets are sensitive to ordering when `names` is NULL

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • For any problems you identify, post a minimal reproducible example like this one so the maintainer can troubleshoot. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Default tar_map() target names are just numeric indexes when names is NULL. This means a target's validation status is sensitive to the ordering of values (see reprex below). I propose we assign hash suffixes when names is NULL. It is a painful change that will invalidate some pipelines, sorry to say, but it improves the stability of pipelines in the long run.

Reproducible example

  • For any problems you identify, post a minimal reproducible example so the maintainer can troubleshoot. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.
library(targets)

tar_script({
  library(tarchetypes)
  tar_map(
    values = list(x = c("a", "b")),
    names = NULL,
    tar_target(y, x)
  )
})

tar_make()
#> • start target y_1
#> • built target y_1
#> • start target y_2
#> • built target y_2
#> • end pipeline
# Switch the order of the values.
tar_script({
  library(tarchetypes)
  tar_map(
    values = list(x = c("b", "a")),
    names = NULL,
    tar_target(y, x)
  )
})

# Targets are renamed and thus invalidated.
tar_make()
#> • start target y_1
#> • built target y_1
#> • start target y_2
#> • built target y_2
#> • end pipeline

Created on 2021-08-03 by the reprex package (v2.0.0)

Convenience triggers for (time-based) target invalidation

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • [x If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

#41 shows how a time based invalidation could happen:

tarchetypes::tar_force(index, seq_len(4),
  force = if (file.exists("_targets/objects/index")) as_date(file.info("_targets/objects/index")$mtime) < (lubridate::today())
    )

I wonder if it would make sense to add some time-based convenience options to force to simplify achieving this and improve readability.

tarchetypes::tar_force(index, seq_len(4),
  force = "daily"
    )

tarchetypes::tar_force(index, seq_len(4),
  force = "weekly"
    )

Not sure what is happening behind the scenes and if this is possible in the first place but it would help a lot :)

`tar_force()` recomputes artifact target even though `force` condition evaluates to `FALSE`

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

I have a database target which I want to invalidate after a specific time (e.g. daily).

I've set a condition within tar_force() which should only evaluate to TRUE once a day.

While this works, I also see a new target in the console log that should not be there (index_change).
This targets runs on every run of tar_make() looks like an unwanted artifact.

Reproducible example

library(targets)
library(tarchetypes)
tar_script({
  library(targets)
  suppressMessages(library(lubridate))
  list(
    tarchetypes::tar_force(index, seq_len(4),
      force = if (file.exists("_targets/objects/index")) as_date(file.info("_targets/objects/index")$mtime) < (lubridate::today())
    )
  )
})
tar_make()
#> ● run target index_change
#> ● run target index
#> ● end pipeline
tar_make()
#> ● run target index_change
#> ✓ skip target index
#> ● end pipeline

Created on 2021-04-21 by the reprex package (v2.0.0)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                                      
#>  version  R version 4.0.5 Patched (2021-03-31 r80136)
#>  os       macOS Big Sur 10.16                        
#>  system   x86_64, darwin17.0                         
#>  ui       X11                                        
#>  language (EN)                                       
#>  collate  en_US.UTF-8                                
#>  ctype    en_US.UTF-8                                
#>  tz       Europe/Zurich                              
#>  date     2021-04-21                                 
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.4)
#>  callr         3.6.0   2021-03-28 [1] CRAN (R 4.0.5)
#>  cli           2.4.0   2021-04-05 [1] CRAN (R 4.0.5)
#>  codetools     0.2-18  2020-11-04 [2] CRAN (R 4.0.5)
#>  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.0.4)
#>  data.table    1.14.0  2021-02-21 [1] CRAN (R 4.0.4)
#>  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.4)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.4)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.4)
#>  fansi         0.4.2   2021-01-15 [1] CRAN (R 4.0.4)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.4)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.4)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.0.5)
#>  htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.0.4)
#>  igraph        1.2.6   2020-10-06 [1] CRAN (R 4.0.4)
#>  knitr         1.32    2021-04-14 [1] CRAN (R 4.0.5)
#>  lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.0.4)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.4)
#>  pillar        1.6.0   2021-04-13 [1] CRAN (R 4.0.5)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.4)
#>  processx      3.5.1   2021-04-04 [1] CRAN (R 4.0.5)
#>  ps            1.6.0   2021-02-28 [1] CRAN (R 4.0.5)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.0.4)
#>  R.cache       0.14.0  2019-12-06 [1] RSPM (R 4.0.4)
#>  R.methodsS3   1.8.1   2020-08-26 [1] CRAN (R 4.0.4)
#>  R.oo          1.24.0  2020-08-26 [1] RSPM (R 4.0.4)
#>  R.utils       2.10.1  2020-08-26 [1] RSPM (R 4.0.4)
#>  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.4)
#>  reprex        2.0.0   2021-04-02 [1] CRAN (R 4.0.5)
#>  rlang         0.4.10  2020-12-30 [1] CRAN (R 4.0.4)
#>  rmarkdown     2.7     2021-02-19 [1] CRAN (R 4.0.5)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.4)
#>  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.4)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.4)
#>  styler        1.4.1   2021-03-30 [1] CRAN (R 4.0.5)
#>  tarchetypes * 0.1.1   2021-03-28 [1] CRAN (R 4.0.5)
#>  targets     * 0.3.1   2021-03-28 [1] CRAN (R 4.0.5)
#>  tibble        3.1.1   2021-04-18 [1] CRAN (R 4.0.5)
#>  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.4)
#>  utf8          1.2.1   2021-03-12 [1] CRAN (R 4.0.5)
#>  vctrs         0.3.7   2021-03-29 [1] CRAN (R 4.0.5)
#>  withr         2.4.2   2021-04-18 [1] CRAN (R 4.0.5)
#>  xfun          0.22    2021-03-11 [1] CRAN (R 4.0.5)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.4)
#> 
#> [1] /Users/pjs/Library/R/4.0/library
#> [2] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

Expected result

No artifact target should appear.

tar_render_rep() with any number of output reports

Discussed in #96

Originally posted by ugoebel73 June 17, 2022
Our bioinformatics "targets" pipeline branches over the experimental contrasts of an input study (e.g., "treatmentA vs treatmentB") and is intended to render a separate output report for each contrast. Ideally any number of contrasts should be handled automatically.

tar_render_rep() with default parameters fails if there is only a single contrast. However it does pass if parameter "branches" is set to 2. This can be traced back to the as.integer(cut(seq_len(nrow(params)), breaks = batches)) call in function tarchetypes::tar_render_rep_run_params(): if nrow(params)=1, then this yields NA with batches=1, but 1 with batches=2.

I post this observation here because I am aware that tar_render_rep() is specifically meant for multiple output reports, so this behavior may not necessarily be considered a bug. However if tar_render_rep_run_params() could be made to handle the special case of a single report, this would simplify the handling of cases like ours, where the number of reports is not known in advance.

I have prepared a minimal example and could file a bug report if it is considered appropriate.

Parallelization in tar_render_rep()

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

When tar_render_rep() is run in parallel (tested with clustermq), intermediate knitr files with the same names (e.g. <Rmd_file>.knit.md etc.) are used for all workers, and, thus, removed before pandoc is run.

assignInNamespace("clean_tmpfiles", function() {}, ns = "rmarkdown") is used because of rstudio/rmarkdown#1632 (comment)

Reproducible example

targets::tar_dir({
  cat(getwd())

  writeLines(
    con = "tar_render_rep.Rmd",
    text = paste(
      "---",
      "title: 'tar_render_rep'",
      "params:",
      "  par: ''",
      "---",
      "```{r}",
      "cat(params$par)",
      "```",
      sep = "\n"
    )
  )

  targets::tar_script(ask = FALSE, {
    library(tarchetypes)
    library(magrittr)
    library(tidyverse)
    library(clustermq)

    assignInNamespace("clean_tmpfiles", function() {}, ns = "rmarkdown", envir = .GlobalEnv)
    options(clustermq.scheduler = "multicore")

    list(
      tar_target(df, tibble::tibble(par = LETTERS[1:20]) %>% dplyr::mutate(output_file = paste0("tar_render_rep_", 1:n(), ".html"))),
      tar_render_rep(df_rendered, "tar_render_rep.Rmd", params = df)
    )
  })

  targets::tar_make_clustermq(workers = 10)
})
#> /tmp/RtmpO2I6nF/targets_3d5831c9d32e── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
#> ✓ ggplot2 3.3.3     ✓ purrr   0.3.4
#> ✓ tibble  3.1.0     ✓ dplyr   1.0.5
#> ✓ tidyr   1.1.3     ✓ stringr 1.4.0
#> ✓ readr   1.4.0     ✓ forcats 0.5.1
#> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
#> x tidyr::extract()   masks magrittr::extract()
#> x dplyr::filter()    masks stats::filter()
#> x dplyr::lag()       masks stats::lag()
#> x purrr::set_names() masks magrittr::set_names()
#> * Option 'clustermq.scheduler' not set, defaulting to ‘LOCAL’
#> --- see: https://mschubert.github.io/clustermq/articles/userguide.html#configuration
#> ● run target df
#> ● run target df_rendered_params
#> ● run branch df_rendered_11894c68
#> ● run branch df_rendered_efc2f5be
#> ● run branch df_rendered_f06ee73c
#> ● run branch df_rendered_46ec1926
#> ● run branch df_rendered_1a3410ff
#> ● run branch df_rendered_fd246783
#> ● run branch df_rendered_915e8a36
#> ● run branch df_rendered_4b0aef2c
#> [WARNING] This document format requires a nonempty <title> element.
#>   Defaulting to 'tar_render_rep.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> [[[WWWAARARNRNINIINNGNG]G]  ] TTThhhiiisss   dddooocccuumumemenetnn tt fo frfmormaotr reqaumtiart ers  eraqe uqinuroienrsee msap  tany o nn<oetnmieptmlptetyy > < <tetilitetlmleee>n> t e.el
#> leem me ennDtte..f
#> 
#> a  u  lDtDieenffgaa uutlltot ii'nnggt  attro_o r 'e'tntadrae_rrr__rerenendpde.erur_t_rrfee8p'p. u.atusft 8ft'8 h'a esa  stt ihttehl eet .it
#> til  To specifyt ela. titlee.
#> ,
#>   use  '  tTTiot lsepo'e  csiipnfe ycm ieaft yat diaat tltaei ,to lrue s,- eu -m'tseiet 'tt[litWaldAeR'eN 'IiantN aGm ]e t itint lmee=t"aa.dd.a.t"a. 
#> oart a- -omrTe th-ai-dsma ettdaao dctauittmale etn=it"t .l.fe.o="r".m
#> .a.t." .r
#> equires a nonempty <title> element.
#>   Defaulting to 'tar_render_rep.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> ● run branch df_rendered_52425d62
#> [WARNING] This document format requires a nonempty <title> element.
#>   Defaulting to 'tar_render_rep.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> [WARNING] This document format requires a nonempty <title> element.
#>   Defaulting to 'tar_render_rep.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> pandoc: tar_render_rep.utf8.md: openBinaryFile: does not exist (No such file or directory)
#> ● run branch df_rendered_c4c44f78
#> x error branch df_rendered_52425d62
#> Warning in self$crew$finalize() :
#>   Unclean shutdown for PIDs: 16308, 16309, 16310, 16311, 16312, 16315, 16316, 16320, 16324, 16327
#> ● end pipeline
#> Error : cannot open the connection
#> In addition: Warning message:
#> 1 targets produced warnings. Run tar_meta(fields = warnings) for the messages. 
#> [WARNING] This document format requires a nonempty <title> element.
#>   Defaulting to 'tar_render_rep.utf8' as the title.
#>   To specify a title, use 'title' in metadata or --metadata title="...".
#> Error: callr subprocess failed: cannot open the connection
#> Visit https://books.ropensci.org/targets/debugging.html for debugging advice.

Created on 2021-04-06 by the reprex package (v1.0.0)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.4 (2021-02-15)
#>  os       Gentoo/Linux                
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language en_US.UTF-8                 
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Prague               
#>  date     2021-04-06                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version  date       lib source        
#>  P assertthat    0.2.1    2019-03-21 [?] CRAN (R 4.0.2)
#>  P backports     1.2.1    2020-12-09 [?] CRAN (R 4.0.2)
#>    callr         3.6.0    2021-03-28 [1] CRAN (R 4.0.4)
#>  P cli           2.3.1    2021-02-23 [?] CRAN (R 4.0.4)
#>    clustermq     0.8.95.1 2020-07-13 [1] CRAN (R 4.0.4)
#>  P codetools     0.2-18   2020-11-04 [?] CRAN (R 4.0.4)
#>  P crayon        1.4.1    2021-02-08 [?] CRAN (R 4.0.2)
#>  P data.table    1.14.0   2021-02-21 [?] CRAN (R 4.0.4)
#>  P digest        0.6.27   2020-10-24 [?] CRAN (R 4.0.2)
#>  P ellipsis      0.3.1    2020-05-15 [?] CRAN (R 4.0.2)
#>  P evaluate      0.14     2019-05-28 [?] CRAN (R 4.0.2)
#>  P fansi         0.4.2    2021-01-15 [?] CRAN (R 4.0.2)
#>  P fs            1.5.0    2020-07-31 [?] CRAN (R 4.0.2)
#>  P glue          1.4.2    2020-08-27 [?] CRAN (R 4.0.2)
#>  P highr         0.8      2019-03-20 [?] CRAN (R 4.0.2)
#>  P htmltools     0.5.1.1  2021-01-22 [?] CRAN (R 4.0.2)
#>  P igraph        1.2.6    2020-10-06 [?] CRAN (R 4.0.2)
#>  P knitr         1.31     2021-01-27 [?] CRAN (R 4.0.2)
#>  P lifecycle     1.0.0    2021-02-15 [?] CRAN (R 4.0.4)
#>  P magrittr      2.0.1    2020-11-17 [?] CRAN (R 4.0.2)
#>  P pillar        1.5.1    2021-03-05 [?] CRAN (R 4.0.4)
#>  P pkgconfig     2.0.3    2019-09-22 [?] CRAN (R 4.0.2)
#>  P processx      3.5.0    2021-03-23 [?] CRAN (R 4.0.4)
#>  P ps            1.6.0    2021-02-28 [?] CRAN (R 4.0.4)
#>  P purrr         0.3.4    2020-04-17 [?] CRAN (R 4.0.2)
#>  P R6            2.5.0    2020-10-28 [?] CRAN (R 4.0.2)
#>  P Rcpp          1.0.6    2021-01-15 [?] CRAN (R 4.0.2)
#>  P reprex        1.0.0    2021-01-27 [?] CRAN (R 4.0.2)
#>  P rlang         0.4.10   2020-12-30 [?] CRAN (R 4.0.2)
#>  P rmarkdown     2.7      2021-02-19 [?] CRAN (R 4.0.4)
#>  P sessioninfo   1.1.1    2018-11-05 [?] CRAN (R 4.0.2)
#>  P stringi       1.5.3    2020-09-09 [?] CRAN (R 4.0.2)
#>  P stringr       1.4.0    2019-02-10 [?] CRAN (R 4.0.2)
#>    styler        1.4.1    2021-03-30 [1] CRAN (R 4.0.4)
#>    targets       0.3.1    2021-03-28 [1] CRAN (R 4.0.4)
#>  P tibble        3.1.0    2021-02-25 [?] CRAN (R 4.0.4)
#>  P tidyselect    1.1.0    2020-05-11 [?] CRAN (R 4.0.2)
#>  P utf8          1.2.1    2021-03-12 [?] CRAN (R 4.0.4)
#>    vctrs         0.3.7    2021-03-29 [1] CRAN (R 4.0.4)
#>  P withr         2.4.1    2021-01-26 [?] CRAN (R 4.0.2)
#>  P xfun          0.22     2021-03-11 [?] CRAN (R 4.0.4)
#>  P yaml          2.2.1    2020-02-01 [?] CRAN (R 4.0.2)
#> 
#> [1] /mnt/raid/Users/novotnyj/projects/Vomastek/2021_01_canis_rnaseq/basenji/nf-core_reverse/R/renv/library/R-4.0/x86_64-pc-linux-gnu
#> [2] /tmp/RtmpBMvJUA/renv-system-library
#> [3] /usr/lib64/R/library
#> 
#>  P ── Loaded and on-disk path mismatch.

Expected result

Different intermediate_dir should be passed to rmarkdown::render() in order to avoid the concurrent removal of intermediate knitr files. My suggestion: for intermediate_dir use basenames of output_files or create random-named directories.

Diagnostic information

See Reproducible example


Thanks in advance for looking into this! For now, I have to run the pipeline sequentially with tar_make() to avoid this problem.

tar_render() doesn't detect dependencies correctly in some cases

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

tar_render() doesn't detect dependencies from calls to tar_load() in the Rmarkdown file in some cases.

It works properly if the name of the target in tar_load() is supplied as a symbol, but not as a character vector.

Reproducible example

_targets.R:

library(targets)
library(tarchetypes)

tar_option_set(packages = c("rmarkdown", "tidyverse"))

list(
  tar_target(
    sw_data,
    dplyr::starwars
  ),
  tar_render(
    sw_report,
    "sw_report.Rmd"
  )
)

sw_report.Rmd:

---
title: "test"
output: html_document
editor_options: 
  chunk_output_type: console
---

```{r setup, include=FALSE} 
knitr::opts_chunk$set(echo = TRUE)

# Works:
tar_load(sw_data)

# Doesn't work:
# tar_load(names = c("sw_data"))
```

A plot of weight vs. height for characters in Star Wars (guess who's the outlier?)

```{r sw, echo=FALSE}
ggplot(sw_data, aes(x = mass, y = height)) +
  geom_point()
```

Expected result

The Rmarkdown file renders correctly when tar_load(sw_data) is used:

targets::tar_make()
> tar_make()
● run target sw_data
● run target sw_report

But not when tar_load(names = c("sw_data")) is used:

> tar_make()
● run target sw_report
Quitting from lines 9-16 (sw_report.Rmd) 
x error target sw_report
Error : names arg of tar_load() must end up as character
Error: callr subprocess failed: names arg of tar_load() must end up as character
Type .Last.error.trace to see where the error occured

It should render correctly in both cases.

Diagnostic information

sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.5.0          stringr_1.4.0          dplyr_1.0.2            purrr_0.3.4           
 [5] readr_1.3.1            tidyr_1.1.2            tibble_3.0.4           ggplot2_3.3.2         
 [9] tidyverse_1.3.0        rmarkdown_2.6          tarchetypes_0.0.0.9002 targets_0.0.0.9004    

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.0  xfun_0.19         haven_2.3.1       colorspace_1.4-1  vctrs_0.3.6      
 [6] generics_0.0.2    htmltools_0.5.0   blob_1.2.1        rlang_0.4.10      pillar_1.4.7     
[11] glue_1.4.2        withr_2.3.0       DBI_1.1.0         dbplyr_1.4.4      readxl_1.3.1     
[16] modelr_0.1.8      lifecycle_0.2.0   cellranger_1.1.0  munsell_0.5.0     gtable_0.3.0     
[21] rvest_0.3.6       codetools_0.2-16  evaluate_0.14     knitr_1.30        callr_3.5.1      
[26] ps_1.5.0          fansi_0.4.1       broom_0.7.0       Rcpp_1.0.5        backports_1.1.10 
[31] scales_1.1.1      jsonlite_1.7.2    fs_1.5.0          hms_0.5.3         digest_0.6.27    
[36] stringi_1.5.3     processx_3.4.5    grid_4.0.2        cli_2.2.0         tools_4.0.2      
[41] magrittr_2.0.1    crayon_1.3.4      pkgconfig_2.0.3   ellipsis_0.3.1    xml2_1.3.2       
[46] data.table_1.13.6 reprex_0.3.0      lubridate_1.7.9   assertthat_0.2.1  httr_1.4.2       
[51] rstudioapi_0.11   R6_2.5.0          igraph_1.2.6      compiler_4.0.2   

Use labels from targets

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • Be considerate of the maintainer's time and make it as easy as possible to troubleshoot any problems you identify. Read here and here to learn about minimal reproducible examples. Format your code according to the tidyverse style guide to make it easier for others to read.

Description

https://github.com/wlandau/targets/labels. I wish I knew an automated way to copy over all these labels.

Set up Discussions

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • For any problem you identify, post a minimal reproducible example like this one so the maintainer can troubleshoot. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Setup for https://github.com/ropensci/tarchetypes/discussions:

Grouped data frame helpers

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

From @liutiming, ropensci/targets#329.

tar_render() does not work well with data.table

Reproducible example

lines <- c(
    "---",
    "title: report",
    "output_format: html_document",
    "---",
    "",
    "```{r}",
    "d <- targets::tar_read(data)",
    "d[, .(x = x + 1), .(y)]",
    "```"
)
writeLines(lines, "report.Rmd")
targets::tar_script({
    library(tarchetypes)
    tar_pipeline(
        tar_target(data, data.table::data.table(x = seq_len(26L), y = letters)),
        tar_render(report, "report.Rmd", quiet = TRUE)
    )
})

targets::tar_make(callr_function = NULL)
#> * run target data
#> * run target report
#> Quitting from lines 7-9 (report.Rmd)
#> x error target report
#> Error: could not find function "." .

Created on 2020-11-15 by the reprex package (v0.3.0)

Session Info

sessionInfo()
#> R version 4.0.2 (2020-06-22)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 19041)
#> 
#> Matrix products: default
#> 
#> locale:
#> [1] LC_COLLATE=English_United States.1252 
#> [2] LC_CTYPE=English_United States.1252   
#> [3] LC_MONETARY=English_United States.1252
#> [4] LC_NUMERIC=C                          
#> [5] LC_TIME=English_United States.1252    
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] nvimcom_0.9-102
#> 
#> loaded via a namespace (and not attached):
#>  [1] compiler_4.0.2  magrittr_1.5    htmltools_0.5.0 tools_4.0.2    
#>  [5] yaml_2.2.1      stringi_1.5.3   rmarkdown_2.4.6 highr_0.8      
#>  [9] knitr_1.30      stringr_1.4.0   xfun_0.18       digest_0.6.27  
#> [13] rlang_0.4.8     evaluate_0.14

tar_knit()

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Just like tar_render(), but for knitr::knit().

In tar_render() and friends, track the `*_files` directory for non-self-contained documents

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Check for the existence of *_files/ first. (It would be too brittle to inspect the source file and try to figure out if it is self-contained.)

tar_render renders distill but returns an error (invalidating the DAG)

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is a genuine bug in tarchetypes and not a known limitation, usage error, or issue from another package that tarchetypes depends on. If you are unsure, please submit a discussion thread instead.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

When try to render a {distill} article using tar_render, it will be correctly rendered, but it returns a strange error:

> targets::tar_make()
* start target report
x error target report
* end pipeline
Error : Distill articles cannot be previewed in this version of RStudio.
Please update to version 1.2.718 or higher at https://www.rstudio.com/products/rstudio/download/

Searching the web, I have discovered this discussion reporting the same issue (even if I can create the HTML report, which is rendered!)

The issue seems related (as stated in an answer to that SO thread) to a missing environmental variable. As suggested, I have also tried rmarkdown::render("distill_target_test.Rmd"), which correctly renders without any error (nor preview...)

My verions are:
OS: Windows 11 Pro for Workstations 21H2
R: 4.1.2
RStudio: 2021.09.0 Build 351
{targets}: 0.8.1.9000
{tarchetypes}: 0.3.2.9000
{distill}: 1.3

Reproducible example

  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Involving an environment, I have set-up a minimal renv-project at CorradoLanera/targets_distill. Just clone and run run.R.

So the main problem is that the error invalidates the network, resulting in an errored node. As I said, the report is rendered!

Expected result

Do not receive an error, or at least the option of safely ignoring it with no invalidation of the DAG's node for the report.

Diagnostic information

Reproducible example

https://github.com/CorradoLanera/targets_distill

Session Info

Session info
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
 [1] jquerylib_0.1.4    pillar_1.6.4       compiler_4.1.2    
 [4] remotes_2.4.1      targets_0.8.1.9000 tools_4.1.2       
 [7] digest_0.6.28      downlit_0.4.0      jsonlite_1.7.2    
[10] lubridate_1.8.0    evaluate_0.14      memoise_2.0.0     
[13] lifecycle_1.0.1    tibble_3.1.6       pkgconfig_2.0.3   
[16] rlang_0.4.12       igraph_1.2.8       cli_3.1.0         
[19] rstudioapi_0.13    distill_1.3        yaml_2.2.1        
[22] xfun_0.28          fastmap_1.1.0      stringr_1.4.0     
[25] withr_2.4.2        knitr_1.36         generics_0.1.1    
[28] vctrs_0.3.8        rprojroot_2.0.2    tidyselect_1.1.1  
[31] glue_1.5.0         data.table_1.14.2  R6_2.5.1          
[34] processx_3.5.2     fansi_0.5.0        bookdown_0.24     
[37] rmarkdown_2.11     callr_3.7.0        purrr_0.3.4       
[40] magrittr_2.0.1     codetools_0.2-18   ps_1.6.0          
[43] ellipsis_0.3.2     htmltools_0.5.2    renv_0.14.0       
[46] utf8_1.2.2         stringi_1.7.5      cachem_1.0.6      
[49] crayon_1.4.2  

Traceback

> rlang::last_error()
<error/tar_condition_run>
callr subprocess failed: Distill articles cannot be previewed in this version of RStudio.
Please update to version 1.2.718 or higher at https://www.rstudio.com/products/rstudio/download/

Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
Backtrace:
 1. targets::tar_make()
 2. targets:::callr_outer(...)
 3. base::tryCatch(...)
 4. base:::tryCatchList(expr, classes, parentenv, handlers)
 5. base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6. value[[3L]](cond)
 7. targets::tar_throw_run(...)
Run `rlang::last_trace()` to see the full context.
> rlang::trace_back()
x

SHA-1 of the GitHub commit of tarchetypes currently installed

4113a61

quarto targets

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

From @petrbouchal, ropensci/targets#813

Helper for downstream batched targets

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

tar_rep() establishes a batching scheme, but then it is up to the user to write custom code to iterate over reps within batches. If possible, it would be nice to have a helper automatically process the iterations of an individual batch. This sort of thing is so general that I am not sure it is possible Sketch:

# _targets.R
library(targets)
library(tarchetypes)
list(
  tar_rep(data1, simulate_data(), batches = 40, reps = 25),
  tar_rep(data2, simulate_data(), batches = 40, reps = 25),
  tar_rep(data3, simulate_data()),
  tar_target(analysis, tar_map_reps(analyze_data(data1, data2, data3)), pattern = map(data1, data2))
)

tar_map_reps() should automatically detect which targets are batched (e.g. data1 and data2 but not data3) and infer how to iterate over the batches given the data types (lists or data frames).

Hooks

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Automated hooks could let users prepend or wrap arbitrary R code around all the commands of all the targets in a target list. This could solve a multitude of issues, including:

  • Custom prework, e.g. conflicted setup (cc @MilesMcBain)
  • Restarts and timeouts.
  • Custom encryption for all non-file targets.

tar_hook_before()

# Regular _targets.R
library(targets)
source("R/functions.R")
list(
  tar_target(x, {
    set_up_conflicted_package()
    do_work()
  }),
  tar_target(y, {
    set_up_conflicted_package()
    do_more_work()
  })
)
# _targets.R with tar_hook_before()
library(targets)
library(magrittr)
source("R/functions.R")
targets <- list(
  tar_target(x, do_work()),
  tar_target(y, do_more_work())
) %>%
  tar_hook_before(set_up_conflicted_package)

tar_hook_inner() and tar_hook_outer()

# Regular _targets.R
library(targets)
source("R/functions.R")
list(
  tar_target(data, encrypt(get_data())),
  tar_target(analysis, encrypt(analyze(decrypt(data))))
)
# _targets.R using inner and outer hooks
library(targets)
library(magrittr)
source("R/functions.R")
targets <- list(
  tar_target(data, get_data()),
  tar_target(analysis, analyze(data))
) %>%
  tar_hook_inner(encrypt(.x)) %>%
  tar_hook_outer(decrypt(.x))

expose function for finding target reads in file

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Short version: Can tarchetypes::knitr_deps() or something similar be exposed to the user to provide a list of symbols of targets in an Rmd file?

I use bookdown to create notebooks. Different weeks get their own chapters (.Rmd files) and then they all get stitched together into a single html file. drake/targets handle the backend data construction and modeling and these analysis objects are loaded into the chapters. The whole book is regenerated if any of the dependencies are changed.

tar_knit() and tar_render() don't help me here because they are designed for single document knits and they make different assumptions. That's fine. The book is compiled with render_site() which is a different function than knit() or render(). So instead I provide a block of commands the will connect the book to Rmd files (in case prose changes) and to a list of targets (in case data changes). It looks like this:

  # build the notebook
  tar_target(
    notebook, {
      extra_targets = rlang::list2(!!! results)
      notebook_rmd_paths    # defined in a different target
      notebook_files        # defined in a different target
      rmarkdown::render_site("notebook", encoding = 'UTF-8')
      here::here("notebook/docs/notebook.html")   # target file
    },
    format = "file"
  ),

where results is a list of symbols of things that are tar_read() or tar_load() in each file. To get results, I had to make a function that crawls overs the Rmd files and pulls out tar_read() and tar_load() files. I'm asking whether this package might provide the function it uses for this purpose.

Shorthand for formats

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Let's think about target archetypes for formats. So instead of tar_target(file, get_path(), format = "file"), we could just write tar_file(). Likewise with tar_fst(), tar_fst_tbl(), tar_qs(), etc.

Target factory to rerun after a given delay

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Check the age of the target using tar_timestamp() and rerun the target if it is old enough (according to a user-supplied difftime object). This pretty much just requires a custom tar_cue() with mode equal to "always" or "thorough", depending on when the target last ran.

I don't think we can support dynamic branching in this one. Cues operate on all branches at once, but tar_timestamp() is branch-specific.

cc @noamross, @jaredlander, @petrbouchal

tar_knitr_deps() analyses call to knitr::read_chunk that is commented out

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

I am using tarchetypes::tar_knitr_deps() to analyse the dependencies in an Rmarkdown file and discovered that it analyses a call to knitr::read_chunk() that is commented out.

Reproducible example

  • Open an Rstudio session
  • Create and save file test_report.Rmd with contents:
---
title: "test report"
output: html_document
---

```{r}
# knitr::read_chunk("non_existent_file.R")
```
  • Execute tarchetaypes::tar_knitr_deps("test_report.Rmd") in the console. (This assumes the working directory is set appropriately.)
  • This generates the following error message, which indicates that tar_knitr_deps() tried to analyse the commented out call to knitr::read_chunk().
Quitting from lines 7-8 (test_report.Rmd) 
Error: Could not parse knitr report test_report.Rmd to detect dependencies: cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'non_existent_file.R': No such file or directory

Expected result

I expected tar_knitr_deps("non_existent_file.R") to return character(0) because there is no executable code to contain dependencies.

It may be that tar_knitr_deps() won't work with knitr::read_chunk(), but I expected all code comments to be effectively invisible to `tar_knitr_deps().

Diagnostic information

> traceback()
14: stop(tar_condition_validate(...))
13: throw_validate("Could not parse knitr report ", path, " to detect dependencies: ", 
        conditionMessage(e))
12: value[[3L]](cond)
11: tryCatchOne(expr, names, parentenv, handlers[[1L]])
10: tryCatchList(expr, classes, parentenv, handlers)
9: tryCatch(parse(text = knitr_lines(path)), error = function(e) {
       throw_validate("Could not parse knitr report ", path, " to detect dependencies: ", 
           conditionMessage(e))
   })
8: knitr_expr(path)
7: FUN(X[[i]], ...)
6: lapply(X = x, FUN = as_function(f), ...)
5: map(path, knitr_deps)
4: unlist(map(path, knitr_deps))
3: unique(unlist(map(path, knitr_deps)))
2: sort(unique(unlist(map(path, knitr_deps))))
1: tarchetypes::tar_knitr_deps("test_report.Rmd")
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.10

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C               LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8     LC_MONETARY=en_AU.UTF-8   
 [6] LC_MESSAGES=en_AU.UTF-8    LC_PAPER=en_AU.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] usethis_2.0.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6             pillar_1.6.0           compiler_4.0.3         later_1.2.0            workflowr_1.6.2        prettyunits_1.1.1     
 [7] sys_3.4                remotes_2.3.0          tarchetypes_0.2.0.9000 targets_0.4.2          tools_4.0.3            pkgbuild_1.2.0        
[13] digest_0.6.27          jsonlite_1.7.2         evaluate_0.14          lifecycle_1.0.0        tibble_3.1.1           pkgconfig_2.0.3       
[19] rlang_0.4.11           igraph_1.2.6           rstudioapi_0.13        cli_2.5.0              curl_4.3.1             yaml_2.2.1            
[25] xfun_0.22              stringr_1.4.0          withr_2.4.2            knitr_1.33             askpass_1.1            fs_1.5.0              
[31] vctrs_0.3.8            rprojroot_2.0.2        tidyselect_1.1.1       glue_1.4.2             data.table_1.14.0      R6_2.5.0              
[37] processx_3.5.2         fansi_0.4.2            rmarkdown_2.8          callr_3.7.0            purrr_0.3.4            magrittr_2.0.1        
[43] credentials_1.3.0      codetools_0.2-17       ps_1.6.0               promises_1.2.0.1       ellipsis_0.3.2         htmltools_0.5.1.1     
[49] httpuv_1.6.1           utf8_1.2.1             stringi_1.6.1          openssl_1.4.4          crayon_1.4.1          
> packageDescription("tarchetypes")$GithubSHA1
[1] "2ec501364a80618cd8b3e072148be8dd6ecaab4d"

Replicate the condition trigger from drake

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

From ropensci/targets#130 and ropensci/targets#131, cc @noamross and @jaredlander.

Archetypes

There are really two logical ways to think about the condition trigger:

  1. Run whenever the condition is TRUE, otherwise defer to other triggers.
  2. Suppress whenever the condition is FALSE, otherwise defer to other triggers.

Since these thought processes are different, the archetypes should also be different.

tar_suppress(dataset, download_data(), suppress = file_is_recent())
tar_force(dataset, download_data(), force = file_is_old())

I have not quite worked out how best to implement this, but I do think branching should be prohibited here.

user case of tar_skip / tar_change, deal with empty targets

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • [x If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Hi,
first of all thank a lot for the fabulous work you are doing, I was using slighly drake (and actually posted a relevant issue that you solved.
Now I moved to targets and I like it a lot. My workflow is to track new lectures/tutorials in dedicated folders and build a website for teaching. The part that is not working is how to deal with empty targets.

Here is a small example of something that is working, I am using tar_change to track the apparition of new files, then we compute md5sum and do stuff.

library(targets)
dir.create("tmp")
file.create("tmp/a")
#> [1] TRUE
writeLines("aa", con = "tmp/a")
file.create("tmp/b")
#> [1] TRUE
writeLines("bb", con = "tmp/b")

tar_script({
  library(tarchetypes)
  options(crayon.enabled = FALSE)
  tar_option_set(packages = c("tarchetypes", "tools"))
  tar_pipeline(
    tar_change(x, dir("tmp", full.names = TRUE),
               change = md5sum(dir("tmp", full.names = TRUE))),
    tar_file(y,
               setNames(x, md5sum(x)),
               pattern = map(x)),
    tar_target(z,
               readLines(y),
               pattern = map(y))
  )
})
tar_make()
#> ● run target x_change
#> ● run target x
#> ● run branch y_41a0680a
#> ● run branch y_d407878d
#> ● run branch z_cc0cfe46
#> ● run branch z_d699c360
tar_make()
#> ● run target x_change
#> ✓ skip target x
#> ✓ skip branch y_41a0680a
#> ✓ skip branch y_d407878d
#> ✓ skip branch z_cc0cfe46
#> ✓ skip branch z_d699c360
# adding a file
file.create("tmp/c")
#> [1] TRUE
tar_read(z)
#> [1] "aa" "bb"
# modify a file
writeLines("new_a", con = "tmp/a")
tar_make()
#> ● run target x_change
#> ● run target x
#> ● run branch y_41a0680a
#> ✓ skip branch y_d407878d
#> ● run branch y_435a25b4
#> ● run branch z_cc0cfe46
#> ✓ skip branch z_d699c360
#> ● run branch z_7ae6e1a8
tar_read(z)
#> [1] "new_a" "bb"

Created on 2020-10-26 by the reprex package (v0.3.0)

Reproducible example

  • Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Now comes the issue, if we don't have any file in the folder, I get callr subprocess failed: cannot branch over empty target.

To control for this, I tried to use tar_skip() that seems to be relevant for this user case:

library(targets)
dir.create("tmp")
tar_script({
  library(tarchetypes)
  options(crayon.enabled = FALSE)
  tar_option_set(packages = c("tarchetypes", "tools"))
  tar_pipeline(
    tar_skip(files,
             dir("tmp", full.names = TRUE),
             skip = !length(dir("tmp"))),
    tar_change(x,
               files,
               change = md5sum(dir("tmp", full.names = TRUE))),
    tar_file(y,
             {
               setNames(x, md5sum(x))
              },
             pattern = map(x)),
    tar_target(z,
               readLines(y),
               pattern = map(y))
  )
})
tar_make()
#> ● run target files
#> ● cancel target files
#> ● run target x_change
#> ● run target x
#> Error : cannot branch over empty target (x)
#> Error: callr subprocess failed: cannot branch over empty target (x)

Created on 2020-10-26 by the reprex package (v0.3.0)

Desired result

I can see that the rule files was indeed cancel as expected. But it didn't prevent the following steps to be processed.
I am a little lost now for preventing dependent branches to be performed.
Many thanks in advance for your kind help

Diagnostic information

Created on 2020-10-26 by the reprex package (v0.3.0)

Session info
sessionInfo()
#> R version 4.0.3 (2020-10-10)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 20.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices datasets  utils     methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] compiler_4.0.3  magrittr_1.5    htmltools_0.5.0 tools_4.0.3    
#>  [5] yaml_2.2.1      stringi_1.5.3   rmarkdown_2.5   highr_0.8      
#>  [9] knitr_1.30      stringr_1.4.0   xfun_0.18       digest_0.6.27  
#> [13] rlang_0.4.8     renv_0.12.0-30  evaluate_0.14
  • A stack trace from traceback() or rlang::trace_back().
  • The SHA-1 hash of the GitHub commit of tarchetypes currently installed. packageDescription("tarchetypes")$GithubSHA1 shows you this.

tarchetypes shasum commit: 585bce24595d672537d2fe19c4540350b81a675e

tar_plan example in README.Rmd

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • For any problem you identify, post a minimal reproducible example so the maintainer can troubleshoot. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

In README.Rmd, you reference tar_plan(), but the accompanying example doesn't use tar_plan(); rather, it uses tar_pipeline(). I think that's a mistake?

I think you could also change the first target to a tar_file(), if you wanted to?

Thanks for a great package, and all the contributed talks lately!

tar_age doesn't work with static branching

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is a genuine bug in tarchetypes and not a known limitation, usage error, or issue from another package that tarchetypes depends on. If you are unsure, please submit a discussion thread instead.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

When tar_age is used with static branching, targets are always run. The help only mentions the interaction with dynamic branching. I would have thought it would work because the generate targets are just normal tar_age targets?

Reproducible example

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
targets::tar_script({
  library(tarchetypes)
  list(
    tar_map(
      values = tibble::tibble(foo = c(1,2)),
    tarchetypes::tar_age(
      data,
      data.frame(x = seq_len(foo)),
      age = as.difftime(10, units = "secs")
    )
  )
  )
})
targets::tar_make()
Sys.sleep(0.5)
targets::tar_make()
})
#> * start target data_1
#> * built target data_1
#> * start target data_2
#> * built target data_2
#> * end pipeline
#> * start target data_1
#> * built target data_1
#> * start target data_2
#> * built target data_2
#> * end pipeline

Created on 2021-09-17 by the reprex package (v2.0.0)

  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Expected result

Targets should be skipped on the second run. Or the help should explain incompatibility with static branching.

Diagnostic information

R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)  
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C
[5] LC_TIME=English_Australia.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] assert_0.1.0        sf_1.0-2            qfesdata_0.1.2.9400 forcats_0.5.1        
 [5] stringr_1.4.0       dplyr_1.0.7         purrr_0.3.4         readr_2.0.1
 [9] tidyr_1.1.3         tibble_3.1.4        ggplot2_3.3.5       tidyverse_1.3.1      
[13] tarchetypes_0.2.0   targets_0.7.0.9001  dotenv_1.0.3        conflicted_1.0.4     

loaded via a namespace (and not attached):
 [1] fs_1.5.0           satellite_1.0.2    lubridate_1.7.10   webshot_0.5.2
 [5] httr_1.4.2         mapview_2.10.2     tools_4.1.0        backports_1.2.1
 [9] utf8_1.2.2         R6_2.5.1           KernSmooth_2.23-20 DBI_1.1.1
[13] colorspace_2.0-2   raster_3.4-13      withr_2.4.2        sp_1.4-5
[17] tidyselect_1.1.1   processx_3.5.2     leaflet_2.0.4.1    compiler_4.1.0
[21] leafem_0.1.6       cli_3.0.1          rvest_1.0.1        xml2_1.3.2
[25] scales_1.1.1       classInt_0.4-3     callr_3.7.0        proxy_0.4-26
[29] digest_0.6.27      rmarkdown_2.9      base64enc_0.1-3    pkgconfig_2.0.3
[33] htmltools_0.5.1.1  styler_1.4.1       highr_0.9          dbplyr_2.1.1
[37] fastmap_1.1.0      htmlwidgets_1.5.3  rlang_0.4.11       readxl_1.3.1      
[41] rstudioapi_0.13    generics_0.1.0     jsonlite_1.7.2     crosstalk_1.1.1
[45] magrittr_2.0.1     Rcpp_1.0.7         munsell_0.5.0      fansi_0.5.0
[49] clipr_0.7.1        lifecycle_1.0.0    stringi_1.7.4      yaml_2.2.1
[53] grid_4.1.0         crayon_1.4.1       lattice_0.20-44    haven_2.4.1
[57] hms_1.1.0          knitr_1.33         ps_1.6.0           pillar_1.6.2
[61] igraph_1.2.6       codetools_0.2-18   stats4_4.1.0       reprex_2.0.0
[65] glue_1.4.2         evaluate_0.14      rmdocs_0.1.0       data.table_1.14.0       
[69] modelr_0.1.8       png_0.1-7          vctrs_0.3.8        tzdb_0.1.2
[73] cellranger_1.1.0   gtable_0.3.0       assertthat_0.2.1   cachem_1.0.5
[77] xfun_0.24          broom_0.7.7        paint_0.1.1.9000   e1071_1.7-8
[81] class_7.3-19       units_0.7-2        ellipsis_0.3.2     using_0.4.0

I'm on a recent dev version of {targets}

packageDescription("targets")$GithubSHA1
[1] "d972f61d67028d2db795c9d72f6c3e18e7cabef2"
  • A reproducible example.
  • Session info, available through sessionInfo() or reprex(si = TRUE).
  • A stack trace from traceback() or rlang::trace_back().
  • The SHA-1 hash of the GitHub commit of tarchetypes currently installed. packageDescription("tarchetypes")$GithubSHA1 shows you this.

Batching in tar_files_input()

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Set the number of batches and partition evenly. Related: https://wlandau.github.io/targets-manual/dynamic.html#batching

tar_plan() now returns a target list instead of a pipeline object

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

tar_plan() now returns a target list instead of a pipeline object. As of ropensci/targets#254, _targets.R files should end with lists of targets instead of pipeline objects. tar_pipeline() and tar_bind() still work but are now deprecated. The return value of tar_plan() has changed but should still work for most use cases. Logging this issue for reference.

`tar_select_targets()` returns incorrect targets when they are not in alphabetical order

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is a genuine bug in tarchetypes and not a known limitation, usage error, or issue from another package that tarchetypes depends on. If you are unsure, please submit a discussion thread instead.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

Thanks for the great package ecosystem! Sorry if I am misunderstanding anything here.

tar_select_targets() returns incorrect targets when they are not in alphabetical order. In the reprex below, I slightly modified the example in the tar_select_targets() help file, to change the order of targets within the targets list, and also make the returned values a bit more obvious. Notice how in the output the returned "y1" target is actually the "x" target (see names and command).

I think that the issue comes from the sort() function in this line:

names(targets) <- sort(map_chr(targets, ~.x$settings$name))

Reproducible example

library(tarchetypes)

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
  targets <- list(
    list(
      targets::tar_target(y1, "y1_val"),
      targets::tar_target(x, "x_val")
    ),
    targets::tar_target(y2, "y2_val"),
    targets::tar_target(z, "z_val")
  )
  tar_select_targets(targets, starts_with("y"), contains("z"))
})
#> $y1
#> <tar_stem> 
#>   name: x 
#>   command:
#>     "x_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL
#> $y2
#> <tar_stem> 
#>   name: y2 
#>   command:
#>     "y2_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL
#> $z
#> <tar_stem> 
#>   name: z 
#>   command:
#>     "z_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL

Created on 2022-04-07 by the reprex package (v2.0.0)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Pop!_OS 20.04 LTS           
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language en_US:en                    
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/Chicago             
#>  date     2022-04-07                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib source                               
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.1.0)                       
#>  backports     1.4.0      2021-11-23 [1] RSPM (R 4.1.0)                       
#>  base64url     1.4        2018-05-14 [1] RSPM (R 4.1.0)                       
#>  callr         3.7.0      2021-04-20 [1] CRAN (R 4.1.0)                       
#>  cli           3.1.0      2021-10-27 [1] CRAN (R 4.1.1)                       
#>  codetools     0.2-18     2020-11-04 [4] CRAN (R 4.0.3)                       
#>  crayon        1.4.2      2021-10-29 [1] CRAN (R 4.1.1)                       
#>  data.table    1.14.2     2021-09-27 [1] RSPM (R 4.1.0)                       
#>  DBI           1.1.1      2021-01-15 [1] CRAN (R 4.1.0)                       
#>  digest        0.6.29     2021-12-01 [1] RSPM (R 4.1.0)                       
#>  dplyr         1.0.8      2022-02-08 [1] RSPM (R 4.1.0)                       
#>  ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.1.0)                       
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 4.1.0)                       
#>  fansi         0.5.0      2021-05-25 [1] CRAN (R 4.1.0)                       
#>  fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.1.0)                       
#>  fs            1.5.2      2021-12-08 [1] RSPM (R 4.1.0)                       
#>  generics      0.1.1      2021-10-25 [1] RSPM (R 4.1.0)                       
#>  glue          1.6.2      2022-02-24 [1] RSPM (R 4.1.0)                       
#>  highr         0.9        2021-04-16 [1] CRAN (R 4.1.0)                       
#>  htmltools     0.5.2      2021-08-25 [1] CRAN (R 4.1.1)                       
#>  igraph        1.2.6      2020-10-06 [1] CRAN (R 4.1.0)                       
#>  knitr         1.36       2021-09-29 [1] RSPM (R 4.1.0)                       
#>  lifecycle     1.0.1      2021-09-24 [1] CRAN (R 4.1.1)                       
#>  magrittr      2.0.2      2022-01-26 [1] RSPM (R 4.1.0)                       
#>  pillar        1.6.4      2021-10-18 [1] RSPM (R 4.1.0)                       
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.1.0)                       
#>  processx      3.5.2      2021-04-30 [1] CRAN (R 4.1.0)                       
#>  ps            1.6.0      2021-02-28 [1] CRAN (R 4.1.0)                       
#>  purrr         0.3.4      2020-04-17 [1] CRAN (R 4.1.0)                       
#>  R6            2.5.1      2021-08-19 [1] CRAN (R 4.1.1)                       
#>  reprex        2.0.0      2021-04-02 [1] CRAN (R 4.1.0)                       
#>  rlang         1.0.1      2022-02-03 [1] RSPM (R 4.1.0)                       
#>  rmarkdown     2.10       2021-08-06 [1] CRAN (R 4.1.1)                       
#>  rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.1.0)                       
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.1.1)                       
#>  stringi       1.7.6      2021-11-29 [1] RSPM (R 4.1.0)                       
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 4.1.0)                       
#>  styler        1.4.1      2021-03-30 [1] CRAN (R 4.1.0)                       
#>  tarchetypes * 0.5.0.9000 2022-04-07 [1] Github (ropensci/tarchetypes@d8bded8)
#>  targets       0.10.0     2022-01-07 [1] RSPM (R 4.1.0)                       
#>  tibble        3.1.6      2021-11-07 [1] RSPM (R 4.1.0)                       
#>  tidyselect    1.1.1      2021-04-30 [1] CRAN (R 4.1.0)                       
#>  utf8          1.2.2      2021-07-24 [1] CRAN (R 4.1.1)                       
#>  vctrs         0.3.8      2021-04-29 [1] CRAN (R 4.1.0)                       
#>  withr         2.4.3      2021-11-30 [1] RSPM (R 4.1.0)                       
#>  xfun          0.29       2021-12-14 [1] RSPM (R 4.1.0)                       
#>  yaml          2.2.1      2020-02-01 [1] CRAN (R 4.1.0)                       
#> 
#> [1] /home/andres/R/x86_64-pc-linux-gnu-library/4.1
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library

Expected result

The returned targets should not depend on how the targets are ordered in the input list. I think that simply removing sort() from the function fixes the issue (see reprex below), but I might be misunderstanding its role?

Thank you!

library(tarchetypes)

tar_select_targets2 <- function (targets, ...) 
{
  targets <- unlist(list(targets), recursive = TRUE)
  names(targets) <- purrr::map_chr(targets, ~.x$settings$name) # no sort()
  out <- tidyselect::eval_select(rlang::expr(c(...)), data = targets, 
                                 strict = FALSE)
  targets[names(out)]
}


targets::tar_dir({ # tar_dir() runs code from a temporary directory.
  targets <- list(
    list(
      targets::tar_target(y1, "y1_val"),
      targets::tar_target(x, "x_val")
    ),
    targets::tar_target(y2, "y2_val"),
    targets::tar_target(z, "z_val")
  )
  tar_select_targets2(targets, starts_with("y"), contains("z"))
})
#> $y1
#> <tar_stem> 
#>   name: y1 
#>   command:
#>     "y1_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL
#> $y2
#> <tar_stem> 
#>   name: y2 
#>   command:
#>     "y2_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL
#> $z
#> <tar_stem> 
#>   name: z 
#>   command:
#>     "z_val" 
#>   format: rds 
#>   iteration method: vector 
#>   error mode: stop 
#>   memory mode: persistent 
#>   storage mode: main 
#>   retrieval mode: main 
#>   deployment mode: worker 
#>   priority: 0 
#>   resources:
#>     list() 
#>   cue:
#>     mode: thorough
#>     command: TRUE
#>     depend: TRUE
#>     format: TRUE
#>     iteration: TRUE
#>     file: TRUE 
#>   packages:
#>     tarchetypes
#>     stats
#>     graphics
#>     grDevices
#>     utils
#>     datasets
#>     methods
#>     base 
#>   library:
#>     NULL

Created on 2022-04-07 by the reprex package (v2.0.0)

Diagnostic information

  • Session info in the reprex above
  • SHA-1 hash of the GitHub commit of tarchetypes currently installed: d8bded8

tar_download()

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

tar_download() could be a target factory that generates 2 targets: one format = "url" target to track a group of remote URLs with , and a downstream format = "file" target to download the URLs by running download.file() on each one.

Limitations:

  1. URL targets are generally far enough upstream that we can expect users to pre-specify fixed URLs and destination files. Dynamically resolved URLs would not be an issue, but then destination files would be tricky to specify safely, correctly, and dynamically.
  2. Because of the prespecification above, we would not allow dynamic branching (no pattern argument).

upstream targets in .Rmd not found when `pointblank` package loaded in packages.R

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

The dependencies to upstream targets in an .Rmd file are not found by tar_render() when the pointblank package is loaded in a packages.R file. With pointbank loaded, the .Rmd appears unconnected in the DAG from tar_visnetwork() and tar_make() fails with Error : names arg of tar_load() must end up as character

Reproducible example


author: scottericr
date: 2021-06-22
output:
reprex::reprex_document:
session_info: true
title: agile-pike_reprex.R

library(targets)
library(tarchetypes)

tar_dir({
  
  lines <- c(
    "---",
    "title: 'Untitled'",
    "output: html_document",
    "---",
    "",
    "```{r}",
    "tar_load(means)",
    "print(means)",
    "```"
  )
  
  writeLines(lines, "test.Rmd")
  
  pkg_lines <- c(
    "library(targets)",
    "library(tarchetypes)",
    "library(pointblank)"
  )
  
  writeLines(pkg_lines, "packages.R")
  
  tar_script({
    source("./packages.R")
    
    tar_plan(
      data = data.frame(x = seq_len(26)),
      means = colMeans(data),
      tar_render(test, "test.Rmd")
    )
  })
  
  tar_visnetwork()
  tar_make()
})
#> • start target test
#> Quitting from lines 7-9 (test.Rmd) 
#> x error target test
#> • end pipeline
#> Error : names arg of tar_load() must end up as character
#> Error: callr subprocess failed: names arg of tar_load() must end up as character
#> Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
packageDescription("tarchetypes")$Version
#> [1] "0.2.0"
#Using the CRAN version because I couldn't get the dev version to install (Error: object `tar_assert_chr` is not exported by `namespace::targets`)

Created on 2021-06-22 by the reprex package (v2.0.0)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS  10.16                
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2021-06-22                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.2)
#>  callr         3.7.0   2021-04-20 [1] CRAN (R 4.0.2)
#>  cli           2.5.0   2021-04-26 [1] CRAN (R 4.0.2)
#>  codetools     0.2-18  2020-11-04 [1] CRAN (R 4.0.2)
#>  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.0.2)
#>  data.table    1.14.0  2021-02-21 [1] CRAN (R 4.0.2)
#>  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.2)
#>  ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.0.2)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.5.0   2021-05-25 [1] CRAN (R 4.0.2)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.0.2)
#>  htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.0.2)
#>  htmlwidgets   1.5.3   2020-12-10 [1] CRAN (R 4.0.2)
#>  igraph        1.2.6   2020-10-06 [1] CRAN (R 4.0.2)
#>  jsonlite      1.7.2   2020-12-09 [1] CRAN (R 4.0.2)
#>  knitr         1.33    2021-04-24 [1] CRAN (R 4.0.2)
#>  lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.0.2)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.2)
#>  pillar        1.6.1   2021-05-16 [1] CRAN (R 4.0.2)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.0)
#>  processx      3.5.2   2021-04-30 [1] CRAN (R 4.0.2)
#>  ps            1.6.0   2021-02-28 [1] CRAN (R 4.0.2)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.0.0)
#>  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.2)
#>  reprex        2.0.0   2021-04-02 [1] CRAN (R 4.0.2)
#>  rlang         0.4.11  2021-04-30 [1] CRAN (R 4.0.2)
#>  rmarkdown     2.9     2021-06-15 [1] CRAN (R 4.0.2)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  stringi       1.6.2   2021-05-17 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  styler        1.4.1   2021-03-30 [1] CRAN (R 4.0.2)
#>  tarchetypes * 0.2.0   2021-05-11 [1] CRAN (R 4.0.2)
#>  targets     * 0.4.2   2021-04-30 [1] CRAN (R 4.0.2)
#>  tibble        3.1.2   2021-05-16 [1] CRAN (R 4.0.2)
#>  tidyselect    1.1.1   2021-04-30 [1] CRAN (R 4.0.2)
#>  utf8          1.2.1   2021-03-12 [1] CRAN (R 4.0.2)
#>  vctrs         0.3.8   2021-04-29 [1] CRAN (R 4.0.2)
#>  visNetwork    2.0.9   2019-12-06 [1] CRAN (R 4.0.2)
#>  withr         2.4.2   2021-04-18 [1] CRAN (R 4.0.2)
#>  xfun          0.24    2021-06-15 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

Expected result

tar_render() should see that test.Rmd loads the target means. Commenting out library(pointblank) in packages.R fixes the issue.

confusing names in tar_map with list of vectors

Starting to learn targets and migrating from drake so not sure if I'm doing things right but I cannot get the naming right when trying to run tar_map() over a list of vectors:

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
targets::tar_script({
  list(
      tarchetypes::tar_map(
        values = list(a = list(c("foo", "bar"), c("test1", "test2"))),
        names = c("first", "second"),
        targets::tar_target(b, paste(a, collapse = "-"))
      )
  )
})
targets::tar_manifest()
})

# A tibble: 2 x 3
  name  command                                            pattern
  <chr> <chr>                                              <chr>  
1 b_1   "paste(c(\"foo\", \"bar\"), collapse = \"-\")"     NA     
2 b_2   "paste(c(\"test1\", \"test2\"), collapse = \"-\")" NA     

Seems targets are assigned numbers instead of values from the vector of names. If I remove the names it calls the two targets "foo" and "bar" but each target still has the intended content. Would have expected the second target to be named "test1".

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
targets::tar_script({
  list(
      tarchetypes::tar_map(
        values = list(a = list(c("foo", "bar"), c("test1", "test2"))),
        targets::tar_target(b, paste(a, collapse = "-"))
      )
  )
})
targets::tar_manifest()
})
# A tibble: 2 x 3
  name  command                                            pattern
  <chr> <chr>                                              <chr>  
1 b_foo "paste(c(\"foo\", \"bar\"), collapse = \"-\")"     NA     
2 b_bar "paste(c(\"test1\", \"test2\"), collapse = \"-\")" NA     

Also, trying to add pattern = map(a) to the target produces Error : x must not be empty.

`tar_render()`ing a .Rmd with a `flextable()` breaks path to `reference_docx`

Prework

  • Read and agree to the code of conduct and contributing guidelines.
  • Confirm that your issue is most likely a genuine bug in tarchetypes and not a known limitation, a usage error, or a bug in another package that tarchetypes depends on.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
    • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
    • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
    • Readable: format your code according to the tidyverse style guide.

Description

I'm using tar_render() to knit a .Rmd report to Word using a reference template. Both the .Rmd and template.docx are in a doc/ folder. The .Rmd contains a call to flextable::flextable(). When I click the "knit" button, or use rmarkdown::render() everything works fine. When I use tar_make() I get the error: Error : could not find file 'template.docx'. This error goes away if I remove the call to flextable::flextable() or if I move the .Rmd file and template.docx to the root directory of the R project.

I suppose it might be a bug with flextable, but since it works fine with knit() or rmarkdown::render() I thought I'd post the issue here to see if you have any idea of a fix or workaround. I've tried specifying different paths to reference_docx and can't seem to fix it. Other paths in the YAML work fine like csl: and bibliography:

Reproducible example

library(targets)
library(tarchetypes)

tar_dir({
  lines <- c(
    "---",
    "title: 'template'",
    "output: word_document",
    "---",
    "",
    "#heading 1",
    "testing"
  )
  dir.create("doc/")
  writeLines(lines, "doc/template.Rmd")
  rmarkdown::render("doc/template.Rmd", output_format = "word_document")
  
  lines <- c(
    "---",
    "title: 'Untitled'",
    "output:",
    "  word_document:",
    "    reference_docx: 'template.docx'",
    "---",
    "",
    "```{r}",
    "flextable(mtcars)",
    "```"
  )
  
  writeLines(lines, "doc/test.Rmd")
  
  pkg_lines <- c(
    "library(targets)",
    "library(tarchetypes)",
    "library(flextable)"
  )
  
  writeLines(pkg_lines, "packages.R")
  
  tar_script({
    source("./packages.R")
    
    tar_plan(
      tar_render(test, "doc/test.Rmd")
    )
  })

  tar_make()
})
#> processing file: template.Rmd
#> output file: template.knit.md
#> /usr/local/bin/pandoc +RTS -K512m -RTS template.knit.md --to docx --from markdown+autolink_bare_uris+tex_math_single_backslash --output template.docx --lua-filter /Library/Frameworks/R.framework/Versions/4.0/Resources/library/rmarkdown/rmarkdown/lua/pagebreak.lua --highlight-style tango
#> 
#> Output created: template.docx
#> • start target test
#> Quitting from lines 9-10 (test.Rmd) 
#> x error target test
#> • end pipeline
#> Error : could not find file 'template.docx'
#> Error: callr subprocess failed: could not find file 'template.docx'
#> Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
packageDescription("tarchetypes")$Version
#> [1] "0.2.0"
#Using the CRAN version because I couldn't get the dev version to install (Error: object `tar_assert_chr` is not exported by `namespace::targets`)

Created on 2021-06-24 by the reprex package (v2.0.0)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS  10.16                
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2021-06-24                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  backports     1.2.1   2020-12-09 [1] CRAN (R 4.0.2)
#>  callr         3.7.0   2021-04-20 [1] CRAN (R 4.0.2)
#>  cli           2.5.0   2021-04-26 [1] CRAN (R 4.0.2)
#>  codetools     0.2-18  2020-11-04 [1] CRAN (R 4.0.2)
#>  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.0.2)
#>  data.table    1.14.0  2021-02-21 [1] CRAN (R 4.0.2)
#>  digest        0.6.27  2020-10-24 [1] CRAN (R 4.0.2)
#>  ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.0.2)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.5.0   2021-05-25 [1] CRAN (R 4.0.2)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.0.2)
#>  htmltools     0.5.1.1 2021-01-22 [1] CRAN (R 4.0.2)
#>  igraph        1.2.6   2020-10-06 [1] CRAN (R 4.0.2)
#>  knitr         1.33    2021-04-24 [1] CRAN (R 4.0.2)
#>  lifecycle     1.0.0   2021-02-15 [1] CRAN (R 4.0.2)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.0.2)
#>  pillar        1.6.1   2021-05-16 [1] CRAN (R 4.0.2)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.0)
#>  processx      3.5.2   2021-04-30 [1] CRAN (R 4.0.2)
#>  ps            1.6.0   2021-02-28 [1] CRAN (R 4.0.2)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.0.0)
#>  R6            2.5.0   2020-10-28 [1] CRAN (R 4.0.2)
#>  reprex        2.0.0   2021-04-02 [1] CRAN (R 4.0.2)
#>  rlang         0.4.11  2021-04-30 [1] CRAN (R 4.0.2)
#>  rmarkdown     2.9     2021-06-15 [1] CRAN (R 4.0.2)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  stringi       1.6.2   2021-05-17 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  styler        1.4.1   2021-03-30 [1] CRAN (R 4.0.2)
#>  tarchetypes * 0.2.0   2021-05-11 [1] CRAN (R 4.0.2)
#>  targets     * 0.4.2   2021-04-30 [1] CRAN (R 4.0.2)
#>  tibble        3.1.2   2021-05-16 [1] CRAN (R 4.0.2)
#>  tidyselect    1.1.1   2021-04-30 [1] CRAN (R 4.0.2)
#>  utf8          1.2.1   2021-03-12 [1] CRAN (R 4.0.2)
#>  vctrs         0.3.8   2021-04-29 [1] CRAN (R 4.0.2)
#>  withr         2.4.2   2021-04-18 [1] CRAN (R 4.0.2)
#>  xfun          0.24    2021-06-15 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

Import targets from other pipelines

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

To make sub-projects easier, I am thinking about a new tar_import() target factory to import a target from another local pipeline. The idea is to depend on the data hashes in the _targets/meta/meta from the other pipeline, then tar_read() the targets from that other pipeline. tar_import() would copy the files, which duplicates storage, but is preferable because it allows the current pipeline to do some exploratory stuff even the other pipeline breaks, and it also lets the user set a different storage format etc. for the imported target in the new pipeline, as well as give the imported targets new names in the new pipeline.

This is similar to @nsheff's unitar package except tarchetypes::tar_import() will be able to handle dynamic branching (and optionally select branches to import) by mapping over the data hashes in the other pipeline's metadata.

Target factory for batched dynamic-within-static branching for data frames

Prework

  • I understand and agree to the code of conduct and contributing guidelines.
  • If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Batched dynamic-within-static branching for data frames comes up a lot. stantargets and jagstargets alone have several examples. I think we can create a target factory to generalize the idea.

RStudio addins

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

I think this is a better place for addins than {targets}, but as with #7, a well-defined scope is key. (Maybe all we need is a better package name and a couple adjustments to the docs?)

Some initial ideas:

  • Load target at cursor.
  • Write tar_target() at cursor.
  • Run the pipeline.
  • Visualize the graph.
  • Show outdated targets.
  • Run tar_progress().

cc @MilesMcBain

Static branching

Prework

  • I understand and agree to tarchetypes' code of conduct.
  • I understand and agree to tarchetypes' contributing guidelines.
  • New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Let's create new functions tar_map(), tar_cross(), and tar_combine() to replicate static branching from drake. Each should accept one or more existing tar_target() objects and produce a list of tar_target() objects. Example:

tar_map(
  tar_target(x, a),
  tar_target(y, a),
  args = list(a = c(1, 2, 3))
)

This should be equivalent to:

list(
  tar_target(x_1, 1),
  tar_target(x_2, 2),
  tar_target(x_3, 3),
  tar_target(y_1, 1),
  tar_target(y_2, 2),
  tar_target(y_3, 3)
)

It will take some extra work and care to preserve the branching settings of the original targets. Need some extra tests for that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.