rstudio/packrat

Packrat is a dependency management system for R

NOTE

Packrat has been soft-deprecated and is now superseded by renv.

While we will continue maintaining Packrat, all new development will focus on renv. If you're interested in switching to renv, you can use renv::migrate() to migrate a project from Packrat to renv.

---

packrat

Packrat is a dependency management system for R.

Use packrat to make your R projects more:

• Isolated: Installing a new or updated package for one project won't break your other projects, and vice versa. That's because packrat gives each project its own private package library.
• Portable: Easily transport your projects from one computer to another, even across different platforms. Packrat makes it easy to install the packages your project depends on.
• Reproducible: Packrat records the exact package versions you depend on, and ensures those exact versions are the ones that get installed wherever you go.

See the project page for more information, or join the discussion on the RStudio Community forums.

Read the release notes to learn what's new in Packrat.

Quick-start Guide

Start by installing Packrat:

install.packages("packrat")

Then, start a new R session at the base directory of your project and type:

packrat::init()

This will install Packrat, set up a private library to be used for this project, and then place you in packrat mode. While in packrat mode, calls to functions like install.packages and remove.packages will modify the private project library, rather than the user library.

When you want to manage the state of your private library, you can use the Packrat functions:

• packrat::snapshot(): Save the current state of your library.
• packrat::restore(): Restore the library state saved in the most recent snapshot.
• packrat::clean(): Remove unused packages from your library.

Share a Packrat project with bundle and unbundle:

• packrat::bundle(): Bundle a packrat project, for easy sharing.
• packrat::unbundle(): Unbundle a packrat project, generating a project directory with libraries restored from the most recent snapshot.

Navigate projects and set/get options with:

• packrat::on(), packrat::off(): Toggle packrat mode on and off, for navigating between projects within a single R session.
• packrat::get_opts, packrat::set_opts: Get/set project-specific settings.

Manage ad-hoc local repositories (note that these are a separate entity from CRAN-like repositories):

• packrat::set_opts(local.repos = ...) can be used to specify local repositories; that is, directories containing (unzipped) package sources.
• packrat::install_local() installs packages available in a local repository.

For example, suppose I have the (unzipped) package sources for digest located within the folder~/git/R/digest/. To install this package, you can use:

packrat::set_opts(local.repos = "~/git/R")
packrat::install_local("digest")

There are also utility functions for using and managing packages in the external / user library, and can be useful for leveraging packages in the user library that you might not want as project-specific dependencies, e.g. devtools, knitr, roxygen2:

• packrat::extlib(): Load an external package.
• packrat::with_extlib(): With an external package, evaluate an expression. The external package is loaded only for the duration of the evaluated expression, but note that there may be other side effects associated with the package's .onLoad, .onAttach and .onUnload calls that we may not be able to fully control.

Workflows

Packrat supports a set of common analytic workflows:

1. As-you-go: use packrat::init() to initialize packrat with your project, and use it to manage your project library while you develop your analysis. As you install and remove packages, you can use packrat::snapshot() and packrat::restore() to maintain the R packages in your project. For collaboration, you can either use your favourite version control system, or use packrat::bundle() to generate a bundled version of your project that collaborators can use with packrat::unbundle().

2. When-you're-done: take an existing or complete analysis (preferably collected within one directory), and call packrat::init() to immediately obtain R package sources for all packages used in your project, and snapshot that state so it can hence be preserved across time.

Setting up your own custom, CRAN-like repositories

Please view the set-up guide here for a simple walkthrough in how you might set up your own, local, custom CRAN repository.

packrat's Issues

Monitor other R-code containing files

We should also look for library calls in .Rmd, .Rnw files.

library.new and library.old files are not excluded from dependency detection

When we scan for dependencies, we ignore R code in the library folder. We also need to ignore R code in library.new and library.old -- these are very temporary directories and should generally be moved out of the way before it matters, but it's reasonably easy to hit a bug wherein they're still around.

status() should do something intelligent with package deps in .Library

Add local library to search path?

When doing package development in parallel to working on a packrat-controlled project, it could be helpful when not everything is controlled by packrat. Another example is devtools: The projects might not need it, but it might be needed to update installed packages, and hence must be accessible from within the packrat-controlled project.

The system library is still part of .libPaths() in a packrat project. How about an option in the packrat.lock that retains the search path?

Why doesn't packrat support running in packages?

I have gotten in the habit of using the R packaging system to organize projects. I put scripts in a folder such as vignettes, scripts, or inst/manuscript. I wanted to test out using packrat and encountered the "Packrat doesn't work when the packages." message from the bootstrap function. Is this to be read "Packrat is not intended to work on packages" or "Packrat does not currently support packages"? I read the documentation and did not find any mention of support for use with packages, and can not understand why this would be disabled by default.

If there is a reason for not using it in the package root directory, would you recommend that I bootstrap directories within the package (like the vignettes folder)?

Use a wrapper function over .libPaths

Since .libPaths will silently fail if the libraries passed do not exist, it can mask some later errors. It would be good to have our own setLibPaths function that throws an error if the library being set doesn't exist, so we can more easily catch / adapt code where it fails.

Full text search in RStudio finds entries from library

In a packrat project, trying to find a string in RStudio (Ctrl+H) will find all occurrences in any of the installed packages (from the library subdirectory). Possible solutions:

• library is moved out of the way into a "global" location (say, a "cache" path supplied by rappdirs)
• RStudio is taught to ignore library for searching (several options here, but off-topic to specify them)
• A combination: RStudio is changed to ignore hidden directories for searching, and library is renamed to .library.

Dependencies not installed in the correct order

We currently install packages in the order given by diff, but this doesn't guarantee that packages will be installed before their dependencies. This is fine when those dependencies are satisfied by another library on the path (i.e. when you're hydrating an app that uses libraries you already have), but not fine when you're hydrating an app that has complex dependencies among several packages you don't yet have.

snapshot() needs to do more

Currently, snapshot() does nothing more than update the lock file. It also needs to:

• Populate packrat.sources/ with up-to-date source packages
• Detect any packages that are installed in base but not the private library, and install them

Mavericks workaround for /usr/bin/gnutar issue

On Mavericks, /usr/bin/gnutar does not exist, but R is often configured to use it anyway. It would be possible for us to detect this state:

1. TAR is /usr/bin/gnutar
2. /usr/bin/gnutar does not exist
3. /usr/bin/tar does exist

and recover by setting TAR=/usr/bin/tar for the duration of the commands that may decompress archives.

See some discussion here: https://stat.ethz.ch/pipermail/r-sig-mac/2013-November/010450.html

Avoid restarting R after bootstrap, restore etc?

Is there a good reason not to just have these functions re-read in the relevant .Rprofile and .Renviron rather than make the user manually restart R?

Running:

source(".Rprofile")
readRenviron(".Renviron")

instead of restarting seems to work for me.

status() doesn't work when code and packrat changes are pulled at once

Repro:

1. make a change on github (or locally) that adds a new package, and code that depends on it (say, "fossil")

2. pull the change on a machine that does not have the package installed

3. from the packrat project on that machine, run packrat::status()

packrat::status()
Error in FUN(c("maps", "fossil", "packrat")[[2L]], ...) :
The package "fossil" is not installed in /Users/jmcphers/r-guest/packrat-screencast/library/x86_64-apple-darwin10.8.0/3.0.1

Local packages don't get updated unless version numbers change

Consider whether or not to .Renviron

The problem with using a project .Renviron is that it overrides the system .Renviron. This is probably good in some sense because it make the code more reproducible, but it also causes problems because my global .Renviron. includes

TMPDIR=/tmp

# Needed for 3.0.2 on Mavericks
TAR=/usr/bin/tar

Fetching sources for packrat (0.1.0) ... Error in untar2(tarfile, files, list, exdir) : unsupported entry type ‘g’

Error message when calling bootstrap on an empty project directory

Fetching sources for packrat (0.1.0) ... Error in untar2(tarfile, files, list, exdir) : unsupported entry type ‘g’

Running on R 2.15.2, Mac OS X 10.8.5

::restore() doesn't prompt for downgrades and removals

::snapshot() offers a prompt when you're about to downgrade or remove packages from Packrat, but ::restore() currently offers no such prompt.

Dependencies installed unnecessarily (Windows-only?)

Log from bootstrapping a project with lots of dependencies. Note that for the package ABCExtremes (and others), its dependencies are downloaded from CRAN even though those dependencies have already been installed in the private library.

packrat::bootstrap("~/r/pack-scenario")

Adding these packages to packrat:
_
ABCExtremes 1.0
bitops 1.0-6
colorspace 1.2-4
combinat 0.0-8
devtools 1.3.99
dichromat 2.0-0
digest 0.6.3
evaluate 0.5.1
ggplot2 0.9.3.1
gtable 0.1.2
httr 0.2
labeling 0.2
lattice 0.20-15
MASS 7.3-26
memoise 0.1
munsell 0.4.2
packrat 0.1.0
plyr 1.8
proto 0.3-10
RColorBrewer 1.0-5
RCurl 1.95-4.1
reshape2 1.2.2
scales 0.2.3
SpatialExtremes 2.0-0
stringr 0.6.2
survival 2.37-4
TTR 0.22-0
whisker 0.3-2
xts 0.9-7
zoo 1.7-10

Fetching sources for bitops (1.0-6) ... OK (CRAN current)
Fetching sources for colorspace (1.2-4) ... OK (CRAN current)
Fetching sources for combinat (0.0-8) ... OK (CRAN current)
Fetching sources for dichromat (2.0-0) ... OK (CRAN current)
Fetching sources for digest (0.6.3) ... OK (CRAN current)
Fetching sources for gtable (0.1.2) ... OK (CRAN current)
Fetching sources for labeling (0.2) ... OK (CRAN current)
Fetching sources for lattice (0.20-15) ... OK (CRAN archived)
Fetching sources for MASS (7.3-26) ... OK (CRAN archived)
Fetching sources for plyr (1.8) ... OK (CRAN current)
Fetching sources for proto (0.3-10) ... OK (CRAN current)
Fetching sources for RColorBrewer (1.0-5) ... OK (CRAN current)
Fetching sources for SpatialExtremes (2.0-0) ... OK (CRAN current)
Fetching sources for stringr (0.6.2) ... OK (CRAN current)
Fetching sources for survival (2.37-4) ... OK (CRAN current)
Fetching sources for whisker (0.3-2) ... OK (CRAN current)
Fetching sources for RCurl (1.95-4.1) ... OK (CRAN current)
Fetching sources for munsell (0.4.2) ... OK (CRAN current)
Fetching sources for memoise (0.1) ... OK (CRAN current)
Fetching sources for zoo (1.7-10) ... OK (CRAN current)
Fetching sources for ABCExtremes (1.0) ... OK (CRAN current)
Fetching sources for evaluate (0.5.1) ... OK (CRAN current)
Fetching sources for reshape2 (1.2.2) ... OK (CRAN current)
Fetching sources for httr (0.2) ... OK (CRAN current)
Fetching sources for scales (0.2.3) ... OK (CRAN current)
Fetching sources for xts (0.9-7) ... OK (CRAN current)
Fetching sources for devtools (1.3.99) ... OK (Github)
Fetching sources for ggplot2 (0.9.3.1) ... OK (CRAN current)
Fetching sources for TTR (0.22-0) ... OK (CRAN current)
Fetching sources for packrat (0.1.0) ... OK (Github)
Snapshot written to C:/Users/jmcphers/Documents/R/pack-scenario/packrat.lock
Warning: unable to access index for repository http://cran.rstudio.com, http://www.stats.ox.ac.uk/pub/RWin/bin/windows/contrib/3.0
Installing bitops (1.0-6) ... OK (copied local binary)
Installing colorspace (1.2-4) ... OK (copied local binary)
Installing combinat (0.0-8) ... OK (built source)
Installing dichromat (2.0-0) ... OK (copied local binary)
Installing digest (0.6.3) ... OK (copied local binary)
Installing gtable (0.1.2) ... OK (copied local binary)
Installing labeling (0.2) ... OK (copied local binary)
Installing lattice (0.20-15) ... OK (copied local binary)
Installing MASS (7.3-26) ... OK (copied local binary)
Installing plyr (1.8) ... OK (copied local binary)
Installing proto (0.3-10) ... OK (copied local binary)
Installing RColorBrewer (1.0-5) ... OK (copied local binary)
Installing SpatialExtremes (2.0-0) ... OK (built source)
Installing stringr (0.6.2) ... OK (copied local binary)
Installing survival (2.37-4) ... OK (copied local binary)
Installing whisker (0.3-2) ... OK (copied local binary)
Installing RCurl (1.95-4.1) ... OK (copied local binary)
Installing munsell (0.4.2) ... OK (copied local binary)
Installing memoise (0.1) ... OK (copied local binary)
Installing zoo (1.7-10) ... OK (copied local binary)
Installing ABCExtremes (1.0) ... Installing dependencies for ABCExtremes:
SpatialExtremes, combinat
Installing packages into ‘C:/Users/jmcphers/Documents/R/win-library/3.0’
(as ‘lib’ is unspecified)
trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.0/SpatialExtremes_2.0-0.zip'
Content type 'application/zip' length 1507803 bytes (1.4 Mb)
opened URL
downloaded 1.4 Mb

trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.0/combinat_0.0-8.zip'
Content type 'application/zip' length 28913 bytes (28 Kb)
opened URL
downloaded 28 Kb

package ‘SpatialExtremes’ successfully unpacked and MD5 sums checked
package ‘combinat’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
C:\tmp\RtmpcVA1aq\downloaded_packages
OK (built source)
Installing evaluate (0.5.1) ... OK (copied local binary)
Installing reshape2 (1.2.2) ... OK (copied local binary)
Installing httr (0.2) ... OK (copied local binary)
Installing scales (0.2.3) ... OK (copied local binary)
Installing xts (0.9-7) ... OK (built source)
Installing devtools (1.3.99) ... OK (copied local binary)
Installing ggplot2 (0.9.3.1) ... OK (copied local binary)
Installing TTR (0.22-0) ... Installing dependencies for TTR:
xts, xts
Installing packages into ‘C:/Users/jmcphers/Documents/R/win-library/3.0’
(as ‘lib’ is unspecified)
trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.0/xts_0.9-7.zip'
Content type 'application/zip' length 661410 bytes (645 Kb)
opened URL
downloaded 645 Kb

package ‘xts’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
C:\tmp\RtmpcVA1aq\downloaded_packages
OK (built source)
Installing packrat (0.1.0) ... OK (copied local binary)
Packrat startup directives installed. Please quit and restart your R session.
Warning messages:
1: In untar2(tarfile, files, list, exdir, restore_times) :
skipping pax global extended headers
2: In untar2(tarfile, files, list, exdir, restore_times) :
skipping pax global extended headers

I haven't been able to reproduce the problem on the Mac.

Allow sources to be installed from bitbucket / gitorious

Let's make it a bit more remote agnostic...

Sorting needs to be done in "C" locale

Otherwise packrat.lock is rearranged back and forth when snapshotting on different systems.

From roxygen2:

sort_c <- function(x) with_collate("C", sort(x))
with_collate <- function(locale, code) {
  old <- set_collate(locale)
  on.exit(set_collate(old))

  force(code)
}

Most places that use cat() should use message() instead

@hadley pointed out that cat() is not the best choice for printing information to the user:

http://adv-r.had.co.nz/Exceptions-Debugging.html

"Function authors can also communicate with their users with print() or cat(), but I don't recommend it because it's hard to capture and selectively ignore this sort of output (and it's not a condition, so you can't use any of the useful condition handling tools)."

We should probably just use message() everywhere; almost all of output is intended to be use user messages of some kind.

Store version of R

Would it be sensible to have packrat store the version of R used when creating the library (including SVN revision if R-devel)? It could then print a warning if a mismatch is detected; an update to R could be recorded using snapshot() just like a package update.

::restore() should have a flag to suppress auto-removal of orphaned packages

makeInstall list, installPkgs, playInstallActions, etc. need to be renamed

These function names were written when they really only did installations. Now they handle any operation (upgrade, downgrade, remove, etc.) -- some renaming and light refactoring is necessary to make the code readable again.

grid package in R 3.1.0

...seems to be part of R

$ locate grid/DESC
/usr/lib/R/library/grid/DESCRIPTION

but packrat doesn't seem to be aware of that:

> packrat::status()
...
The following packages are referenced in your code, but are not present
in your library nor in packrat:
    grid
You will need to install these packages manually, then use
packrat::snapshot() to record these packages in packrat.

'rstudio' and 'manipulate' packages are treated as orphans in RStudio projects

Repro:

1. Bring a directory under Packrat control
2. Open a new project in the directory using RStudio
3. Run packrat::restore()

Packrat doesn't know what to do with the manipulate and rstudio packages that RStudio auto-installs to the library.

Suggested workaround for now is to ignore these packages by name. The work required in RStudio to fix this problem is nontrivial, and even if we did it, we'd still have a problem for everyone with an older version of the IDE.

Load dependencies from *.r and .Rprofile

The dirDependencies function currently ignores files with extension .r. Furthermore, as noted in #47, the .Rprofile file should be analyzed, too. (When editing this function, also consider changing the code that filters all entries in library, library.new and library.old to use a single grep call.)

Sources aren't cleaned up when packages are removed

When packages are added to packrat, sources are added, but when they're removed, the packrat.sources entries that correspond are not removed.

Improve error message when dependencies are missing

Bootstrap a project that doesn't include xts, then:

cat() should always include a trailing \n

@hadley pointed this out. R at the console doesn't start a new line automatically with every console prompt, so output from cat() can look pretty silly if it doesn't start a new line.

> cat("abc")
abc>

We should audit our usage of cat() and make sure we output \n consistently.

Am I really supposed to commit the .tar.gz archives in packrat.sources?

My global .gitignore ignores .tar.gz files, a common configuration I think, since git uses it's own compression algorithm committing these as .tar.gz is inefficient anyhow. As a consequence, calling git add packrat.sources doesn't add anything, but doesn't throw any warning or errors either. (I suppose this could be an easy stumbling block for anyone pasting in from the documentation?)

I don't quite appreciate why packrat needs the whole packrat.sources directory in the first place in addition to the libraries directory, (particularly as they contain many of the same packages). Is it possible to design packrat such that it can just install it's sources from github (say, adding a script that grabs the correct packrat hash from github and installs it), rather than having to carry a copy of packrat and all its dependencies around in my git repo?

bundle() command to make zip

A bundle() that creates a zip of the current project, excluding the local library, would be handy for emailing to others.

Rcpp dependency is unnecessary

sourcePackagePaths argument not working?

Hello,

First, thanks for building this package - looks like a very useful tool.

We have some packages in our local repository which I need to make Packrat aware of by the sourcePackagePaths argument...

packrat::bootstrap("lib/R/", sourcePackagePaths = c('/media/shared/code/R/repository/src/fooBar_4.0.tar.gz'))

However, i get the error:

Cannot treat /media/shared/code/R/repository/src/fooBar_4.0.tar.gz as a source package directory; /media/shared/code/R/repository/src/fooBar_4.0.tar.gz/DESCRIPTION is missing

The file does exist, however it just seems that file.exists cannot look inside tar.gz files, which is causing the error.

Am i misunderstanding something, or is this a bug? As an aside, I think it would be better to be able to specify /path/to/local/repo which contains all your local .tar.gz packages for Packrat to include in its search path.

Thanks

Incorrect status when package is removed

Start with a project that is up-to-date and has a dependency on, say, digest.

1. Remove code references to digest
2. remove.packages('digest')
3. packrat::status()

In this case the package is present in packrat, missing from the code, and missing from the library. snapshot() correctly removes the library, but status() says that the package is missing and that you should call restore().

Edit: The user intent for this scenario is the simple removal of a dependency.

Packrat can't find sources for packages in local (file://) repositories

download.packages doesn't actually store anything in the given target folder when the repo is local; we need to manually copy the sources in this case.

Remove DESCRIPTION

Remove the DESCRIPTION file and store the information it currently contains in packrat.lock.

Get the library location associated w/ each package on search path

search() (and, more concisely, .packages()) can be used to return the set of R packages currently in the search path. However, it would be useful to resolve the library location from which each package is loaded from, especially for the IDE integration.

Warning when source packages have long paths

Warning messages:
1: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘devtools-631f4b1268e0217b955643820bcf355811dd1466/tests/testthat/testDllRcpp/src/rcpp_hello_world.cpp’
2: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘devtools-631f4b1268e0217b955643820bcf355811dd1466/tests/testthat/testMarkdownVignettes/vignettes/test.Rmd’
3: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘devtools-631f4b1268e0217b955643820bcf355811dd1466/tests/testthat/testVignetteExtras/vignettes/new.Rnw’
4: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘devtools-631f4b1268e0217b955643820bcf355811dd1466/tests/testthat/testVignettesBuilt/vignettes/new.Rnw’
5: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘packrat-503ebdf506eef6e9ee5af72cdd07545bfe9304fd/tests/testthat/projects/libraries/library.new/lib-new.R’
6: In tar(tarfile = dest, files = basename(basedir), compression = "gzip", ... :
storing paths of more than 100 bytes is not portable:
‘packrat-503ebdf506eef6e9ee5af72cdd07545bfe9304fd/tests/testthat/projects/libraries/library.old/lib-old.R’

Add a function to list packages in the private library and/or last snapshot

The function should output a summary of the packages with information about each package's name, version, install source, and some other Packrat metadata.

@hadley suggested this would be very useful for bug reports, among other scenarios.

Package installation errors emitted by packrat are useless

When Packrat fails to install a package, the error displayed is usually Command failed (1), which doesn't help much with troubleshooting. Packrat could capture the output and/or messages emitted during the build and install phases of source packages and show them if a problem occurs.

parse errors encountered during dependency detection are not displayed

If packrat encounters a parse error while detecting dependencies, it ignores the file that has the parse error, with a warning. However, it does not report what the actual parse error was, which would be helpful for resolving the problem.

Incomplete final line warning when running most Packrat commands if source files have an incomplete final line

Autoload packages

Since packrat is providing an .Rprofile anyway, how about a clean interface to specify which packages to load in each new session? This information could be stored as a new field in packrat.lock.

Revisit status()

Implementation should use diff() and action messages should reflect new API

Install deps for file

This might be out of scope for packrat, but it occurs to me that you could apply the same dependency extraction code to an individual file. This would be useful because in the IDE you could automatically expose an error when you open a file (or package!) and you don't have the needed dependencies.

Better error message when bootstrap encounters missing dependencies

I have a project that has library(lubridate) which I didn't realize wasn't installed. The error I got was:

Snapshot written to /home/jcheng/development/testproj/packrat.lock 
Installing digest (0.6.3) ... OK (copied local binary)
Installing lubridate (1.3.0) ... Error in rownames(availablePkgs) : object 'availblePkgs' not found

Package update tools

It would be useful if packrat provided tools to tell you what upgrades are available, and to upgrade and snapshot.

This is particularly useful for development - for example, while I'm working on dplyr, I have other packages that use it for data analysis, and it's currently a bit of a pain to keep in sync. It would also be useful to help people upgrade their packages to the latest version in a principled manner.

Show Packrat status (if dirty) on R session start

We think this will keep folks from waiting forever to do snapshot() and restore().

Improve packrat output

1. Output from some commands exceeds 72 chars/line
2. Use cat and message consistently--cat for informative text, message for diagnostic traces ("installing", etc.)
3. .Rprofile message text should separate itself from output on R boot (with "--\n\n" or something)

Offer bootstrapping if library exists but does not contain packrat

A situation I ran into today. Removing the library folder helped, but I wish packrat had told me what's wrong.