R packages can depend on one another, but they can also depend on software
external to the R ecosystem. On Ubuntu 18.04, for example, in order to install
the curl
R package, you must have previously run apt-get install libcurl
. R
packages often note these dependencies inside their DESCRIPTION files, but this
information is free-form text that varies by package.
This repository contains a catalog of "rules" that can be used to systematically identify these dependencies and generate commands to install them.
You may be expecting to see a list like:
Package | SystemRequirements Field | Dependency |
---|---|---|
rgdal | "for building from source: GDAL >= ..." | libgdal-dev |
Storing this information as a table in this format is not efficient. Many R packages do not have any system dependencies, so the table would be very sparse. Moreover, R packages are added at an exponential rate, so maintaining this data would be nearly impossible.
Instead, this repository contains a set of rules that map a
SystemRequirements
field, e.g. rgdal
's "for building from source: GDAL >=
1.11.4 and <= 2.5.0, library from ..." to a platform specific install command:
apt-get install libgdal-dev gdal-bin libproj-dev
.
The primary purpose of this catalog is to support RStudio Package Manager which knows how to translate these rules into install steps for specific packages or repositiories. However, the community is free to use and contribute to these rules subject to the MIT license.
RStudio Package Manager is professionally supported, but RStudio does not offer support for these rules. Please file questions in RStudio Community or open an issue in this repository.
A similar project is maintained by R-Hub. The two catalogs have different data formats, test coverage, and target different operating systems.
The rules presented in this repository are extensively tested with the following process:
- A Docker container is started with a minimal base R image.
- A target R package is identified. The catalog of rules is applied to install any known requirements for the package into the Docker container.
- The package is installed.
If the package install is successful, there is a high chance the existing rules are sufficient. If the install fails, there is an indication that a rule is missing. This process is repeated for all CRAN packages across 6 Linux distributions: Ubuntu 16/18, CentOS 6/7/8, openSUSE 42/15.
The results are summarized below:
Percentage of CRAN Packages that Install Successfully
Ubuntu 16 | Ubuntu 18 | CentOS 6 | CentOS 7 | CentOS 8 | openSUSE 42.3 | openSUSE 15.0 | |
---|---|---|---|---|---|---|---|
No Rules | 78% | 78.1% | 77.4% | 77.8% | 77.7% | 78.2% | |
With Rules | 93.5% | 95.8% | 91.9% | 93.7% | 88.5% | 89.7% |
Percentage Weighted by Downloads
This table contains similar results as the table above, but adjusted by download. This metric indicates how good the rules are for the majority of packages R users are likely to install, discounting the long tail of packages that have system requirements but are not frequently used.
Ubuntu 16 | Ubuntu 18 | CentOS 6 | CentOS 7 | CentOS 8 | openSUSE 42.3 | openSUSE 15.0 | |
---|---|---|---|---|---|---|---|
No Rules | 90.1% | 90.1% | 90% | 90.1% | 90% | 90.2% | |
With Rules | 98.5% | 99.2% | 98.1% | 98.6% | 96.1% | 96.3% |
Both tests run with R 3.5.3 for all CRAN packages as of April 4, 2019.
The rules in this catalog are designed to be used with the following distributions:
- Ubuntu 14.04,16.04,18.04
- Debian 8,9
- CentOS 6,7,8
- Red Hat Enterprise Linux 6,7
- openSUSE 42.3, 15.0
- SUSE Linux Enterprise 12.3, 15.0
We welcome contributions to this catalog! To report a bug or request a rule, please open an issue in this repository. To add or update a rule, fork this repository and submit a pull request.
Each system requirement rule is described by a JSON file in the rules
directory. The file is named rule-name.json
, where
rule-name
is typically the name of the system dependency.
For example, here's an excerpt from a rule for the Protocol Buffers (protobuf)
library at rules/libprotobuf.json
.
{
"patterns": ["\\blibprotobuf\\b"], // regex which matches "libprotobuf" or "LIBPROTOBUF; libxml2"
"dependencies": [
{
"packages": ["protobuf-devel"], // to install the package: "yum install protobuf-devel"
"pre_install": [
{
"command": "yum install -y epel-release" // add the EPEL repository before installing
}
],
"constraints": [
{
"os": "linux",
"distribution": "centos", // make these instructions specific to CentOS 6
"versions": ["6"]
}
]
}
]
}
Other examples:
- Simple rule: git.json
- OS version constraints (package names vary by OS version): libmysqlclient.json
- Pre-install steps (adding the EPEL repo on CentOS/RHEL): gdal.json
- Post-install steps (reconfiguring R for Java): java.json
{
"patterns": [...],
"dependencies": [
{
"packages": [...],
"constraints": [
{
"os": ...,
"distribution": ...,
"versions": [...]
}
],
"pre_install": [
{
"command": ...,
"script": ...
}
],
"post_install": [
{
"command": ...,
"script": ...
}
]
}
]
}
Field | Type | Description |
---|---|---|
patterns |
Array | Regular expressions to match SystemRequirements fields. Case-insensitive. Note that the escape character must be escaped itself (\\. to match a dot). Use word boundaries (\\b ) for more accurate matches.Example: ["\\bgnu make\\b", "\\bgmake\\b"] to match GNU Make or gmake; OpenSSL |
dependencies |
Array | Rules for installing the dependency on one or more operating systems. See dependencies. |
Field | Type | Description |
---|---|---|
packages |
Array | Packages installed through the default system package manager (e.g. apt, yum, zypper). Examples: ["libxml2-dev"] , ["tcl", "tk"] |
constraints |
Array | One or more operating system constraints. See constraints. |
pre_install |
Array | Optional commands or scripts to run before installing packages (e.g. adding a third-party repository). See pre/post-install actions. |
post_install |
Array | Optional commands or scripts to run after installing packages (e.g. cleaning up). See pre/post-install actions. |
Field | Type | Description |
---|---|---|
os |
String | Operating system. Only "linux" is supported for now. |
distribution |
String | Linux distribution. One of "ubuntu" , "debian" , "centos" , "redhat" , "opensuse" , "sle" |
versions |
Array | Optional set of OS versions. If unspecified, the rule applies to all supported versions. See systems.json for supported values by OS. Example: ["16.04", "18.04"] for Ubuntu. |
Pre-install and post-install actions can be specified as either a command
or
script
. Commands are preferred unless there's complicated logic involved.
Field | Type | Description |
---|---|---|
command |
String | A shell command. Example: "yum install -y epel-release" |
script |
String | A shell script found in the scripts directory. Example: "centos_epel.sh" |
A typical workflow for adding a new rule:
-
Come up with regular expressions to match all R packages with the system dependency. See
sysreqs.json
for a sample list of CRAN packages and theirSystemRequirements
fields. Note that the applicable R packages don't have to be on CRAN; they can be on GitHub or other repositories, such as Bioconductor and rOpenSci. -
Determine the system packages and any pre/post-install steps if needed. The more operating systems covered, the better, but it's fine if only some operating systems are covered.
Useful resources for finding packages across different OSs:
Or to search for packages on each OS:
# Ubuntu/Debian apt-cache search <package-name> # CentOS/RHEL yum search <package-name> # openSUSE/SLE zypper search <package-name>
Add the new rule as a
rule-name.json
file in therules
directory.Run the schema tests and (optionally) the system package tests locally.
Submit a pull request.
To lint and validate rules against the schema, you'll need Node.js.
# Install dependencies npm install # Run the tests npm test
To list R packages and system requirements matched by a rule:
# List matching system requirements for a rule npm run test-patterns -- rules/libcurl.json --verbose # List matching system requirements for all rules npm run test-patterns-all -- --verbose # Fail if a rule doesn't match any system requirements npm run test-patterns-all -- --strict
Docker images are provided to help validate system packages on supported OSs.
Available tags:
trusty
(Ubuntu 14.04)xenial
(Ubuntu 16.04)bionic
(Ubuntu 18.04)jessie
(Debian 8)stretch
(Debian 9)centos6
(CentOS 6)centos7
(CentOS 7)centos8
(CentOS 8)opensuse42
(openSUSE 42.3)opensuse15
(openSUSE 15.0)
To build the images:
# Build a specific image (e.g. trusty) make build-trusty # Build all images make build-all
To test the rules:
# Test a specific rule on an OS (e.g. trusty) make test-trusty RULES=rules/libcurl.json # Test a specific rule on all OSs make test-all RULES=rules/libcurl.json # Test all rules on all OSs make test-all
The JSON schema is defined in the file
schema.json
. Do not modify this file directly, since it is automatically generated. Instead, modifyschema.template.json
and then runnpm run generate-schema
. Thegenerate-schema
target is automatically run when runningnpm test
.If you need to modify the distros and/or versions supported in the schema definitions, modify
systems.json
.Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.