extendr/extendr

R extension library for rust designed to be familiar to R users.

extendr - A safe and user friendly R extension interface using Rust

Installation - Rust

Extendr is available on crates.io.

Simply add this line to the [dependencies] section of a rust crate. You will then be able to call R code from Rust.

[dependencies]
extendr-api = "0.6"

Installation - R

There are two ways you can use the extendr API from R. First, you can use the rextendr package to call individual Rust functions from an R session. Second, you can write an R package that uses compiled Rust code, see the helloextendr repo for a minimal example.

Overview

Extendr is a Rust extension mechanism for R

It is intended to be easier to use than the C interface and Rcpp as Rust gives type safety and freedom from segfaults.

The following code illustrates a simple structure trait which is written in Rust. The data is defined in the struct declaration and the methods in the impl.

use extendr_api::prelude::*;

struct Person {
    pub name: String,
}

#[extendr]
impl Person {
    fn new() -> Self {
        Self { name: "".to_string() }
    }

    fn set_name(&mut self, name: &str) {
        self.name = name.to_string();
    }

    fn name(&self) -> &str {
        self.name.as_str()
    }
}

#[extendr]
fn aux_func() {
}


// Macro to generate exports
extendr_module! {
    mod classes;
    impl Person;
    fn aux_func;
}

The #[extendr] attribute causes the compiler to generate wrapper and registration functions for R which are called when the package is loaded.

The extendr_module! macro lists the module name and exported functions and interfaces.

This library aims to provide an interface that will be familiar to first-time users of Rust or indeed any compiled language.

Anyone who knows the R library should be able to write R extensions.

Goals of the project

Instead of wrapping R objects, we convert to Rust native objects on entry to a function. This makes the wrapped code clean and dependency free. The ultimate goal is to allow the wrapping of existing Rust libraries without markup, but in the meantime, the markup is as light as possible.

#[extendr]
pub fn my_sum(v: &[f64]) -> f64 {
    v.iter().sum()
}

You can interact in more detail with R objects using the RObj type which wraps the native R object type. This supports a large subset of the R internals functions, but wrapped to prevent accidental segfaults and failures.

extendr roadmap

Basic

• Be able to build simple rust extensions for R.
• Wrap the R SEXP object safely (Robj)
• Iterator support for matrices and vectors.
• Class support.

Documentation

• Begin documentation.
• Begin book-form documentation.
• Paper for Bioinformatics.
• Build and publish CRAN R package.
• Publish Use R! series book.

Automation

• Auto-generate binding wrappers.
• Auto-generate NAMESPACE and lib.R.

Features

• Feature-gated support for ndarray.
• Feature-gated support for rayon.

R packages

• Bindings for rust-bio

Contributing

We are happy about any contributions!

To get started you can take a look at our Github issues.

You can also get in contact via our Discord server!

Development

The documentation for the latest development version is available here: https://extendr.github.io/extendr/extendr_api/

extendr's Issues

Differences between start_r in libR-sys and extendr-api

I'm noticing some differences in the testing code in libR-sys and the deployed code in extendr-api when it comes to starting an R process. In particular:
https://github.com/extendr/libR-sys/blob/b141c83c9e4b7329253e4025182b87c54f69ea7e/src/lib.rs#L80-L88
vs:

extendr/extendr-api/src/engine.rs

Lines 64 to 70 in 5227020

	// In case you are curious.
	// Maybe 8MB is a bit small.
	// eprintln!("R_CStackLimit={:016x}", R_CStackLimit);
	R_CStackLimit = usize::max_value();
	setup_Rmainloop();

@CGMossa Could this cause some of the trouble you have compiling on Windows?

There's also this difference:
https://github.com/extendr/libR-sys/blob/b141c83c9e4b7329253e4025182b87c54f69ea7e/src/lib.rs#L78
vs:

extendr/extendr-api/src/engine.rs

Lines 59 to 62 in 5227020

	Rf_initialize_R(
	3,
	[cstr_mut!("R"), cstr_mut!("--slave"), cstr_mut!("--no-save")].as_mut_ptr(),
	);

Is NumericVector the right way to go?

It may be better to wrap rust ndarray instead. We can't make an ndarray owned object, but we can wrap a view or view_mut.

These can be handled by the export_function macro,

Conversion of string vector to string scalar fails with "not a string object"

If a Rust function expecting a single string is handed a string vector from R, then it raises an error stating "not a string object". See here:

extendr/tests/extendrtests/tests/testthat/test-type-conversion.R

Line 29 in 9752c7f

expect_error(.Call(wrap__char_scalar, c("hello", "world")), "not a string object") # why this error message and not "Input must be of length 1"?

The error is appropriate, but the error message is confusing. A better error would be "Input must be of length 1", as is used for the other types, see e.g. here:

extendr/tests/extendrtests/tests/testthat/test-type-conversion.R

Line 7 in 9752c7f

expect_error(.Call(wrap__double_scalar, c(.45, .46)), "Input must be of length 1")

Unable to run on Windows

Currently, I am running stable-gnu-toolchain, with MSYS installed.
I'm missing something, that I am not entirely sure how to configure:

cargo build
   Compiling libR-sys v0.1.6
error: failed to run custom build command for `libR-sys v0.1.6`

Caused by:
  process didn't exit successfully: `C:/cargo_target_dir/debug\build\libR-sys-f8980ef9d4a34bd4\build-script-build` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBR_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=LIBR_STATIC
  cargo:rerun-if-env-changed=LIBR_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failure { command: "\"pkg-config\" \"--libs\" \"--cflags\" \"libR\"", output: Output { status: ExitStatus(ExitStatus(1)), stdout: "", stderr: "Package libR 
was not found in the pkg-config search path.\r\nPerhaps you should add the directory containing `libR.pc\'\r\nto the PKG_CONFIG_PATH environment variable\r\nNo package \'libR\' found\r\n" } }', C:\Users\Tyler\.cargo\registry\src\github.com-1ecc6299db9ec823\libR-sys-0.1.6\build.rs:37:51
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I tried to run pacman -Ss r-base-dev but that doesn't work for me.

Result handling on return.

After a bit of experimentation. I think we should implement Result<T, E> in the following way:

library(rextendr)

res = rust_function(
  code = "fn rust_error(val: i32) -> Result<i32, Box<dyn Error>> {
     if val == 0 {
         Err("val is zero!".into())
     } else {
         Ok(val)
     }
  }",
)

res <- rust_error(input)
if (inherits(res, "rust.error")) {
  # handle error (res is actually a string)
} else {
  # use res
}

What do you think as R users?

Return module metadata from DLL

As described in this comment #40 (comment) by @andy-thomason:

We have code in the #[extendr_module] generator to register
functions with R using the standard DLL mechanism.

extendr/extendr-macros/src/lib.rs

Lines 334 to 354 in 2d2cbcf

	///
	/// extendr_module! {
	/// mod name;
	/// fn my_func1;
	/// fn my_func2;
	/// impl MyTrait;
	/// }
	///
	#[proc_macro]
	pub fn extendr_module(item: TokenStream) -> TokenStream {
	let module = parse_macro_input!(item as Module);
	let Module {modname, fnnames, implnames} = module;
	let modname = modname.unwrap();
	let module_init_name = format_ident!("R_init_lib{}", modname);
	let fninitnames = fnnames.iter().map(|id| format_ident!("{}{}", INIT_PREFIX, id));
	let implinitnames = implnames.iter().map(|id| format_ident!("{}{}", INIT_PREFIX, id));
	TokenStream::from(quote!{
	#[no_mangle]
	#[allow(non_snake_case)]

It would be nice to add functions to provide metadata, including doc strings, to a function that
could return them to R.

extern "C" fn __get_module_metadata() -> Robj {
  ...
  data_frame!(name=..., args=..., doc=...)
}

This way we could use the rextendr code to generate functions, NAMESPACE and documentation
for all the functions and methods.

For regular functions, name is just the name called by R.
For methods and traits, the name is qualified: Fred::new()

The doc string is available as an attribute of the function but will need
to be translated to the roxygen2 format.

Panic handling

We need to catch panics in function calls from R code.

The important thing is to avoid killing R.

It should be pointed out, however, that panic handling is not
safe as it causes memory leaks.

Handling of missing values

Do we have a systematic way of handling NA values on the Rust side? It's not clear to me what the current intent is to handling missing values. Most basic Rust data types (e.g., String) cannot handle missing values. Instead, Rust uses options. Maybe we should systematically wrap all incoming data types into options?

As an example, the current implementation silently converts NA_character_ into "NA", which I would argue is incorrect behavior.

library(rextendr)

rust_function(
  code = "fn rust_strings(input: &str) -> String {
    input.to_string()
  }",
  patch.crates_io = c(
    'extendr-api = { git = "https://github.com/extendr/extendr" }',
    'extendr-macros = { git = "https://github.com/extendr/extendr" }'
  )
)

rust_strings("a")
#> [1] "a"
rust_strings(NA_character_)
#> [1] "NA"

^{Created on 2020-12-05 by the reprex package (v0.3.0)}

Switch Github Actions to ubuntu-20.04

The GH Actions scripts currently use ubuntu-latest, which is ubuntu-18.04. Over the next few weeks, this will switch over to ubuntu-20.04 (actions/runner-images#1816), and then this line will break:

extendr/.github/workflows/test.yml

Line 24 in 5ebf0ab

run: sudo apt-get install r-base-dev llvm-3.9-dev libclang-3.9-dev

I think it would be best to switch over to ubuntu-20.04 explicitly now. You can see a workflow for that version here:
https://github.com/clauswilke/helloextendr/blob/main/.github/workflows/R-CMD-check.yaml
Notably, it doesn't require any setup for libclang, it just works.

cc @hobofan @CGMossa

Compilation of "hello" example fails when using extendr from crates.io

In the "hello" example package, if I use the extendr crate from crates.io by modifying my Cargo.toml file as follows:

[dependencies]
#extendr-api = {path = "../../../../extendr-api"}
extendr-api = "0.1.3"

then compilation fails. rustc can't find the extendr attribute in this code fragment:

#[extendr]
fn hello() -> &'static str {
    "hello"
}

Full compilation log is in the details.

==> R CMD INSTALL --no-multiarch --with-keep.source hello

rm -Rf hello.so ../target/release/libhello.a empty.o
* installing to library ‘/Users/clauswilke/Library/R/4.0/library’
* installing *source* package ‘hello’ ...
** using staged installation
** libs
/Library/Frameworks/R.framework/Resources/share/make/shlib.mk:6: warning: overriding commands for target `hello.so'
Makevars:9: warning: ignoring old commands for target `hello.so'
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c empty.c -o empty.o
echo BUILDING STATLIB
BUILDING STATLIB
cargo build --release --manifest-path=../Cargo.toml
    Updating crates.io index
   Compiling libc v0.2.80
   Compiling memchr v2.3.4
   Compiling proc-macro2 v1.0.24
   Compiling autocfg v1.0.1
   Compiling version_check v0.9.2
   Compiling cc v1.0.62
   Compiling glob v0.3.0
   Compiling unicode-xid v0.2.1
   Compiling log v0.4.11
   Compiling lazy_static v1.4.0
   Compiling bitflags v1.2.1
   Compiling quick-error v1.2.3
   Compiling regex-syntax v0.6.21
   Compiling cfg-if v0.1.10
   Compiling unicode-width v0.1.8
   Compiling termcolor v1.1.0
   Compiling strsim v0.8.0
   Compiling ansi_term v0.11.0
   Compiling bindgen v0.53.3
   Compiling vec_map v0.8.2
   Compiling shlex v0.1.1
   Compiling rustc-hash v1.1.0
   Compiling peeking_take_while v0.1.2
   Compiling lazycell v1.3.0
   Compiling syn v1.0.48
   Compiling rawpointer v0.2.1
   Compiling ndarray v0.13.1
   Compiling extendr-api v0.1.3
   Compiling thread_local v1.0.1
   Compiling humantime v1.3.0
   Compiling textwrap v0.11.0
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling num-complex v0.2.4
   Compiling nom v5.1.2
   Compiling matrixmultiply v0.2.3
   Compiling clang-sys v0.29.3
   Compiling aho-corasick v0.7.15
   Compiling atty v0.2.14
   Compiling which v3.1.1
   Compiling clap v2.33.3
   Compiling quote v1.0.7
   Compiling regex v1.4.2
   Compiling env_logger v0.7.1
   Compiling cexpr v0.4.0
   Compiling libloading v0.5.2
   Compiling extendr-macros v0.1.2
   Compiling libR-sys v0.1.9 (/Users/clauswilke/github/libR-sys)
   Compiling hello v0.1.0 (/Users/clauswilke/github/extendr/extendr-api/examples/R/hello)
error: cannot find attribute `extendr` in this scope
 --> src/lib.rs:3:3
  |
3 | #[extendr]
  |   ^^^^^^^

warning: unused import: `extendr_api::*`
 --> src/lib.rs:1:5
  |
1 | use extendr_api::*;
  |     ^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

error: aborting due to previous error; 1 warning emitted

error: could not compile `hello`.

To learn more, run the command again with --verbose.
make: *** [../target/release/libhello.a] Error 101
ERROR: compilation failed for package ‘hello’
* removing ‘/Users/clauswilke/Library/R/4.0/library/hello’
* restoring previous ‘/Users/clauswilke/Library/R/4.0/library/hello’

Exited with status 1.

By comparison, if I use the local source for the crate, as it is set up in the github repo, then compilation completes without issues. (There's a problem with the package function, though, but that's a different issue: #29)

==> R CMD INSTALL --no-multiarch --with-keep.source hello

* installing to library ‘/Users/clauswilke/Library/R/4.0/library’
* installing *source* package ‘hello’ ...
** using staged installation
** libs
rm -Rf hello.so ../target/release/libhello.a empty.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c empty.c -o empty.o
/Library/Frameworks/R.framework/Resources/share/make/shlib.mk:6: warning: overriding commands for target `hello.so'
Makevars:9: warning: ignoring old commands for target `hello.so'
echo BUILDING STATLIB
BUILDING STATLIB
cargo build --release --manifest-path=../Cargo.toml
    Updating crates.io index
   Compiling libc v0.2.80
   Compiling memchr v2.3.4
   Compiling proc-macro2 v1.0.24
   Compiling autocfg v1.0.1
   Compiling cc v1.0.62
   Compiling version_check v0.9.2
   Compiling unicode-xid v0.2.1
   Compiling glob v0.3.0
   Compiling lazy_static v1.4.0
   Compiling log v0.4.11
   Compiling bitflags v1.2.1
   Compiling regex-syntax v0.6.21
   Compiling cfg-if v0.1.10
   Compiling quick-error v1.2.3
   Compiling unicode-width v0.1.8
   Compiling termcolor v1.1.0
   Compiling strsim v0.8.0
   Compiling bindgen v0.53.3
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling lazycell v1.3.0
   Compiling rustc-hash v1.1.0
   Compiling shlex v0.1.1
   Compiling peeking_take_while v0.1.2
   Compiling syn v1.0.48
   Compiling ndarray v0.13.1
   Compiling rawpointer v0.2.1
   Compiling extendr-api v0.1.3 (/Users/clauswilke/github/extendr/extendr-api)
   Compiling thread_local v1.0.1
   Compiling humantime v1.3.0
   Compiling textwrap v0.11.0
   Compiling num-traits v0.2.14
   Compiling num-complex v0.2.4
   Compiling num-integer v0.1.44
   Compiling nom v5.1.2
   Compiling clang-sys v0.29.3
   Compiling matrixmultiply v0.2.3
   Compiling aho-corasick v0.7.15
   Compiling atty v0.2.14
   Compiling which v3.1.1
   Compiling quote v1.0.7
   Compiling clap v2.33.3
   Compiling regex v1.4.2
   Compiling cexpr v0.4.0
   Compiling env_logger v0.7.1
   Compiling libloading v0.5.2
   Compiling extendr-macros v0.1.2 (/Users/clauswilke/github/extendr/extendr-macros)
   Compiling libR-sys v0.1.9 (/Users/clauswilke/github/libR-sys)
   Compiling hello v0.1.0 (/Users/clauswilke/github/extendr/extendr-api/examples/R/hello)
clang -mmacosx-version-min=10.13 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o hello.so empty.o -L../target/release -lhello -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
    Finished release [optimized] target(s) in 1m 12s
installing to /Users/clauswilke/Library/R/4.0/library/00LOCK-hello/00new/hello/libs
** R
** byte-compile and prepare package for lazy loading
No man pages found in package  ‘hello’ 
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (hello)

doc.rs still not showing

I submitted a PR to doc.rs to get libR included in the build. As yet I haven't seen an update on the site
but am waiting for it to update.

Is this a long-term project?

I am quite interested in this project but would like to know if it will be maintained in the long run. For now, it looks like a one-person hobby project.

question: is there already a way to create a R STRSEXP?

I'm not able to find a function that provide the API to directly returns a STRSXP, that is a string vector, or a list of CHARSXP, from Rust side.

For example, say on Rust side I have created foo: Vec<Cstring> and wish to transfer it to R's C interface,

The first edition of Advanced R provided some information:

abc <- cfunction(NULL, '
  SEXP out = PROTECT(allocVector(STRSXP, 3));

  SET_STRING_ELT(out, 0, mkChar("a"));
  SET_STRING_ELT(out, 1, mkChar("b"));
  SET_STRING_ELT(out, 2, mkChar("c"));

  UNPROTECT(1);

  return out;
')
abc()

I guess roughly what I need is the following

use libR_sys::*;
use std::ffi::CString;

fn some_function(in: Vec<CString>) -> SEXP {
    let out = Rf_protect(Rf_allocVector(STRSXP, foo.len());

    for (idx, cstring) in foo.into_iter().enumerate() {
        SET_STRING_ELT(out, idx, Rf_mkchar(cstring.into_raw()));
    }

    Rf_unprotect(1);

    return out
}

Is this correct?
Do you think we can incorporate this functionality to easily convert Vec<CString> or Vec<String> to STRSXP(and consuming the vector)?

Getting involved.

Let me know if you are interested in getting involved with this project.

Leave a comment in this issue if you want to join in.

Integration tests create GitHub merge chicken-and-egg problem

The integration tests contain a separate R package that uses extendr just like any stand-alone package would. It pulls the latest extendr sources from GitHub. This creates a chicken-and-egg problem if the integration tests depend on some new extendr feature that hasn't been pushed to the main branch on GitHub yet.

I ran into this when I bumped libR-sys to version 0.2.0. You can see that the latest commit before the merge failed the CI tests: 42b5ccb. However, the actual merge into master had all tests succeed: cecc67f.

I don't currently see a way to fix this. The offending lines in the code are here:

extendr/tests/extendrtests/src/rust/Cargo.toml

Lines 19 to 21 in cecc67f

	extendr-api = { git = "https://github.com/extendr/extendr" }
	extendr-engine = { git = "https://github.com/extendr/extendr" }
	extendr-macros = { git = "https://github.com/extendr/extendr" }

We might think to simply use relative paths within the repo here, but that won't work because the entire extendrtests folder gets copied into a temp directory when the integration tests are run, and thus relative paths break.

I'm not sure there's anything that can be done, so this issue may primarily serve as an explanation and reminder of what is happening.

Confusing name

I feel like the library name extendr sounds confusing and unclear. Is it possible to just call it as something similar to RustR?

Make ndarray optional

Thought I'd open up a separate issue for this, as it has come up in comments repeatedly (and is in the Roadmap of the README).

This would probably a good first step towards feature flags (which is also a concern in #60), which I guess also would need to get some support in rextendr.

R integration tests issue on linux

After updating GitHub Actions to include all OSs and all required architectures, I noticed that all linux R integration tests produce a Note, complaining about the size of some package subfolder:
https://github.com/extendr/extendr/runs/1603463755?check_suite_focus=true#step:14:111
https://github.com/extendr/extendr/runs/1603463765?check_suite_focus=true#step:14:111
https://github.com/extendr/extendr/runs/1603463770?check_suite_focus=true#step:14:111
https://github.com/extendr/extendr/runs/1603463775?check_suite_focus=true#step:14:109

This, however, happens neither for MacOS nor for Windows builds (both architectures):
https://github.com/extendr/extendr/runs/1603463728?check_suite_focus=true#step:14:203
https://github.com/extendr/extendr/runs/1603463746?check_suite_focus=true#step:14:98

Don't know if it is important, but I find it strange that only one OS consistently produces notes when tested.

Clippy warnings

There are numerous clippy warnings. Most are related to publicly exposed unsafe functions. I would like to go an fix this, but what is the guidelines in the project regarding public unsafe functions? Allow them or wrap them so no public functions are unsafe?

Fix bump.sh and publish.sh for new crate structure

I have added a new crate extendr-engine (#50) to remove the engine part from the default api. However, the scripts bump.sh and publish.sh don't reflect this change yet.

Compiled Rust library contains symbols R doesn't like

I have been trying to build an R package that can pass R CMD check: https://github.com/clauswilke/helloextendr

This doesn't currently work, because the compiled Rust library contains symbols that R doesn't like:

* checking compiled code ... NOTE
File ‘helloextendr/libs/helloextendr.so’:
  Found non-API calls to R: ‘R_CStackLimit’, ‘R_CleanTempDir’,
    ‘R_RunExitFinalizers’, ‘Rf_initialize_R’, ‘setup_Rmainloop’

Compiled code should not call non-API entry points in R.

See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.

I believe these are all functions that are needed to create a Rust binary calling R as a library. So we may need to take these out and put them into a separate crate, maybe called extendr-standalone or extendr-bin. Unfortunately we need these calls to run cargo test, so the tests would have to import that crate I assume.

Serde for R lists

It would be nice to provide conversion to/from structs using #[derive(Serialize, Deserialize)]

This should be fairly straightforward to do if anyone felt like a challenge.

eg.

#[derive(Serialize, Deserialize)]
struct Fred {
  x: String,
  y: f64,
}

fn consume_list(list: Robj) -> Fred {
   // Call from_robj on every member of Fred.
   Robj::from_list::<Fred>(&list).unwrap()
}

fn produce_list(fred: Fred) -> Robj {
  // Call from<Robj> on every member of Fred.
  Robj::to_list(&fred)
}

Note that the default behaviour on structs is to convert to/from
an EXTPTRSXP.

"cannot find attribute `extendr` in this scope"

I just stumbled across this repo and it's really cool! Thanks!

I was trying to get a toy example running and I hit this error. I think it's fixed already because I switched my Cargo.toml from extendr-api = "0.1.3" to extendr-api = { git = "https://github.com/extendr/extendr" } and it works, but I'm making this issue in case anyone else hits it too.

Switch CI to Github Actions

From what I can gather on Discord there seems to be a consensus to switch to GH Actions (and away from TravisCI).

Some of the goals:
Build on

• Linux (added in #44)
• macOS (added in #44)
• Windows (unfinished version added in #44)
• Automatic publish on pushed tags

Do you want to tackle this @CGMossa ? Otherwise I'd take a stab at it soon(-ish).

extendr-api cannot be compiled when ndarray is enabled

As I commented on extendr/rextendr#10 (comment), I see this error on my local, but this doesn't happen on CI runs. Does anyone see the same errors? (I'm using Rust 1.49 on Ubuntu 20.04).

❯ cargo test --manifest-path extendr-api/Cargo.toml --features ndarray
   Compiling libR-sys v0.2.0
   Compiling extendr-engine v0.1.11 (/home/yutani/repo/extendr/extendr-engine)
   Compiling extendr-api v0.1.11 (/home/yutani/repo/extendr/extendr-api)
error[E0106]: missing lifetime specifier
  --> extendr-api/src/robj_ndarray.rs:14:24
   |
14 |     Robj: AsTypedSlice<T>,
   |                        ^ expected named lifetime parameter
   |
help: consider using the `'a` lifetime
   |
14 |     Robj: AsTypedSlice<'a, T>,
   |                        ^^^

error[E0106]: missing lifetime specifier
  --> extendr-api/src/robj_ndarray.rs:14:24
   |
14 |     Robj: AsTypedSlice<T>,
   |                        ^ expected named lifetime parameter
   |
help: consider using the `'a` lifetime
   |
14 |     Robj: AsTypedSlice<'a, T>,
   |                        ^^^

error[E0310]: the parameter type `T` may not live long enough
  --> extendr-api/src/robj_ndarray.rs:17:31
   |
12 | impl<'a, T> FromRobj<'a> for ArrayView1<'a, T>
   |          - help: consider adding an explicit lifetime bound...: `T: 'static`
...
17 |         if let Some(v) = robj.as_typed_slice() {
   |                               ^^^^^^^^^^^^^^ ...so that the reference type `&[T]` does not outlive the data it points at

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0106, E0310.
For more information about an error, try `rustc --explain E0106`.
error: could not compile `extendr-api`

.dylib, not .so files are produced on Mac, which makes installation fail

When I tried to R CMD INSTALL . the hello example on Mac, it fails because:

cp: ../target/release/libhello.so: No such file or directory

and I fount that libhello.dylib was produced instead

Avoid R_PreserveObject for performance reasons

Just filing this here as an issue that should be addressed at some point. The current code uses R_PreserveObject to create objects that are "owned" (not really) by Rust and protected from the GC. See e.g. here:

extendr/extendr-api/src/robj.rs

Line 1057 in 5245703

R_PreserveObject(sexp);

This works fine as proof of concept but should be replaced eventually. It will not scale. (Until recently, it could crash R.)

See a detailed explanation of this issue here: RcppCore/Rcpp#1081
See the code that cpp11 wrote to work around this issue here:
https://github.com/r-lib/cpp11/blob/f1f971ffdd94d7949b7881fccadef825ba17a0a4/inst/include/cpp11/protect.hpp#L209-L364

Basically, we need to build our own data structure (e.g., an R linked list) that is understood by R's GC and that can rapidly created and remove protected objects.

Stricter input checking for extendr functions

Related to #63, I think conversion from Robj to Rust native types should be stricter and only succeed when the coercion is unique and provably correct. As an example, vector input should not be silently converted into a single value. See reprex for an example. If input is f64, not &[f64], then only vectors of length 1 should be allowed as input.

library(rextendr)

rust_function(
  code = "fn one_float(input: f64) -> f64 {
    5.*input
  }",
  patch.crates_io = c(
    'extendr-api = { git = "https://github.com/extendr/extendr" }',
    'extendr-macros = { git = "https://github.com/extendr/extendr" }'
  )
)

# correct
one_float(numeric(0))
#> Error in one_float(numeric(0)): zero length vector

# correct
one_float(numeric(1))
#> [1] 0

# incorrect, should complain about vector of length > 1
one_float(numeric(2))
#> [1] 0

^{Created on 2020-12-06 by the reprex package (v0.3.0)}

R floats are silently converted into Rust ints

If a Rust function expecting an int is handed a float from R, then the float is silently converted to int, stripping any decimal values in the process. See e.g. here:

extendr/tests/extendrtests/tests/testthat/test-type-conversion.R

Line 10 in 9752c7f

expect_equal(.Call(wrap__int_scalar, 4.4), 4L) # is this deliberate? seems dangerous

I think this should instead raise an "unable to convert" error. Silent conversion from float to int is dangerous.

R CRAN package

The CRAN package should have the following features:

A function to install cargo.
A function to build a skeleton project (wrapping cargo new).

Optionally:

An inline rust code builder.

Any other suggestions welcome, please comment.

cargo test fails extendr-api

When trying to run cargo test for extendr-api, I get a segfault. Is this OS X specific or does the same problem show up on other platforms? I note that the "hello" example builds and runs fine.

clauswilke ~/github/extendr/extendr-api> cargo test
   Compiling extendr-api v0.1.4 (/Users/clauswilke/github/extendr/extendr-api)
    Finished test [unoptimized + debuginfo] target(s) in 1.76s
     Running /Users/clauswilke/github/extendr/target/debug/deps/extendr_api-6486cbe7629caf2f

running 7 tests
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `/Users/clauswilke/github/extendr/target/debug/deps/extendr_api-6486cbe7629caf2f` (signal: 11, SIGSEGV: invalid memory reference)

The "hello" example compiles but doesn't work.

I have been able to compile the "hello" package without apparent issue (see log in details, and see #28 for a caveat on this statement). However, when I try to run the hello() function, I get an unresolved symbol. The Rust function doesn't seem to be registered with R.

library(hello)

hello()
#> Error in .Call("wrap__hello"): "wrap__hello" not resolved from current namespace (hello)

^{Created on 2020-11-11 by the reprex package (v0.3.0)}

==> R CMD INSTALL --no-multiarch --with-keep.source hello

* installing to library ‘/Users/clauswilke/Library/R/4.0/library’
* installing *source* package ‘hello’ ...
** using staged installation
** libs
rm -Rf hello.so ../target/release/libhello.a empty.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c empty.c -o empty.o
/Library/Frameworks/R.framework/Resources/share/make/shlib.mk:6: warning: overriding commands for target `hello.so'
Makevars:9: warning: ignoring old commands for target `hello.so'
echo BUILDING STATLIB
BUILDING STATLIB
cargo build --release --manifest-path=../Cargo.toml
    Updating crates.io index
   Compiling libc v0.2.80
   Compiling memchr v2.3.4
   Compiling proc-macro2 v1.0.24
   Compiling autocfg v1.0.1
   Compiling cc v1.0.62
   Compiling version_check v0.9.2
   Compiling unicode-xid v0.2.1
   Compiling glob v0.3.0
   Compiling lazy_static v1.4.0
   Compiling log v0.4.11
   Compiling bitflags v1.2.1
   Compiling regex-syntax v0.6.21
   Compiling cfg-if v0.1.10
   Compiling quick-error v1.2.3
   Compiling unicode-width v0.1.8
   Compiling termcolor v1.1.0
   Compiling strsim v0.8.0
   Compiling bindgen v0.53.3
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling lazycell v1.3.0
   Compiling rustc-hash v1.1.0
   Compiling shlex v0.1.1
   Compiling peeking_take_while v0.1.2
   Compiling syn v1.0.48
   Compiling ndarray v0.13.1
   Compiling rawpointer v0.2.1
   Compiling extendr-api v0.1.3 (/Users/clauswilke/github/extendr/extendr-api)
   Compiling thread_local v1.0.1
   Compiling humantime v1.3.0
   Compiling textwrap v0.11.0
   Compiling num-traits v0.2.14
   Compiling num-complex v0.2.4
   Compiling num-integer v0.1.44
   Compiling nom v5.1.2
   Compiling clang-sys v0.29.3
   Compiling matrixmultiply v0.2.3
   Compiling aho-corasick v0.7.15
   Compiling atty v0.2.14
   Compiling which v3.1.1
   Compiling quote v1.0.7
   Compiling clap v2.33.3
   Compiling regex v1.4.2
   Compiling cexpr v0.4.0
   Compiling env_logger v0.7.1
   Compiling libloading v0.5.2
   Compiling extendr-macros v0.1.2 (/Users/clauswilke/github/extendr/extendr-macros)
   Compiling libR-sys v0.1.9 (/Users/clauswilke/github/libR-sys)
   Compiling hello v0.1.0 (/Users/clauswilke/github/extendr/extendr-api/examples/R/hello)
clang -mmacosx-version-min=10.13 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o hello.so empty.o -L../target/release -lhello -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
    Finished release [optimized] target(s) in 1m 12s
installing to /Users/clauswilke/Library/R/4.0/library/00LOCK-hello/00new/hello/libs
** R
** byte-compile and prepare package for lazy loading
No man pages found in package  ‘hello’ 
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (hello)

Code formatting

I saw a lot of irrelevant changes on #84 due to auto-formatting. Can we follow the default of cargo fmt and check it on CI?

c.f. https://github.com/dimforge/nalgebra/blob/d8fa3ff24147dee0c0b481881aad1c2890fad942/.circleci/config.yml#L13-L22

Unable to compile with R 3.2.0

The following includes cause extendr to not be able to be built against R 3.2.0. I would like to do conditional compilation to allow extendr to build.

extendr/extendr-api/src/lib.rs

Lines 79 to 83 in 5ebf0ab

	pub use libR_sys::DllInfo;
	pub use libR_sys::R_CallMethodDef;
	pub use libR_sys::R_forceSymbols;
	pub use libR_sys::R_registerRoutines;
	pub use libR_sys::R_useDynamicSymbols;

Some examples are broken

For example, examples/R/data doesn't compile anymore. It breaks with:

error[E0599]: no method named `namesAndValues` found for enum `extendr_api::Robj` in the current scope
  --> src/lib.rs:19:43
   |
19 |     if let Some(names_and_values) = input.namesAndValues() {
   |                                           ^^^^^^^^^^^^^^ method not found in `extendr_api::Robj`

clauswilke ~/github/extendr/extendr-api/examples/R/data> R CMD INSTALL .
* installing to library ‘/Users/clauswilke/Library/R/4.0/library’
* installing *source* package ‘data’ ...
** using staged installation
** libs
/Library/Frameworks/R.framework/Resources/share/make/shlib.mk:6: warning: overriding commands for target `data.so'
Makevars:9: warning: ignoring old commands for target `data.so'
rm -Rf data.so ../target/release/libdata.a entrypoint.o ../target
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c entrypoint.c -o entrypoint.o
echo BUILDING STATLIB
BUILDING STATLIB
cargo build --release --manifest-path=../Cargo.toml
   Compiling proc-macro2 v1.0.24
   Compiling libR-sys v0.2.0
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.56
   Compiling extendr-engine v0.1.11 (/Users/clauswilke/github/extendr/extendr-engine)
   Compiling lazy_static v1.4.0
   Compiling quote v1.0.8
   Compiling extendr-macros v0.1.11 (/Users/clauswilke/github/extendr/extendr-macros)
   Compiling extendr-api v0.1.11 (/Users/clauswilke/github/extendr/extendr-api)
   Compiling data v0.1.0 (/Users/clauswilke/github/extendr/extendr-api/examples/R/data)
error[E0599]: no method named `namesAndValues` found for enum `extendr_api::Robj` in the current scope
  --> src/lib.rs:19:43
   |
19 |     if let Some(names_and_values) = input.namesAndValues() {
   |                                           ^^^^^^^^^^^^^^ method not found in `extendr_api::Robj`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: could not compile `data`

To learn more, run the command again with --verbose.
make: *** [../target/release/libdata.a] Error 101
ERROR: compilation failed for package ‘data’
* removing ‘/Users/clauswilke/Library/R/4.0/library/data’

In general, I think we should move the examples elsewhere, e.g. into the integration tests. Since we don't automatically test examples, they may break at any moment.

From<String> missing for Robj?

It looks like a From<String> implementation is missing for Robj.

I tried to compile the following code:

use extendr_api::*;

#[extendr]
fn md_to_html(input: &str) -> String {
    markdown::to_html(input)
}

and got this compiler error:

error[E0277]: the trait bound `extendr_api::robj::Robj: std::convert::From<std::string::String>` is not satisfied
 --> src/lib.rs:3:1
  |
3 | #[extendr]
  | ^^^^^^^^^^ the trait `std::convert::From<std::string::String>` is not implemented for `extendr_api::robj::Robj`
  |
  = help: the following implementations were found:
            <extendr_api::robj::Robj as std::convert::From<&'a [&str]>>
            <extendr_api::robj::Robj as std::convert::From<&'a [extendr_api::logical::Bool]>>
            <extendr_api::robj::Robj as std::convert::From<&'a [i32]>>
            <extendr_api::robj::Robj as std::convert::From<&'a str>>
          and 22 others
  = note: required by `std::convert::From::from`
  = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

Alternatively, the following code compiles:

use extendr_api::*;

#[extendr]
fn md_to_html(input: &str) -> Robj {
    Robj::from(&*markdown::to_html(input))
}

Module registration currently doesn't work

In trying to compile an R package using extendr on Windows, I have found that the module registration doesn't work. I believe the problem is in this line:

extendr/extendr-macros/src/lib.rs

Line 347 in 4177272

let module_init_name = format_ident!("R_init_lib{}", modname);

The function to be called should be named R_init_<modname>, not R_init_lib<modname>. See e.g. here:
https://github.com/r-rust/hellorust/blob/0b53c737ead980f58fe9584fbf513f15fdfe0582/src/wrapper.c#L30

When doing manual, C-style registration the checks work:
https://github.com/extendr/helloextendr/runs/1561836399?check_suite_focus=true

When doing the current module registration the Windows check doesn't work, the package compiles but then the exported function isn't found:
https://github.com/extendr/helloextendr/runs/1561757762?check_suite_focus=true#step:11:251

For some reason, Linux and macOS don't care whether the functions are properly registered or not.

Add Rprintln macro

It would be great if there was an Rprintln!() macro that would function exactly like regular println!() but use Rprintf() to send the output to R. In addition, it might be useful to add REprintln!() to replace eprintln!() for errors.

Relevant documentation from Writing R Extensions: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Printing

Release version 0.2.0

I'd like to release a version 0.2.0 that provides a minimally viable product on all platforms. Below is the checklist for the release. If I'm forgetting anything, please let me know in the comments.

• Address all issues/PRs in the 0.2.0 milestone
• Finalize and release libR-sys 0.2.0 (extendr/libR-sys#20)
• Fix bump.sh and publish.sh (#70) (see #82)
• Bump libR-sys dependency to latest version
• Remove any patch_crates.io statements from top-level Cargo.toml
• Bump version to 0.2.0
• Bump version number in changelog
• Verify that all crates in the workspace have the same version and the same extendr version dependencies (see #82)
• Tag release on github
• Release to crates.io

Happy Christmas everyone!

May you have an excellent winter solstice related festival of your personal choice.

Many thanks, all of you for contributing this year and I hope that the next year
will see some good results for all of us.

Different behavior regarding strings in data.frame between R 3.6.3 and R 4.0.3

I am seeing strange behavior with data.frame between R 3.6.3 and R 4.0.3. I have reproduced this on both Ubuntu 20.04.1 and Mac OS 11.0.1. I am using extendr version 0.1.10.

In R 3.6.3 the following code prints out: [[1, 2, 3], [3.0, 4.0, 5.0], [6, 7, 8]]
In R 4.0.3 the following code prints out: [["abcd", "def", "fg"], [3.0, 4.0, 5.0], [6, 7, 8]]

I have spent a few hours at this. Any suggestions on where to look to debug this would be most grateful.

fn main() {
    start_r();
    let x_vec = &["abcd", "def", "fg"][..];
    let y_vec = &[3.0, 4.0, 5.0][..];
    let z_vec = &[6, 7, 8][..];

    let data_frame = data_frame!(x = x_vec.clone(), y = y_vec.clone(), z = z_vec.clone());

    println!("{:?}", data_frame);

    let c = data_frame.list_iter().unwrap().collect_vec();
    if let Some(v) = c[0].str_iter() {
        // Evaluated in R 4.0.3
        // v.collect_vec() is ["abcd", "def", "fg"]
        assert_eq!(v.collect_vec(), x_vec);
    } else {
        // c[0] evaluates to [1,2,3] and is of type i32
        // str_iter() fails in R 3.6.3
    }
    assert_eq!(c[1].as_f64_slice(), Some(y_vec));
    assert_eq!(c[2].as_i32_slice(), Some(z_vec));
    end_r();
}

Windows CI failing with mingw-w64-x86_64-pkg-config and mingw-w64-x86_64-pkgconf are in conflict.

Windows CI failing with mingw-w64-x86_64-pkg-config and mingw-w64-x86_64-pkgconf are in conflict.

Do we need pkg-config anymore?

We should just remove it.

Extendr fails unit tests on R devel

The test suite does not currently run without error on R devel. See e.g. here:
https://github.com/extendr/extendr/runs/1579358171

This is not related to whether or not libR-sys is compiled with bindgen or uses precompiled bindings.

How to handle errors in extension code?

C extensions call the error macro (which just wraps Rf_error) to raise R exception. What should Rust extensions do? Calling panic! might be feasible given a catch_unwind in the generated wrapper, but doesn't seem particularly in-line with general Rust preferences as I understand them.

Could plausibly take the same approach as C, i.e. call Rf_error (presumbly via a println!-like macro for convenient error-message formatting). My concern with this is that Rf_error doesn't return (under the hood, it does a LONGJMP I believe), so I think this means that a function that's calling it won't necessarily drop objects that are in scope at the time of the call (so a potential leak). Would be happy if someone can tell me either than Rust FFI has a way of annotating functions which don't return, or that I'm just over-thinking this!

An alternative model might be to allow wrapped functions to return a Result<X, Error>, and have the wrapper call Rf_error instead. This seems the more "Rusty" approach to me, but again happy to be contradicted. There's a simple proof-of-concept of this latter aproach in dasmoth@d38e9e7 in case of interest.

License of logo

Rust's logo is licensed as CC-BY 4.0 (c.f. https://github.com/rust-lang/rust-artwork) and R's logo as CC-BY-SA 4.0 (c.f. https://www.r-project.org/logo/). As both licenses allow modification, so there's no license problem with extendr's logo (as long as we distribute the logo as CC-BY-SA 4.0), but probably we need proper attributions to the original copyright holder somewhere. LICENSE or README?

Progress.

Sorry for the lack of progress this week. I have badly broken my right arm and am limited to left-hand typing! Surgery (Arthroplasty) today or tomorrow, hopefully, depending on the waiting list.

Compiling some examples of R plugins is going to be a priority.

We should liase with rust-bio to wrap their more mature features.

Andy.

Guide for pre-compiling binaries using GitHub Action CI

(Sorry if this isn't the right place to ask)

As probably you already notice, the installation instruction is a bit complicated to users who just use a package that uses extendr, especially if they are on Windows.

https://github.com/extendr/libR-sys#installation

My experimental package prepares pre-compiled binaries for Windows so that the user can avoid compilation on their local.

c.f. "Precompiled binary for Windows" section of https://yutani.rbind.io/post/some-more-notes-about-using-rust-code-in-r-packages/

This is what rwinlib does for many C/C++ packages. I don't find any elegant way of doing this, but I think it's important to provide some easy way to install from GitHub in order for a pre-CRAN package to get used by ordinary R users. So, it's nice if you provide some guidance of how to setup GitHub Actions CI for this.

Remove/update r_output_test for Windows compatibility

I stumbled upon this issue when trying to run tests on Windows.
The tests can actually be executed once linked against dev version of libR-sys, including PR #12 in that repo.

I observed that the last out of 8 tests, r_output_test, simply freezes while being executed.
This was confirmed at least on two separate Windows machines.
I was able to pinpoint this issue to bugged fifo function in R (documentation says it is supported under Windows, yet it does not work as expected) .

R repro

fi <- fifo("")
sink(fi)
cat("Hello world!") # This call freezes R process
sink()

So this test can be either removed or replaced with something like this:

r_output_test

 #[test]
    fn r_output_test() {
        let con_target = "test_con";
        let txt_con = lang!("textConnection", Robj::from(con_target) , open = Robj::from("w"))
            .eval().unwrap();
        let con = unsafe { txt_con.get() };
        lang!("sink", txt_con).eval_blind();
        rprintln!("Hello world");
        lang!("sink").eval_blind();
        let result : Robj = lang!("get", Robj::from(con_target)).eval_blind();
        assert_eq!(result, Robj::from("Hello world"));
        lang!("close", con).eval_blind();
    }

Add documentation for lang! macro

It looks like the lang! macro is a core concept of how extendr works. Unfortunately, it's not clear to me what exactly it does, and the documentation is rather short:

extendr/extendr-api/src/args.rs

Lines 85 to 87 in 8114138

	/// A macro for constructing R langage objects.
	#[macro_export]
	macro_rules! lang {

It would be great if a bit more explanation could be added, and maybe 2-3 different examples of how it can be used.

Compiling rust using the Rtools toolchain (mingw) on Windows?

I huge barrier to entry is the whole needing a compiler thing.
With the new Rtools4.0 I believe it might be doable to configure rustup/rust to specifically
build for the target that may be built using the compiler in Rtools.

I'll have to investigate this further. Let me know if anyone has already (or is already) doing this.
A wiki-page on it could be nice.