Unofficial Introduction To extendr (1): Your First R Package With Rust

Rust extendr

Integrate R and Rust with extendr.

Hiroaki Yutani true
06-06-2021

extendr is a project that provides an interface between R and Rust. While I’m also a member of this project, my contributions were small and I’m still a newbie to Rust, so let me add “Unofficial” to the title at the moment :P

For the full introduction, including the motivation to use Rust, please watch this great presentation by Claus Wilke:

The talk covers almost everything (history of the project, getting started with Rust, and how to compile a Rust function into an R function), so, instead of providing an overview, this post will focus on the one missing topic, how to write an R package using (r)extendr.

Rust crates and R packages under extendr

That said, to avoid confusion, probably I need to write a minimal introduction about the crates and packages we’ll look into… Let’s take a look briefly.

libR-sys (Rust crate)

libR-sys is low-level bindings to R’s C API, and powers extendr. This is not what the ordinary users use directly, so we can forget this name for now.

extendr (Rust crate)

extendr is a user-friendly Rust interface to R. To be precise, “extendr” itself is not a crate’s name; it’s extendr-api that we’ll actually use. There are some more crates that might be needed in some use cases, but let’s forget them for now.

rextendr (R package)

rextendr is an R package. This provides two types of functions:

  1. Functions for compiling and running Rust code on the fly (e.g. rust_function())
  2. Functions for developing R package using extendr

What we’ll use in this post is type 2.

Setup

Rust

First of all, you need Rust.

If you are using macOS or Linux, you can follow the official guide. That’s all.

If you are using Windows, extendr requires you to install the latest Rtools, and use the following toolchains:

rustup default stable-msvc
rustup target add x86_64-pc-windows-gnu  # 64-bit
rustup target add i686-pc-windows-gnu    # 32-bit

For the details, please refer to the installation instructions on libR-sys’s repo.

rextendr

rextendr is not on CRAN at the time of writing this blog post, so please install it from GitHub.

remotes::install_github("extendr/rextendr")

Create a template package

Creating an R package with extendr is very easy with usethis and rextendr.

First, create an R package by using usethis::create_package() as usual.

usethis::create_package("path/to/my1stextendrpkg")

Then, create the scaffolding to use extendr. This can be done with rextendr::use_extendr().

rextendr::use_extendr()
✓ Creating src/rust/src.
✓ Setting active project to 'path/to/my1stextendrpkg'
✓ Writing 'src/entrypoint.c'
✓ Writing 'src/Makevars'
✓ Writing 'src/Makevars.win'
✓ Writing 'src/.gitignore'
✓ Writing src/rust/Cargo.toml.
✓ Writing 'src/rust/src/lib.rs'
✓ Writing 'R/extendr-wrappers.R'
✓ Finished configuring extendr for package my1stextendrpkg.
• Please update the system requirement in DESCRIPTION file.
• Please run `rextendr::document()` for changes to take effect.

Done! Now we are just one step away (as the message says, we need to run rextendr::document()) from calling Rust fucntions from R. But, before moving forward, let’s look at the files added.

Package structure

The below files are the ones rextendr::use_extendr() added.

.
├── R
│   └── extendr-wrappers.R
...
└── src
    ├── Makevars
    ├── Makevars.win
    ├── entrypoint.c
    └── rust
        ├── Cargo.toml
        └── src
            └── lib.rs

So, in short, what we should really look at is only these two files:

src/rust/Cargo.toml

[package]
name = 'my1stextendrpkg'
version = '0.1.0'
edition = '2018'

[lib]
crate-type = [ 'staticlib' ]

[dependencies]
extendr-api = '*'

The create name is the same name as the R package’s name by default. You can change this, but it might be a bit tired to tweak other files accordingly, so I recommend leaving this.

To try the dev version of the extendr, you can modify the last line to

extendr-api = { git = 'https://github.com/extendr/extendr' }

src/rust/src/lib.rs

use extendr_api::prelude::*;

/// Return string `"Hello world!"` to R.
/// @export
#[extendr]
fn hello_world() -> &'static str {
    "Hello world!"
}

// Macro to generate exports.
// This ensures exported functions are registered with R.
// See corresponding C code in `entrypoint.c`.
extendr_module! {
    mod my1stextendrpkg;
    fn hello_world;
}

Let’s explain this part by part.

The first line use extendr_api::prelude::*; loads functions used frequently.

Next, your eyes might notice the / are repeated 3 times, while the usual Rust comment requires only twice (i.e. //). These are treated as roxygen comments and copied to the auto-generated R code. This is analogous to Rcpp/cpp11’s //'.

/// Return string `"Hello world!"` to R.
/// @export

The next line is the core of extendr’s mechanism. If the function is marked with this macro, the corresponding R function will be generated automatically (I’ll explain the detail later). This is analogous to Rcpp’s [[Rcpp::export]] and cpp11’s [[cpp11::register]].

#[extendr]

The last 3 lines are the macro for generating exports, as the comment explains. If we implement another function than hello_world, it needs to be listed here as well as marking it with #[extendr] macro.

extendr_module! {
    mod my1stextendrpkg;
    fn hello_world;
}

Compile and use the package

Compile

Compiling Rust code into R functions is as easy as this one command:

rextendr::document()
✓ Saving changes in the open files.
ℹ Generating extendr wrapper functions for package: my1stextendrpkg.
! No library found at src/my1stextendrpkg.so, recompilation is required.
Re-compiling my1stextendrpkg
─  installing *source* package ‘my1stextendrpkg’ ... (347ms)
   ** using staged installation
   ** libs
   rm -Rf my1stextendrpkg.so ./rust/target/release/libmy1stextendrpkg.a entrypoint.o
   gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-tbZjLv/r-base-4.1.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c entrypoint.c -o entrypoint.o
   cargo build --lib --release --manifest-path=./rust/Cargo.toml
       Updating crates.io index
      Compiling proc-macro2 v1.0.27
      Compiling unicode-xid v0.2.2
      Compiling libR-sys v0.2.1
      Compiling syn v1.0.72
      Compiling extendr-engine v0.2.0
      Compiling lazy_static v1.4.0
      Compiling quote v1.0.9
      Compiling extendr-macros v0.2.0
      Compiling extendr-api v0.2.0
      Compiling my1stextendrpkg v0.1.0 (path/to/my1stextendrpkg/src/rust)
       Finished release [optimized] target(s) in 19.05s
   gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o my1stextendrpkg.so entrypoint.o -L./rust/target/release -lmy1stextendrpkg -L/usr/lib/R/lib -lR
   installing to /tmp/RtmpfMcL08/devtools_install_e2d6351b843c/00LOCK-my1stextendrpkg/00new/my1stextendrpkg/libs
   ** checking absolute paths in shared objects and dynamic libraries
─  DONE (my1stextendrpkg)
✓ Writing 'R/extendr-wrappers.R'.
ℹ Updating my1stextendrpkg documentation
ℹ Loading my1stextendrpkg
Writing NAMESPACE
Writing NAMESPACE
Writing hello_world.Rd

You might wonder why compilation is triggered while the function name is just document(). Well, this is because the compilation is actually needed to generate document from Rust code. This is consistent with devtools::document()’s behavior for C/C++ codes1.

Anyway, by doing above, the following files are updated or generated:

.
...
├── NAMESPACE                       ----------(4)
├── R
│   └── extendr-wrappers.R          ----------(3)
├── man
│   └── hello_world.Rd              ----------(4)
└── src
    ├── my1stextendrpkg.so          ----------(2)
    └── rust
        └── target
            └── release
                ├── libmy1stextendrpkg.a   ---(1)
                ...
  1. src/rust/target/release/libmy1stextendrpkg.a (the extension depends on the OS): This is the static library built from Rust code. This will be then used for compiling shared library my1stextendrpkg.so.
  2. src/my1stextendrpkg.so (the extension depends on the OS): This is the shared object that is actually called from R.
  3. R/extendr-wrappers.R: The auto-generated R functions, including roxygen comments, goes to this file. The roxygen comments are accordingly converted into Rd files and NAMESPACE.
  4. man/, NAMESPACE: These are generated from roxygen comments.

Load and use

As all things are done by rexetndr::document() already, we can just load it (or install it if you want) and call the function.

devtools::load_all(".")

hello_world()
[1] "Hello world!"

Achievement unlocked, you called a Rust function from R!

Rust code vs generated R code

We don’t open R/extendr-wrappers.R yet. While we never edit this file by hand, it might be good to know what R code is generated from a Rust code. Here it is:

# Generated by extendr: Do not edit by hand
#
# This file was created with the following call:
#   .Call("wrap__make_my1stextendrpkg_wrappers", use_symbols = TRUE, package_name = "my1stextendrpkg")

#' @docType package
#' @usage NULL
#' @useDynLib my1stextendrpkg, .registration = TRUE
NULL

#' Return string `"Hello world!"` to R.
#' @export
hello_world <- function() .Call(wrap__hello_world)

.Call("wrap__make_my1stextendrpkg_wrappers", use_symbols = ... is what was actually done inside rextendr::document().

A section of @docType package is needed to generate useDynLib(my1stextendrpkg, .registration = TRUE) entry in NAMESPACE.

The last section is for hello_world(). We can see the roxygen comments are copied to here. As the Rust function hello_world() has no arguments so this R function also has no arguments. If the function is like this,

fn add(x: i32, y: i32) -> i32 {
    x + y
}

then the generated function also has arguments like this:

add <- function(x, y) .Call(wrap__add, x, y)

Implement a new Rust function

Now that we roughly figured out how extendr works (hopefully!), let’s implment a new Rust function. The development flow would be:

  1. Modify src/rust/src/lib.rs
  2. Run rextendr::document()
  3. Run devtools::load_all(".") and test the function

As an exercise, let’s add add(i32, i32) I showed above.

1. Modify src/rust/src/lib.rs

Add the function with @export.

/// @export
#[extendr]
fn add(x: i32, y: i32) -> i32 {
    x + y
}

Don’t forget to add the function to extendr_module!.

extendr_module! {
    mod my1stextendrpkg;
    fn hello_world;
    fn add;
}

2. Run rextendr::document()

Just run the command:

rextendr::document()

3. Run devtools::load_all(".") and test the function

Ditto.

devtools::load_all(".")

add(1L, 2L)
[1] 3

Achievement unlocked, you called a Rust function you implemented from R!

What’s not covered in this post

I hope this post illustrates how easy it is to get started with extendr to create an R package. But, if you play with the function, you might get wondered about the topics that this post doesn’t cover.

For example, while the signature is i32 (i.e. integer), the function also accepts numeric. What’s the rule behind this coercion? (Confession: I need to study to answer this question…)

add(1, 2)
[1] 3

Another question might be how to handle vectors. This function accepts only length-one vectors. Otherwise, it errors. (Spoiler: This is very simple; we can use Vec<_>. But, let me leave this topic to the next post…)

add(1:2, 2:3)
Error in add(1:2, 2:3): Input must be of length 1. Vector of length >1 given.

I’ll try explaining these in the next post. Stay tuned…

Until then, you can ask questions on extendr’s Discord, which you can find on https://github.com/extendr/extendr#contributing :)


  1. The current mechanism under devtools::document() doesn’t have extensible mechanism for other languages than C/C++ (c.f. r-lib/pkgbuild#115), so we needed to have our own one.

    ↩︎

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".