senseyeio/roger

Golang RServe client. Use R from Go

Roger

Roger is a Go RServe client, allowing the capabilities of R to be used from Go applications.

The communication between Go and R is via TCP. It is thread safe and supports long running R operations synchronously or asynchronously (using channels).

package main

import (
	"fmt"

	"github.com/senseyeio/roger"
)

func main() {
	rClient, err := roger.NewRClient("127.0.0.1", 6311)
	if err != nil {
		fmt.Println("Failed to connect")
		return
	}

	value, err := rClient.Eval("pi")
	if err != nil {
		fmt.Println("Command failed: " + err.Error())
	} else {
		fmt.Println(value) // 3.141592653589793
	}

	helloWorld, _ := rClient.Eval("as.character('Hello World')")
	fmt.Println(helloWorld) // Hello World

	arrChan := rClient.Evaluate("Sys.sleep(5); c(1,1)")
	arrResponse := <-arrChan
	arr, _ := arrResponse.GetResultObject()
	fmt.Println(arr) // [1, 1]
}

Response Type Support

Roger currently supports the following response types from R:

• string and string arrays
• booleans and boolean arrays
• doubles and double arrays
• ints and int arrays
• complex and complex arrays
• lists
• raw byte arrays

With the use of JSON, this capability can be used to transfer any serializable object. For examples see sexp_parsing_test.go.

Assignment Support

Roger allows variables to be defined within an R session from Go. Currently the following types are supported for variable assignment:

• string and string arrays
• byte arrays
• doubles and double arrays
• ints and int arrays

For examples see assignment_test.go.

Setup

Rserve should be installed and started from R:

install.packages("Rserve")
require('Rserve')
Rserve()

More information is available on RServe's website.

If you would like to exploit the current R environment from go, start RServe using the following command:

install.packages("Rserve")
require('Rserve')
run.Rserve()

Install Roger using:

go get github.com/senseyeio/roger

Testing

To ensure the library functions correctly, the end to end functionality must be tested. This is achieved using Docker and Docker Compose. To run tests, ensure you have both Docker and Docker Compose installed, then run docker-compose build && docker-compose up -d from within the test directory. This command will build and start a docker container containing multiple RServe servers. These servers will be utilized when running go test from the project's base directory. To stop the docker container call docker-compose stop from the test directory.

Contributing

Issues, pull requests and questions are welcomed. If required, assistance can be found in the project's gitter chat room.

Pull Requests

• Fork the repository
• Make changes
• Ensure tests pass
• Raise pull request

roger's Issues

isLong in sexp causes index out of range error

This line throws an error in case of big outputs:

https://github.com/senseyeio/roger/blob/master/sexp/factory.go#L19

I am trying to export ~30 Mbytes JSON output from R.

Commenting this block out solves the issue.

Could you spare some info?

Support XT_LIST_TAG lists

Support XT_ARRAY_CPLX complex array

Allow assignments of lists

Is assignment of lists on the roadmap?

Large string assignment

Adding the following test in assignment_test.go fails due to the string length:

func TestLargeStringAssignment(t *testing.T) {
	checkAssignment(t, strings.Repeat("a", 20000000))
}

Support XT_VECTOR vectors

unsupported rserve version

Hi,

I recently upgraded roger to address #23 but now when I try to connect to our Rserve daemon roger reports:

The version of RServe installed is not officially supported. Please consider upgrading 
to the latest version of RServe.

In terms of the Rserve package version we're using a relatively recent install from cran.rstudio.com. sessionInfo() reports Rserve_1.7-3.
I'm relatively new to R but is there some way to install a newer version? What version is required? We don't install a specific version of Rserve so I'm wondering why our recent install is behind Roger's minimum version?

Thanks,
brian

Proposal: Please start using Semantic Versioning

I found that this project already supports Go modules. But sadly, the tags doesn't follow Semantic Versioning, which means that all tags of this project will be ignored by Go modules and replaced by pseudo-versions, go get acts weirdly when tags are not in that form. It would be great to have the tagged release be named in the format vX.X.X format so that go mod can read it.

	github.com/senseyeio/roger v0.0.0-20191009211040-43e330bee47f

Else the mod file shows something like github.com/senseyeio/roger v0.0.0-20191009211040-43e330bee47f which is not very readable and difficult to upgrade. It’s hard to verify which version is in use. This is not conducive to version control.

So, I propose this project to follow Semantic Versioning in future versions. For example, v1.0.1, v2.0.0, v3.1.0-alpha, v3.1.0-beta.2etc.

Unexpected end of buffer

Returning a list with a single entry causes an error:

"Warning: Error whilst constructing vector: Abruptly reached end of buffer"

Support XT_ARRAY_INT int array

Support XT_S4

panic when creating a list using bool or NA values

When sending a command to create a list I'm getting a panic: runtime error: index out of range with certain combinations of values. What I've observed is the following.

1. Creating a list of strings, ints, and floats in any combination works.
   Ex: list(int=1,string='s',float=0.5)

2. If the list has a bool or NA as the last value, the correct result is returned but there is a Warning: Error whilst constructing vector: Abruptly reached end of buffer outputted.
   Ex: list(int=1,string='s',float=0.5,bool=TRUE)
   Ex: list(var=NA)

3. If the list contains a bool or NA value with anything else following it, a panic happens.
   Ex: list(int=1,string='s',float=0.5,bool=TRUE,anything='this causes a panic')
   Ex: list(var1=NA,var2=1)

Here is the full stack trace for the panic:

panic: runtime error: index out of range

goroutine 1 [running]:
roger/sexp.getLength(0xc820431484, 0x38, 0x38, 0x29, 0x1, 0x420460, 0x0, 0x0)
    roger/sexp/factory.go:25 +0x12f
roger/sexp.parseReturningOffset(0xc820431484, 0x38, 0x38, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0)
    roger/sexp/factory.go:35 +0xa0
roger/sexp.parseVector(0x437840, 0xc820436960, 0xc820431484, 0x38, 0x38, 0x29, 0x38, 0x0, 0x0, 0x0, ...)
    roger/sexp/xt-vector.go:37 +0xba
roger/sexp.parseReturningOffset(0xc820431484, 0x38, 0x38, 0x20, 0x0, 0x0, 0x8591d, 0x0, 0x0)
    roger/sexp/factory.go:80 +0x752
roger/sexp.Parse(0xc820431484, 0x38, 0x38, 0x0, 0x0, 0x0, 0x0, 0x0)
    roger/sexp/factory.go:13 +0x61
roger.(*packet).GetResultObject(0xc820436930, 0x0, 0x0, 0x0, 0x0)
    roger/packet.go:77 +0x31e

Unsupported expression type: 23 (XT_LANG_TAG)

roger seems unable to handle this type. If I try:

  rClient.EvaluateSync("try(source('does not exist'))")

Based on this header it seems like the missing sexp type is XT_LANG_TAG.

E2E tests

Improve testing: benchmarks

Roger need benchmarks to examine the performance of go code.

Error on common.go

I'm having this issue:

$ go build

github.com/senseyeio/roger/assign

......\github.com\senseyeio\roger\assign\common.go:27: constant 4278190080 overflows int
......\github.com\senseyeio\roger\assign\common.go:45: constant 4278190080 overflows int

I believe this is because of data type int which can't handle the shift >> 24.

panic with echo disabled

Hi,

I can reliably generate a panic by disabling echo within a session:

    session, err := rClient.GetSession()
    if err != nil {
        log.Fatal(err)
    }
    defer session.Close()

    pkt := session.SendCommand(`options(echo = FALSE)`)
    v, err := pkt.GetResultObject()

results in

panic: runtime error: slice bounds out of range

goroutine 1 [running]:
_vendor/github.com/senseyeio/roger/sexp.parseReturningOffset(0xc82000e7e4, 0x2c, 0x2c, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/bfallik/sandbox/gillnet/go/src/_vendor/github.com/senseyeio/roger/sexp/factory.go:26 +0xc41
_vendor/github.com/senseyeio/roger/sexp.parseVector(0x13d6c0, 0xc82000e840, 0xc82000e7e4, 0x2c, 0x2c, 0x29, 0x2c, 0x0, 0x0, 0xc, ...)
    /Users/bfallik/sandbox/go/src/_vendor/github.com/senseyeio/roger/sexp/xt-vector.go:33 +0xbd
_vendor/github.com/senseyeio/roger/sexp.parseReturningOffset(0xc82000e7e4, 0x2c, 0x2c, 0x20, 0x0, 0x0, 0x7a15d, 0x0, 0x0)
    /Users/bfallik/sandbox/go/src/_vendor/github.com/senseyeio/roger/sexp/factory.go:68 +0x78e
_vendor/github.com/senseyeio/roger/sexp.Parse(0xc82000e7e4, 0x2c, 0x2c, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/bfallik/sandbox/go/src/_vendor/github.com/senseyeio/roger/sexp/factory.go:13 +0x61
_vendor/github.com/senseyeio/roger.(*packet).GetResultObject(0xc82000e810, 0x0, 0x0, 0x0, 0x0)
    /Users/bfallik/sandbox/go/src/_vendor/github.com/senseyeio/roger/packet.go:77 +0x31e
main.main()
    /Users/bfallik/sandbox/go/src/tools/rogercmd/main.go:49 +0x4ed

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1
exit status 2

keep session open for several command

Hi Daniel,

Now, in roger struct, defined address *net.TCPAddr, there is no session (or rconnection). Each Eval function will be closed and the variables evaluated will be released.
It make the code below not possible

r.Eval("x <- 1")


x, err := r.Eval(x) // since it is a new connection, x will not exit in R.



Do you think we can define sess *session in roger struct
And each r.Eval will use the same r.sess.
Of course define a r.Close() to close the connection when no r scripts will be evaluated.

Regards,
Jia

Support XT_ARRAY_BOOL bool array

Extremely long strings

Adding the following test to SEXP parsing test suite fails:

func TestReallyLongResponse(t *testing.T) {
	obj, err := getResultObject("paste(rep(\"a\", 16780000), sep=\"\", collapse = \"\")")
	assert.Nil(t, err)
	str, ok := obj.(string)
	assert.Equal(t, ok, true, "Return obj should be a string")
	assert.Equal(t, len(str), 16780000, "String length expected to be 16780000 characters")
}

Reporting unsupport RServe version for an unused but open port

Discovered during exploration for #25. If an connection is attempted to an incorrect port, the lib reports an unsupported RServe version. Should really be reporting an incorrect handshake.

Support authentication

Add TLS support

Since 1.7 Rserve supports TLS. Is supporting TLS on the roadmap of roger?

Support transfer of images

Improve error messages

The status code - error message map in packet.go does not provide detailed enough error messages. Sometimes error messages are misleading, as we receive the message "Command error with status: Unknown variable/method" even when the command failed due to other reasons.

A better approach could be to query the R session and return the message of last error such that the following test passes.

func TestErrorReturn(t *testing.T) {
    con, err := NewRClient("localhost", 6311)
    if err != nil {
        t.Error("Could not connect to RServe: " + err.Error())
        return
    }

    errorText := "this is a test"
    _, err = con.Eval("stop('" + errorText + "')")

    if err == nil {
        t.Error("No error was returned")
    }

    expectedMessage := "Command error with status: " + errorText
    if err.Error() != expectedMessage {
        t.Error("Expected '" + err.Error() + "' to equal '" + expectedMessage + "'")
    }
}

Introduce proper versioning

Support go modules

assign function

Hi Daniel,

Do you plan implement some basic assignment functions? Which can assign go variable values to R variable. May be start from string array -> string vector or others.
If do so, any idea how to programming. May be convert go values to strings with R eval format and evaluate the string?

Regards,
Jia