ropensci/crul

R6 based http client for R (for developers)

crul

An HTTP client, taking inspiration from Ruby's faraday and Python's requests

Package documentation: https://docs.ropensci.org/crul/

Some Features:

• HttpClient - Main interface to making HTTP requests. Synchronous requests only.
• HttpResponse - HTTP response object, used for all responses across the different clients.
• Paginator - Auto-paginate through requests - supports a subset of all possible pagination scenarios - will fill out more scenarios soon
• Async - Asynchronous HTTP requests - a simple interface for many URLS - whose interface is similar to HttpClient - all URLs are treated the same.
• AsyncVaried - Asynchronous HTTP requests - accepts any number of HttpRequest objects - with a different interface than HttpClient/Async due to the nature of handling requests with different HTTP methods, options, etc.
• set curl options globally: set_auth(), set_headers(), and more
• Writing to disk and streaming: available with both synchronous requests as well as async requests
• Hooks on requests and responses are available in the HttpClient method only, and allow you to trigger functions to run on requests or responses, or both. See ?hooks for the details and examples
• Mocking: crul integrates with webmockr to mock HTTP requests. Checkout the http testing book
• Test caching: crul also integrates with vcr to cache http requests/responses. Checkout the http testing book

Installation

CRAN version

install.packages("crul")

Latest binaries from rOpenSci

install.packages("crul", repos = "https://dev.ropensci.org")

Dev version from GitHub

install.packages("pak")
pak::pak("ropensci/crul")

library("crul")

Meta

• Please report any issues or bugs.
• License: MIT
• Get citation information for crul in R doing citation(package = 'crul')
• Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
• Where does the package name come from? It was a play on "curl", the popular command line client.
• Where does the sticker design come from? The sticker idea arose from a tweet - crul is close (ish) to Krull, a 1980's movie with a "mystical five-pointed weapon". The association with Krull was not known before naming the package.

crul's Issues

test suite

use Accept-Encoding: gzip, deflate on all methods by deafult, not just GET

vignette

AsyncVaried behave well when requests not made yet

when users access objects/functions

Vignette for using curl options

tidy client request preparation

... method for passing in curl options

form/multipart requests for post/put/patch/delete

async - parallelization use cases

via https://stackoverflow.com/questions/45573770/how-do-i-perform-parallel-asynchronous-post-api-calls-in-r/45575252

url encode params in make_url helper fxn

curl opts: drop any named options that aren't in set of curl options

good way to eliminate some problems

better docs for HttpRequest

make sure users know about directly accessing verbs on the object definition .e.g, get()/post()

Add stream support

move to markdown docs

webmockr integration

make file uploads work

urlencode paths

cause they can be things like stuff things

new man files for different http tasks

cainfo

header parsing not working on FTP servers

presumably all

headers aren't returned in AsyncVaried

replace vignettes with prebuilt versions

simle auth helper

set curl options globally

main use case is when using crul inside a package, becomes esp. important when e.g., a pkg foo uses crul, then another pkg bar uses foo, which makes likelihood of arg. collisions more likely

sticker idea

https://twitter.com/hadleywickham/status/875554298729574400

http://krull.wikia.com/wiki/Glaive

Add async support

Missing raise_for_status method

raise_for_status present in examples for the HttpResponse,

Support different url modifiers

port, scheme, etc.

useragent -> User-Agent

multiple query params of the same name not allowed now

fix it

export fxn to build urls

and maybe parse

write disk helper

query passed to `head` gives error

due to new curl options checker - porobably put that param back

Set custom user agent and proxy

Hi.

Many thanks for the nice package.

How can I set a some custom options for the requests? For example user agent and proxy server.

get/set cookie?

Hi,

I have a use case where I need to set up cookie value when running GET/POST method (for a tmp authentication purpose). Is there a way to do so in this package?

possibly make flow chart for deciding what to use for which use case

fix put in httpclient/httprequest

incorporate fauxpas

https://github.com/sckott/fauxpas

probably in Suggests though so users optionally use it

make sure user supplied headers override default headers

parse response headers to a list

remove webmockr stuff for now

put back in once on CRAN

Async response objects sometimes point to wrong requests objects

When making async requests, responses come back unordered which is fine. However it seems like the associated request objects accessible via $request are always ordered in the original way.

This means that in case responses do not come back in the same order as the original requests, the associated request of the respective response might be wrong.

Here is a minimal example that sometimes works and sometimes not:

library(crul)
(cc <- Async$new(
  urls = c(
    'https://httpbin.org/get?a=5',
    'https://httpbin.org/get?a=5&b=6',
    'https://httpbin.org/ip'
  )
))
(res <- cc$get())

# the following line prints all urls of the original request object
print(sapply(res, function(x) x$request$url))
# this prints all response urls
print(sapply(res, function(x) x$url))
# those two values should be equal I assume

R version 3.4.0 (2017-04-21)
Platform: x86_64-apple-darwin16.5.0 (64-bit)
Running under: macOS Sierra 10.12.5

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] crul_0.3.6

loaded via a namespace (and not attached):
 [1] compiler_3.4.0  R6_2.2.1        tools_3.4.0     withr_1.0.2    
 [5] rstudioapi_0.6  curl_2.6        memoise_1.1.0   Rcpp_0.12.11   
 [9] urltools_1.6.0  triebeard_0.3.0 digest_0.6.12   devtools_1.13.1

Documentation not clear about optional parameters for authentificatio

In the documentation about crul::auth() is said to have all its argument optional.

However, crul::auth(auth = "any) send back an error

Error in assert(user, "character") : 
  argument "user" is missing, with no default

it is because crul:::make_up does a check abour argument user and pwd which should not be empty

make_up <- function (user, pwd) 
{
  assert(user, "character")
  assert(pwd, "character")
  if (!is.null(user) || !is.null(pwd)) {
    return(paste0(user, ":", pwd))
  }
  NULL
}

It seems that it is expected to be set to NULL considering the code above the the crul:::cpp function.
I check : with auth(user = NULL, pwd = NULL, auth = "any") it is working.

It could be useful if this is explained in the documentation help page or maybe if the arguments are set to NULL by default.

Either way, I can help with a PR if you want to change thing

gzip encoding by default

fix delete - shouldn't pass a content-type if no body given

maybe body = FALSE by default

make sure request headers are returned correctly in response

handles reuse

[Question] Async requests to disk

Hi,

How can I use the async feature with disk or stream params for the GET method?

~~
Regards

implement progress tool

in mock() make sure webmockr is installed

and in HttpClient check again before using webmockr