wojtekmach/req

Req is a batteries-included HTTP client for Elixir.

Req

Req is a batteries-included HTTP client for Elixir.

With just a couple lines of code:

Mix.install([
  {:req, "~> 0.5.0"}
])

Req.get!("https://api.github.com/repos/wojtekmach/req").body["description"]
#=> "Req is a batteries-included HTTP client for Elixir."

we get automatic response body decompression & decoding, following redirects, retrying on errors, and much more. Virtually all of the features are broken down into individual functions called steps. You can easily re-use and re-arrange built-in steps (see Req.Steps module) and write new ones.

Features

• An easy to use high-level API: Req.request/1, Req.new/1, Req.get!/2, Req.post!/2, etc.

• Extensibility via request, response, and error steps.

• Request body compression (via compress_body step)

• Automatic response body decompression (via compressed and decompress_body steps). Supports gzip, brotli, and zstd.

• Request body encoding. Supports urlencoded and multipart forms, and JSON. See encode_body.

• Aautomatic response body decoding (via decode_body step.)

• Encode params as query string (via put_params step.)

• Setting base URL (via put_base_url step.)

• Templated request paths (via put_path_params step.)

• Basic, bearer, and .netrc authentication (via auth step.)

• Range requests (via put_range) step.)

• Use AWS V4 Signature (via put_aws_sigv4) step.)

• Request body streaming (by setting body: enumerable.)

• Response body streaming (by setting into: fun | collectable | :self.)

• Follows redirects (via redirect step.)

• Retries on errors (via retry step.)

• Raise on 4xx/5xx errors (via handle_http_errors step.)

• Verify response body against a checksum (via checksum step.)

• Basic HTTP caching (via cache step.)

• Easily create test stubs (see Req.Test.)

• Running against a plug (via run_plug step.)

• Pluggable adapters. By default, Req uses Finch (via run_finch step.)

Usage

The easiest way to use Req is with Mix.install/2 (requires Elixir v1.12+):

Mix.install([
  {:req, "~> 0.5.0"}
])

Req.get!("https://api.github.com/repos/wojtekmach/req").body["description"]
#=> "Req is a batteries-included HTTP client for Elixir."

If you want to use Req in a Mix project, you can add the above dependency to your mix.exs.

Here's an example POST with JSON data:

iex> Req.post!("https://httpbin.org/post", json: %{x: 1, y: 2}).body["json"]
%{"x" => 1, "y" => 2}

You can stream request body:

iex> stream = Stream.duplicate("foo", 3)
iex> Req.post!("https://httpbin.org/post", body: stream).body["data"]
"foofoofoo"

and stream the response body:

iex> resp = Req.get!("http://httpbin.org/stream/2", into: IO.stream())
# output: {"url": "http://httpbin.org/stream/2", ...}
# output: {"url": "http://httpbin.org/stream/2", ...}
iex> resp.status
200
iex> resp.body
%IO.Stream{}

(See Req module documentation for more examples of response body streaming.)

If you are planning to make several similar requests, you can build up a request struct with desired common options and re-use it:

req = Req.new(base_url: "https://api.github.com")

Req.get!(req, url: "/repos/sneako/finch").body["description"]
#=> "Elixir HTTP client, focused on performance"

Req.get!(req, url: "/repos/elixir-mint/mint").body["description"]
#=> "Functional HTTP client for Elixir with support for HTTP/1 and HTTP/2."

See Req.new/1 for more information on available options.

Virtually all of Req's features are broken down into individual pieces - steps. Req works by running the request struct through these steps. You can easily reuse or rearrange built-in steps or write new ones. Importantly, steps are just regular functions. Here is another example where we append a request step that inspects the URL just before requesting it:

req =
  Req.new(base_url: "https://api.github.com")
  |> Req.Request.append_request_steps(
    debug_url: fn request ->
      IO.inspect(URI.to_string(request.url))
      request
    end
  )

Req.get!(req, url: "/repos/wojtekmach/req").body["description"]
# output: "https://api.github.com/repos/wojtekmach/req"
#=> "Req is a batteries-included HTTP client for Elixir."

Custom steps can be packaged into plugins so that they are even easier to use by others. Here are some examples:

• req_easyhtml
• req_s3
• req_hex
• req_github_oauth

And here is how they can be used:

Mix.install([
  {:req, "~> 0.5.0"},
  {:req_easyhtml, "~> 0.1.0"},
  {:req_s3, "~> 0.2.0"},
  {:req_hex, "~> 0.2.0"},
  {:req_github_oauth, "~> 0.1.0"}
])

req =
  (Req.new(http_errors: :raise)
  |> ReqEasyHTML.attach()
  |> ReqS3.attach()
  |> ReqHex.attach()
  |> ReqGitHubOAuth.attach())

Req.get!(req, url: "https://elixir-lang.org").body[".entry-summary h5"]
#=>
# #EasyHTML[<h5>
#    Elixir is a dynamic, functional language for building scalable and maintainable applications.
#  </h5>]

Req.get!(req, url: "s3://ossci-datasets/mnist/t10k-images-idx3-ubyte.gz").body
#=> <<0, 0, 8, 3, ...>>

Req.get!(req, url: "https://repo.hex.pm/tarballs/req-0.1.0.tar").body["metadata.config"]["links"]
#=> %{"GitHub" => "https://github.com/wojtekmach/req"}

Req.get!(req, url: "https://api.github.com/user").body["login"]
# output:
# paste this user code:
#
#   6C44-30A8
#
# at:
#
#   https://github.com/login/device
#
# open browser window? [Yn]
# 15:22:28.350 [info] response: authorization_pending
# 15:22:33.519 [info] response: authorization_pending
# 15:22:38.678 [info] response: authorization_pending
#=> "wojtekmach"

Req.get!(req, url: "https://api.github.com/user").body["login"]
#=> "wojtekmach"

See Req.Request module documentation for more information on low-level API, request struct, and developing plugins.

Presentations

• Req: A batteries-included HTTP client for Elixir - ElixirConf 2023, 2023-09-08
• Req: A batteries included HTTP client for Elixir - Elixir Kenya, 2022-08-26

Acknowledgments

Req is built on top of Finch and is inspired by cURL, Requests, Tesla, and many other HTTP clients - thank you!

License

Copyright (c) 2021 Wojtek Mach

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

req's Issues

`auth`: Support bearer auth

curl example:

$ curl --fail -H "Authorization: Bearer foo" https://httpbin.org/bearer
{
  "authenticated": true,
  "token": "foo"
}

We should support:

Req.get!("https://httpbin.org/bearer", auth: {:bearer, "foo"})

`retry`: Handle 429 Retry-After header

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

Add run_plugins step

This is another stab at #35. Instead of making plugins special, we'd just have a run_plugins(request, plugins) function which is a request step. It will be one of the default steps that is going to be added if :plugins option is set.

per process default options

If I have several long running gen servers that use req, they may need different base urls.

Well, I could just write a build function that loads base url from process state/dictionary and returns a basic request for further customization. Feel free to close.

New step: `put_aws_sigv4`

Similar to https://curl.se/docs/manpage.html#--aws-sigv4.

iex> Req.put!(url, body: body, aws_sigv4: [region: "us-east-1", service: "s3"])

`auth`: Support digest auth

curl example:

$ curl --fail --user foo:bar --digest https://httpbin.org/digest-auth/auth/foo/bar
{
  "authenticated": true,
  "user": "foo"
}

We should support:

Req.get!("https://httpbin.org/digest-auth/auth/foo/bar", auth: {:digest, "foo", "bar"})

`encode_body`: Set `Accept` header on `json: body`

Pluggable caching system?

Thanks for req! I love it.

The if_modified_since/2 caching is nice, but not all servers implement it correctly.

At times, within LiveBook for instance, I would like to be able to:

• cache for 1 day, no matter the result, as long as it is a 200
• cache based on some custom logic

I wonder if something could be done to have something pluggable that would let the user add a step, just like if_modified_since/2, but with custom logic.

Maybe it is already doable, I don't know yet! I wonder what your thoughts on that are.

function typespecs

normalize_headers: Rename to encode_headers

This way if we ever add the corresponding one for response headers, it's going to be called decode_headers.

New step: `compress`

This compresses the request body. Gzips the request body and sets the appropriate request header.

Usage:

Req.post!(url, body: "foo", compress: :gzip)

`follow_redirects`: Do not send authorization header on redirects to different hosts

See curl --locattion and curl --location-trusted for more information.

Request: rename finch* fields to private.req_finch*

New step: `progress`

Requires #10.

When running in IEx, by default we'll simply write to stdout:

iex> Enum.each(1..99, fn i -> IO.write "\r" <> String.pad_leading("#{i}%", 4, " ") ; Process.sleep(50) end)
  1%.. 2%..

there will also be an option to pass a callback to e.g. send a message somewhere.

get_private, put_private: Ensure keys are atoms

Let's do this now so that in the future we might support "paths", .eg.: put_private(request, [:foo, :bar], :baz).

`default_headers`: Break up into `put_user_agent` and `compressed` steps

Note, compressed is different than compress step (#25). Naming follows curl --compressed. It just sets the accept-encoding header. It can also be disabled by passing compressed: false. Being able to disable is especially useful because old apache versions incorrectly handle accept-encoding and if-modified-since, they don't return the 304 on cache hit.

New API

Currently we have two ways of making requests, the high- and low-level API:

Req.get(url)

Req.build(:get, url)
|> Req.prepend_default_steps()
|> Req.run()

We can extend Req by adding steps which is possible with the low-evel API. However, given we build-up the request pipeline starting with the method and the URL, the resulting struct isn't ideal to re-use for different invocations. (It doesn't hurt, but it somewhat feels dirty to then overwrite the method, the URL, etc.)

A simple change is to move things around. Instead of build/2 + run/0 we have build/0 + run/3:

Req.build()
|> Req.prepend_default_steps()
|> Req.run(:get, url)

Then there's also the plugins API proposal, #35. The idea was to encapsulate some common logic but more importantly make it super easy to use, without building up the request pipeline. Plugins are modules so we can very easily refer to them but then again from the day one we wanted to avoid modules in favour of anonymous functions or MFArgs, the steps. So plugins shouldn't cover for the limitations of the api, instead it should be more straightforward to achieve things with just steps, and then plugins are just sugar.

Finally we also have #34.

Putting this all together, I'm now thinking about this API:

# high-level

Req.get!(url)

Req.new()
|> Req.get!(url)

# low-level

Req.Request.new()
|> Req.Request.prepend_request_steps(...)
|> Req.Request.run()

# mix of high- and low-level

Req.new()
|> Req.Request.prepend_request_steps(...)
|> then(&IO.inspect/1)
|> Req.get!(url)

and regarding plugins we can do either or both:

Req.get!(url, plugins: [ReqS3])

Req.new()
|> ReqS3.call()
|> get!(url)

To sum up, we'd add these functions:

Req.new(options \\ [])
Req.get!(request, url, options)

We'd rename these:

Req.build/1 -> Req.Request.new/1
Req.{append,prepend}_{request, response,error}_steps -> Req.Response.*

and remove Req.prepend_default_steps in favour of Req.new/1. Req.new is basically Req.Request.new + prepend_default_steps.

With this, the get function and friends will become a bit more complicated now:

def get(url) when is_binary(url)

def get(url, options) when is_binary(url) and is_list(options)

def get(%Req.Request{} = request, url, options) when is_binary(url) and is_list(options)

Another complication is this. Lets say we want to enable retry and cache steps. We'll be able to do it in two places now:

Req.get(url, retry: true, cache: true)

Req.new(retry: true) |> Req.get!(url, cache: true)

So basically on Req.new we'd add all the default steps + the opt-in ones based on the options. On Req.get/2 we'd do the same. On Req.get/3 we'd additionally opt-in steps per options.

SO yeah it might be a bit complicated but I think it is still reasonable enough.

WDYT?

Move "Low-level" API to Req.Request

If we do #33, then we'd add 6 new functions (insert_{request,response,error}_step_{after,before}). Given the low-level API always takes the request pipeline as the first argument, Req.Request pipeline is a nice home for it. For a typical end-user that just wants to use Req, Req.<tab> would show up high-level API (e.g.: Req.get!) as well as steps (e.g. Req.decode/2) but not the low-level API so I think we'd keep them more focused on what they want to do at the moment. Not sure what to do about prepend_default_steps which is somewhat a bridge between high-level and low-level API.

Steps naming convention

Currently we have a mix of verbs and nouns: decode, range, netrc, if_modified_since. I think standardising on verbs would make things more consistent: put_range, load_netrc, put_if_modified_since, etc.

`auth`: Cache `.netrc` file read

It's inefficient to read and parse the file on every single request. Let's cache it based on mtime.

Add `:adapter` option

So that we can conveniently do: Req.get!(url, adapter: adapter). See #13 (comment). Not sure whether this should be on Req.request or Req.Steps.put_default_ateps though.

New step: `run_plug`

Example:

defmodule Foo do
  def init(opts), do: opts

  def call(conn) do
    Plug.Conn.send_resp(conn, 200, "foo")
  end
end

iex> Req.get("http://foo/bar", plug: Foo).body
"foo"

See https://www.python-httpx.org/advanced/#calling-into-python-web-apps

prepend_default_steps: Replace captures with MFArgs

Currently our request struct looks like this:

iex> Req.build(:get, "https://api.github.com") |> Req.prepend_default_steps()
%Req.Request{
  request_steps: [#Function<0.30967734/1 in Req."-fun.normalize_headers/1-">,
   #Function<1.30967734/1 in Req."-fun.default_headers/1-">,
   #Function<2.30967734/1 in Req."-fun.encode/1-">],
  response_steps: [#Function<9.30967734/2 in Req."-fun.follow_redirects/2-">,
   #Function<10.30967734/2 in Req."-fun.decompress/2-">,
   #Function<11.30967734/2 in Req."-fun.decode/2-">],
  ...
}

let's change add_default_steps/2 to use MFArgs so that request, response, and error steps are easier to understand:

%Req.Request{
  request_steps: [
    {Req, :normalize_headers, []},
    {Req, :default_headers, []},
    ...
  ],
  ...
}

we'll of course still continue to support passing anonymous functions.

`decompress_body`: Support transfer-encoding header

`auth`: Add `:netrc | {:netrc, path}` as possible values

Instead of having a distinct load_netrc step, make it an option to auth, e.g.: auth: :netrc which also would be the default?

Replace add_*_steps with {append,prepend}_*_steps

Protocol negotiation

We start Finch pool with default options and end up with an HTTP/1 pool. It would be nice to have protocol negotiation which afaik Finch doesn't support.

encode_headers: Encode header values

Requires #22

Currently we just encode header names. We should encode header values too.

iex> Req.get!(url, headers: [max_forwards: 10])
iex> Req.get!(url, headers: [if_unmodified_since: ~U[2021-01-01 00:00:00Z]])

For DateTime & NaiveDateTime we should convert it to "internet time". All other data types should go through String.Chars.to_string.

Request streaming

Add `Request.prepare`

This function will return a request that executed all the request steps. The response and error steps are yet to be executed of course.

Example:

iex> r = Req.prepare(method: :get, url: "http://example.com", params: [x: 1], auth: {"foo", "bar"})
iex> r.url
"http://example.com/?x=1"
iex> r.headers
[{"authorization", "Basic " <> ...}]
iex> Req.Request.run!(r)
%Req.Response{...}

New step: `output`

% curl -o foo.html https://elixir-lang.org/index.html
% curl --remote-name https://elixir-lang.org/index.html

Let's add:

iex> Req.get!(url, output: path)
iex> Req.get!(url, output: :remote_name)

Change meaning of step `{mod, args}`

Currently {mod, args} is the same as {mod, :run, args}. However, given it's a contract, we expect the run function, let's change it to mean {mod, :run, [opts]}, that is run has arity of 2.

Add `request.options` field

With this we could have steps that update options, e.g. load_global_options, load_dot_req, etc. The challenge is how this affects writing steps. Does:

def put_base_url(request, base_url) do

become:

def put_base_url(request) do
  base_url = request.options[:base_url]
  # ...
end

? how does this play out with enabling/disabling steps based on options, do we now always run steps, it is up to the steps to control whether they are no-op based on options?

Release 1.0.2

Hi @wojtekmach ,
I am having trouble using Req in an umbrella Phoenix app, due the mime dependency for the mime package is v2.0.
If you could make a release with at least this update which is already in mix.exs

Thank you.

New step: `fail`

Similar to curl --fail, we automatically raise on 4xx/5xx status codes. This is tricky because we have the retry step and it doesn't make sense to retry most 4xx errors.

New step: `trace`

This step will aid debugging, it will print which steps are executed.

Add Req.put!/3 and Req.delete!/2

Add put_base_url step

Requires #44

Example:

req = Req.new(base_url: "https://api.github.com")
Req.get!(req, "/repos/wojtekmach/req")

Support unix sockets

Requires https://github.com/keathley/finch/issues/139 and #30

Curl allows to do:

$ curl --unix-socket /var/run/docker.sock http:///v1.41/_ping
OK

Let's support that in Req too:

iex> Req.get!("http:///v1.41/_ping", unix_socket: "/var/run/docker.sock").body
"OK"

maybe the url could even simply be: "/v1.41/_ping" and we'd default it to http + localhost.

Insert step before/after a particular step

Once we have #16, we could add conveniences for inserting steps before/after a particular step.

Imagine you want to inspect the request pipeline state just before the Req.decode step. We can uniquely identify the step as either {:Req, :decode} or &Req.decode (we can ignore arity but should document it as such as well as nudge people towards using &Mod.fun captures to identify steps.) Thus we could have API like this:

|> Req.insert_request_step_before({Req, :decode}, &IO.inspect/1)

configurable default steps

something like default_options that permanently modifies what "default steps" are.

Maybe? Not sure if it's a good idea.

`follow_redirects`: Send a GET request after a 301..303

Currently we use the method of the original request. See curl --location. We should also support all the other 3xx codes provided they have the location header.

Response streaming

`follow_redirects`: Add `:max_redirects` option

New step: `parse_url`

In curl, these three invocations are equivalent:

$ curl http://localhost:4000/api
$ curl localhost:4000/api
$ curl :4000/api

let's add a parse_url step that will fill in the missing scheme and host.

Add load_dot_req step

Example:

# ~/.req.exs
[
  netrc: true,
  retry: true
]

retry: Rename max_attempts to max_retries

Rename request.uri to request.url

Rename encode/decode to *_body

`load_netrc`: Support NETRC environment variable

https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html