wojtekmach/req

Req is a batteries-included HTTP client for Elixir.

req's Issues

encode_headers: Encode header values

Requires #22

Currently we just encode header names. We should encode header values too.

iex> Req.get!(url, headers: [max_forwards: 10])
iex> Req.get!(url, headers: [if_unmodified_since: ~U[2021-01-01 00:00:00Z]])

For DateTime & NaiveDateTime we should convert it to "internet time". All other data types should go through String.Chars.to_string.

Rename encode/decode to *_body

New step: `fail`

Similar to curl --fail, we automatically raise on 4xx/5xx status codes. This is tricky because we have the retry step and it doesn't make sense to retry most 4xx errors.

Support unix sockets

Requires https://github.com/keathley/finch/issues/139 and #30

Curl allows to do:

$ curl --unix-socket /var/run/docker.sock http:///v1.41/_ping
OK

Let's support that in Req too:

iex> Req.get!("http:///v1.41/_ping", unix_socket: "/var/run/docker.sock").body
"OK"

maybe the url could even simply be: "/v1.41/_ping" and we'd default it to http + localhost.

New step: `progress`

Requires #10.

When running in IEx, by default we'll simply write to stdout:

iex> Enum.each(1..99, fn i -> IO.write "\r" <> String.pad_leading("#{i}%", 4, " ") ; Process.sleep(50) end)
  1%.. 2%..

there will also be an option to pass a callback to e.g. send a message somewhere.

Request streaming

prepend_default_steps: Replace captures with MFArgs

Currently our request struct looks like this:

iex> Req.build(:get, "https://api.github.com") |> Req.prepend_default_steps()
%Req.Request{
  request_steps: [#Function<0.30967734/1 in Req."-fun.normalize_headers/1-">,
   #Function<1.30967734/1 in Req."-fun.default_headers/1-">,
   #Function<2.30967734/1 in Req."-fun.encode/1-">],
  response_steps: [#Function<9.30967734/2 in Req."-fun.follow_redirects/2-">,
   #Function<10.30967734/2 in Req."-fun.decompress/2-">,
   #Function<11.30967734/2 in Req."-fun.decode/2-">],
  ...
}

let's change add_default_steps/2 to use MFArgs so that request, response, and error steps are easier to understand:

%Req.Request{
  request_steps: [
    {Req, :normalize_headers, []},
    {Req, :default_headers, []},
    ...
  ],
  ...
}

we'll of course still continue to support passing anonymous functions.

Pluggable caching system?

Thanks for req! I love it.

The if_modified_since/2 caching is nice, but not all servers implement it correctly.

At times, within LiveBook for instance, I would like to be able to:

• cache for 1 day, no matter the result, as long as it is a 200
• cache based on some custom logic

I wonder if something could be done to have something pluggable that would let the user add a step, just like if_modified_since/2, but with custom logic.

Maybe it is already doable, I don't know yet! I wonder what your thoughts on that are.

Add `:adapter` option

So that we can conveniently do: Req.get!(url, adapter: adapter). See #13 (comment). Not sure whether this should be on Req.request or Req.Steps.put_default_ateps though.

Add `Request.prepare`

This function will return a request that executed all the request steps. The response and error steps are yet to be executed of course.

Example:

iex> r = Req.prepare(method: :get, url: "http://example.com", params: [x: 1], auth: {"foo", "bar"})
iex> r.url
"http://example.com/?x=1"
iex> r.headers
[{"authorization", "Basic " <> ...}]
iex> Req.Request.run!(r)
%Req.Response{...}

Add `request.options` field

With this we could have steps that update options, e.g. load_global_options, load_dot_req, etc. The challenge is how this affects writing steps. Does:

def put_base_url(request, base_url) do

become:

def put_base_url(request) do
  base_url = request.options[:base_url]
  # ...
end

? how does this play out with enabling/disabling steps based on options, do we now always run steps, it is up to the steps to control whether they are no-op based on options?

Response streaming

New step: `put_aws_sigv4`

Similar to https://curl.se/docs/manpage.html#--aws-sigv4.

iex> Req.put!(url, body: body, aws_sigv4: [region: "us-east-1", service: "s3"])

Insert step before/after a particular step

Once we have #16, we could add conveniences for inserting steps before/after a particular step.

Imagine you want to inspect the request pipeline state just before the Req.decode step. We can uniquely identify the step as either {:Req, :decode} or &Req.decode (we can ignore arity but should document it as such as well as nudge people towards using &Mod.fun captures to identify steps.) Thus we could have API like this:

|> Req.insert_request_step_before({Req, :decode}, &IO.inspect/1)

get_private, put_private: Ensure keys are atoms

Let's do this now so that in the future we might support "paths", .eg.: put_private(request, [:foo, :bar], :baz).

New step: `run_plug`

Example:

defmodule Foo do
  def init(opts), do: opts

  def call(conn) do
    Plug.Conn.send_resp(conn, 200, "foo")
  end
end

iex> Req.get("http://foo/bar", plug: Foo).body
"foo"

See https://www.python-httpx.org/advanced/#calling-into-python-web-apps

Add load_dot_req step

Example:

# ~/.req.exs
[
  netrc: true,
  retry: true
]

New step: `trace`

This step will aid debugging, it will print which steps are executed.

New step: `compress`

This compresses the request body. Gzips the request body and sets the appropriate request header.

Usage:

Req.post!(url, body: "foo", compress: :gzip)

`follow_redirects`: Do not send authorization header on redirects to different hosts

See curl --locattion and curl --location-trusted for more information.

New API

Currently we have two ways of making requests, the high- and low-level API:

Req.get(url)

Req.build(:get, url)
|> Req.prepend_default_steps()
|> Req.run()

We can extend Req by adding steps which is possible with the low-evel API. However, given we build-up the request pipeline starting with the method and the URL, the resulting struct isn't ideal to re-use for different invocations. (It doesn't hurt, but it somewhat feels dirty to then overwrite the method, the URL, etc.)

A simple change is to move things around. Instead of build/2 + run/0 we have build/0 + run/3:

Req.build()
|> Req.prepend_default_steps()
|> Req.run(:get, url)

Then there's also the plugins API proposal, #35. The idea was to encapsulate some common logic but more importantly make it super easy to use, without building up the request pipeline. Plugins are modules so we can very easily refer to them but then again from the day one we wanted to avoid modules in favour of anonymous functions or MFArgs, the steps. So plugins shouldn't cover for the limitations of the api, instead it should be more straightforward to achieve things with just steps, and then plugins are just sugar.

Finally we also have #34.

Putting this all together, I'm now thinking about this API:

# high-level

Req.get!(url)

Req.new()
|> Req.get!(url)

# low-level

Req.Request.new()
|> Req.Request.prepend_request_steps(...)
|> Req.Request.run()

# mix of high- and low-level

Req.new()
|> Req.Request.prepend_request_steps(...)
|> then(&IO.inspect/1)
|> Req.get!(url)

and regarding plugins we can do either or both:

Req.get!(url, plugins: [ReqS3])

Req.new()
|> ReqS3.call()
|> get!(url)

To sum up, we'd add these functions:

Req.new(options \\ [])
Req.get!(request, url, options)

We'd rename these:

Req.build/1 -> Req.Request.new/1
Req.{append,prepend}_{request, response,error}_steps -> Req.Response.*

and remove Req.prepend_default_steps in favour of Req.new/1. Req.new is basically Req.Request.new + prepend_default_steps.

With this, the get function and friends will become a bit more complicated now:

def get(url) when is_binary(url)

def get(url, options) when is_binary(url) and is_list(options)

def get(%Req.Request{} = request, url, options) when is_binary(url) and is_list(options)

Another complication is this. Lets say we want to enable retry and cache steps. We'll be able to do it in two places now:

Req.get(url, retry: true, cache: true)

Req.new(retry: true) |> Req.get!(url, cache: true)

So basically on Req.new we'd add all the default steps + the opt-in ones based on the options. On Req.get/2 we'd do the same. On Req.get/3 we'd additionally opt-in steps per options.

SO yeah it might be a bit complicated but I think it is still reasonable enough.

WDYT?

`auth`: Support bearer auth

curl example:

$ curl --fail -H "Authorization: Bearer foo" https://httpbin.org/bearer
{
  "authenticated": true,
  "token": "foo"
}

We should support:

Req.get!("https://httpbin.org/bearer", auth: {:bearer, "foo"})

Add Req.put!/3 and Req.delete!/2

function typespecs

`auth`: Cache `.netrc` file read

It's inefficient to read and parse the file on every single request. Let's cache it based on mtime.

Rename request.uri to request.url

Steps naming convention

Currently we have a mix of verbs and nouns: decode, range, netrc, if_modified_since. I think standardising on verbs would make things more consistent: put_range, load_netrc, put_if_modified_since, etc.

Request: rename finch* fields to private.req_finch*

Protocol negotiation

We start Finch pool with default options and end up with an HTTP/1 pool. It would be nice to have protocol negotiation which afaik Finch doesn't support.

per process default options

If I have several long running gen servers that use req, they may need different base urls.

Well, I could just write a build function that loads base url from process state/dictionary and returns a basic request for further customization. Feel free to close.

Replace add_*_steps with {append,prepend}_*_steps

`encode_body`: Set `Accept` header on `json: body`

Move "Low-level" API to Req.Request

If we do #33, then we'd add 6 new functions (insert_{request,response,error}_step_{after,before}). Given the low-level API always takes the request pipeline as the first argument, Req.Request pipeline is a nice home for it. For a typical end-user that just wants to use Req, Req.<tab> would show up high-level API (e.g.: Req.get!) as well as steps (e.g. Req.decode/2) but not the low-level API so I think we'd keep them more focused on what they want to do at the moment. Not sure what to do about prepend_default_steps which is somewhat a bridge between high-level and low-level API.

Add run_plugins step

This is another stab at #35. Instead of making plugins special, we'd just have a run_plugins(request, plugins) function which is a request step. It will be one of the default steps that is going to be added if :plugins option is set.

`auth`: Add `:netrc | {:netrc, path}` as possible values

Instead of having a distinct load_netrc step, make it an option to auth, e.g.: auth: :netrc which also would be the default?

`load_netrc`: Support NETRC environment variable

https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html

Change meaning of step `{mod, args}`

Currently {mod, args} is the same as {mod, :run, args}. However, given it's a contract, we expect the run function, let's change it to mean {mod, :run, [opts]}, that is run has arity of 2.

normalize_headers: Rename to encode_headers

This way if we ever add the corresponding one for response headers, it's going to be called decode_headers.

`follow_redirects`: Add `:max_redirects` option

`auth`: Support digest auth

curl example:

$ curl --fail --user foo:bar --digest https://httpbin.org/digest-auth/auth/foo/bar
{
  "authenticated": true,
  "user": "foo"
}

We should support:

Req.get!("https://httpbin.org/digest-auth/auth/foo/bar", auth: {:digest, "foo", "bar"})

New step: `output`

% curl -o foo.html https://elixir-lang.org/index.html
% curl --remote-name https://elixir-lang.org/index.html

Let's add:

iex> Req.get!(url, output: path)
iex> Req.get!(url, output: :remote_name)

`retry`: Handle 429 Retry-After header

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

New step: `parse_url`

In curl, these three invocations are equivalent:

$ curl http://localhost:4000/api
$ curl localhost:4000/api
$ curl :4000/api

let's add a parse_url step that will fill in the missing scheme and host.

Release 1.0.2

Hi @wojtekmach ,
I am having trouble using Req in an umbrella Phoenix app, due the mime dependency for the mime package is v2.0.
If you could make a release with at least this update which is already in mix.exs

Thank you.

Add put_base_url step

Requires #44

Example:

req = Req.new(base_url: "https://api.github.com")
Req.get!(req, "/repos/wojtekmach/req")

`follow_redirects`: Send a GET request after a 301..303

Currently we use the method of the original request. See curl --location. We should also support all the other 3xx codes provided they have the location header.

`decompress_body`: Support transfer-encoding header

retry: Rename max_attempts to max_retries

configurable default steps

something like default_options that permanently modifies what "default steps" are.

Maybe? Not sure if it's a good idea.

`default_headers`: Break up into `put_user_agent` and `compressed` steps

Note, compressed is different than compress step (#25). Naming follows curl --compressed. It just sets the accept-encoding header. It can also be disabled by passing compressed: false. Being able to disable is especially useful because old apache versions incorrectly handle accept-encoding and if-modified-since, they don't return the 304 on cache hit.