appcues/mojito

An easy-to-use Elixir HTTP client, built on the low-level Mint library.

mojito's Issues

Allow configuring a default CA certificate file

Instead of using the CAStore package, I would like to be able to specify the location of the certificate file that is managed by my OS. It would be great if I could configure Mojito to use that by default instead and make the CAStore package an optional dependency.

I would like to be able to have the following code in my config/config.exs file:

config :mojito, :cacertfile, "/etc/ssl/certs/ca-certificates.crt"

And then I would like all SSL connections to use that file unless the function call specifies something different in the transport_opts option.

I know this would be a breaking change that would require people currently using the package to either set the configuration or include CAStore as a dependency, so I thought I'd suggest it here before attempting to work on this myself.

no function clause matching in Mojito.ConnServer.apply_resp/2

I'm receiving this error intermittently, where the first arg is the ConnServer state and the second arg is

{:error, #Reference<0.1440070861.3270508547.43309>, %Mint.HTTPError{module: Mint.HTTP2, reason: :unprocessed}}

Feature request: Streaming request body?

I have been dealing with weirdly large HTTP1 multipart requests in one of the projects I am working on, and this lead to hacking on Finch lib , where I have open an unmerged as of now PR https://github.com/keathley/finch/pull/107/files#diff-48431cc1d91063480b5006d7585c96ea39433e319aca2b5e3a6c597fdbd7e10fR145

I actually use that code in limited capacity so far on production and it seems to work well so far.

The objective is to be able to send large, multipart requests with files to an API that we use, which expects them, without loading whole files into BEAM memory. Think about these files being many megabytes in size, and multiple users uploading files at the same time. The server would, and did, use quite a bit of memory if it has to load them up to binary to stream first.

I had a look at Mojito and while it seems to be sending body in chunks, even on HTTP1 protocol (that I don't think needs to do, by the way - but maybe I'm wrong), it doesn't appear to work in case where body is a Stream itself.

My use case may be pretty specific, but I implemented the API where I can pass either String as a body parameter or a tuple of {:stream, stream}, not sure if the same API would be desired here.

While my mind is fresh and on the subject, I was wondering if you would like to have this also implemented in Mint? I could possibly work on a PR based on the Finch code I wrote earlier.

Telemetry 1.0 compatibility

Telemetry 1.0 has no changes and purely marks the stability of the API:

https://github.com/beam-telemetry/telemetry/blob/main/CHANGELOG.md

Need Mojito to work with Telemetry 1.0. Thx.

Individual requests shouldn't spawn a process

Currently, we launch a Mojito.ConnServer to make each individual request. This makes handling the request chunking slightly easier, but we could do it instead in a recursive function and avoid the penalty of spawning.

Favourite testing strategy?

I'm starting to use Mojito in a new data integration, and wondered: what people using Mojito currently prefer, when writing ExUnit test which use Mojito but should work offline?

A first strategy is to run a test mock server in the Elixir process (like Mojito does in its own tests, or like fake_server). The advantage of this is that it is closer from reality (but a bit more work). (also described in this article).

A second strategy is to use Mox, but this requires to wrap code into a module, use behaviour etc.

A third strategy (usually not popular in the Elixir circles) is to use Mock, to mock specific method calls.

Are there other techniques?

Typically with Mojito, is there something that works better for some reason (maybe because it relies on Mint etc?).

Your input is most welcome!

Timeout issues, where there should be no timeout

I think I found a bug, and since I am using Mojito directly, I am filing it here. It might be upstream, but lets start a deeper investigation here.

On certain requests Mojito times out after receiving a response.
That really weirds me out, because those servers respond just fine. And with a different client, for example curl, I receive a response, no problem.

So there is certainly something wrong with the HTTP-client on the Elixir side.

Disclaimer: I have no control over the server, I just want to use its response, and found it to be not working in our application.

How to reproduce

Mojito Version: 0.6.3

$ elixir --version
Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.10.2 (compiled with Erlang/OTP 22)

Make a GET-request:

Mojito.get("https://www.monheim.de/typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml")
{:error, %Mojito.Error{message: nil, reason: :timeout}}

Compare that to a simple curl:

$ curl -vs 'https://www.monheim.de/typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml'
> GET /typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml HTTP/1.1
> Host: www.monheim.de
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 05 May 2020 18:18:27 GMT
< Content-Type: text/xml
< Content-Length: 6747
< Last-Modified: Tue, 05 May 2020 18:00:14 GMT
< Connection: keep-alive
< ETag: "5eb1a9ae-1a5b"
< Strict-Transport-Security: max-age=31536000;
< Public-Key-Pins: pin-sha256="e7U2Z4tqcX/DCqXsu+FJCNUgazB9kF9lnOrLJ58Oalk="; pin-sha256="Z3VcImUmJVdCSrl04NcQ26JZs+VGNjRLC698dPgQg8Y="; pin-sha256="0cz7JUPqWBD3fHuZh+ha+cf4dGiKrbBv0whzpmiyM7w="; pin-sha256="i0JwsOk4wxTovRpMFfOtAO8JJyL0xayvtJoZFYCr5oo="; report-uri="https://meissnerit.report-uri.io/r/default/hpkp/reportOnly"; max-age=3600;
< Accept-Ranges: bytes
<

I also checked that the server responds with the correct response size as stated in Content-Length.

$ curl -s 'https://www.monheim.de/typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml' | wc -c
  6747

Back to Elixir/Mojito, even when I increase the Mojito timeout, I always times out.

Same buggy-result with 30s timeout:

Mojito.get("https://www.monheim.de/typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml", [], timeout: 30_000)
{:error, %Mojito.Error{message: nil, reason: :timeout}}

Same buggy-result with a 60s timeout:

Mojito.get("https://www.monheim.de/typo3conf/ext/ews_podcast/Resources/Public/rss/podcasts.xml", [], timeout: 60_000)
{:error, %Mojito.Error{message: nil, reason: :timeout}}

Getting a lot of %Mint.HTTPError{module: Mint.HTTP2, reason: :closed_for_writing}

It looks as if either half-closed connections are being returned to the pool when they shouldn't, or :closed_for_writing connections should use the same retry logic that just :closed connections get. I'm happy to take a stab at a PR for this, but could use a little guidance.

Request timeout broken for Mojito.Request

The request timeout works in Mojito.Pool because the whole request process is waiting for a response in a single receive with an after based on the timeout, but I believe because Mojito.Request.receive_response is recursive-ish (it calls handle_msg which can call receive_response again) the after in each receive is for each message received rather than the request as a whole.

Timing data in Mojito

Playing around with an HTTP forward-proxy powered by Mojito. One question I keep asking myself: how much overhead am I adding?

• Round trip time

  The consumers of my service, of course, can sample their language equivalent of System.monotonic_time() immediately before and after the request. I would like to provide them with a number to subtract from the result to understand the overhead.

• Proxy spent time

  My proxy could sample the same, right after receiving and right before returning the response for, the proxied request.

• Proxy waiting time

  It could also figure out how long requests took outside of its own logic via the same mechanism, by tightly wrapping Mojito calls in timing measurements.

• External waiting time

  The main metric I am interested in that I cannot obtain is how long Mojito itself spent waiting for the upstream to respond.

I would like to return the time spent outside the proxy to the caller, but I can't do this without knowing how long Mojito itself was blocked, since I want to incorporate time spent in Mojito into the time spent in the proxy, and separate it (as much as possible) from the external time spent waiting for the request to be processed.

The only way I can see this working is having Mojito aggregate the time spent waiting for connection responses from Mint, and return the sum once the request is completed to the caller of Mojito.request.

Any thoughts about how this could work to guide a PR?

Disabling transfer-encoding: chunked

Hello Team,

I've noticed an issue when using mojito to send post requests to a third party service: https://sheet.best/

When making a POST request with Mojito the service doesn't work, whereas the same post request sent from curl or httppoison does work (so it is not an issue with the third party service).

The reason the service doesn't work when requested with Mojito is due to Mojito forcing the transfer-encoding: chunked format on the request. I've written a quick gist comparing outputs between curl / httppoison / mojito: https://gist.github.com/avaitla/0d6fc2e1ee7a01325cda8e0e5dd28bae

Note that Mojito does NOT return the right results, since sheet.best cannot understand chunked transfer encoding.

Is there anyway that users of this library can disable this feature as not all external services (which we cannot control may not support this encoding).

Thank you,

• Anil

The receive loop is too permissive

Mojito reuses whatever process it was called on, and therefore is not in control of what messages arrive in its mailbox.

However, when waiting for a response from the ConnServer, Mojito will take the first message that it finds:

 receive do
        reply ->
          GenServer.stop(pid)
reply

This is too permissive and causes erroneous :error messages to be returned as a Mojito request, as well as eating a message that was not destined for mojito

Disable Mojito Pool

I don't plan using the pool provided out of the box for mojito, is there a way that I can disable the pool. Thanks and kudos for the awesome work.

Mojito - inconsistencies between pool: true/false in Error response

We had to switch to pool: false because of #57 but then we ran into another problem.. Our handle_response helpers have to be changed to different head, because "same" request (but with pool: false) returns different response:

In first scenario (with pool: false) we receive:

{:error, %Mint.TransportError{reason: :timeout}}

and in second scenario (with pool: true) we receive:

{:error,
 %Mojito.Error{
   message: nil,
   reason: %Mint.TransportError{reason: :econnrefused}
 }}

1. The first thing is difference between Error types which you receive as Result error: %Mojito.Error{} vs %Mint.TransportError{}

2. Second thing is level of immersion, where with (pool: true) you receive Mint.TransportError within Mojito.Error within Error result which is little confusing.

3. Last thing is documentation which describing the Error type (https://hexdocs.pm/mojito/Mojito.Base.html#t:error/0)
   So Error should be
   error() :: %Mojito.Error{message: String.t() | nil, reason: any()}
   not
   {:error, %Mint.TransportError{reason: :timeout}}

Could someone please explain why this is the case or this is a bug?

Thank you

---

More details about the requests

Pool: false

iex(3)> "1234" |> Service.get() |> Client.query("https://192.168.99.103:2376", cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem", certfile: "/usr/src/app/certs/local-vm-1/cert.pem", keyfile: "/usr/src/app/certs/local-vm-1/key.pem")
%Mojito.Request{
  body: "",
  headers: [],
  method: :get,
  opts: [
    pool: false,
    transport_opts: [
      server_name_indication: :disable,
      verify: :verify_peer,
      cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem",
      certfile: "/usr/src/app/certs/local-vm-1/cert.pem",
      keyfile: "/usr/src/app/certs/local-vm-1/key.pem"
    ]
  ],
  url: "https://192.168.99.103:2376/services/1234"
}
{:ok,
 %Mojito.Response{
   body: "{\"message\":\"service 1234 not found\"}\n",
   complete: true,
   headers: [
     {"api-version", "1.40"},
     {"content-type", "application/json"},
     {"docker-experimental", "false"},
     {"ostype", "linux"},
     {"server", "Docker/19.03.12 (linux)"},
     {"date", "Tue, 12 Oct 2021 08:51:07 GMT"},
     {"content-length", "37"}
   ],
   size: 37,
   status_code: 404
 }}
iex(4)> "1234" |> Service.get() |> Client.query("https://192.168.99.102:2376", cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem", certfile: "/usr/src/app/certs/local-vm-1/cert.pem", keyfile: "/usr/src/app/certs/local-vm-1/key.pem")
%Mojito.Request{
  body: "",
  headers: [],
  method: :get,
  opts: [
    pool: false,
    transport_opts: [
      server_name_indication: :disable,
      verify: :verify_peer,
      cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem",
      certfile: "/usr/src/app/certs/local-vm-1/cert.pem",
      keyfile: "/usr/src/app/certs/local-vm-1/key.pem"
    ]
  ],
  url: "https://192.168.99.102:2376/services/1234"
}
{:error, %Mint.TransportError{reason: :timeout}}

Pool: true

iex(4)> "1234" |> Service.get() |> Client.query("https://192.168.99.103:2376", cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem", certfile: "/usr/src/app/certs/local-vm-1/cert.pem", keyfile: "/usr/src/app/certs/local-vm-1/key.pem")
%Mojito.Request{
  body: "",
  headers: [],
  method: :get,
  opts: [
    transport_opts: [
      server_name_indication: :disable,
      verify: :verify_peer,
      cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem",
      certfile: "/usr/src/app/certs/local-vm-1/cert.pem",
      keyfile: "/usr/src/app/certs/local-vm-1/key.pem"
    ]
  ],
  url: "https://192.168.99.103:2376/services/1234"
}
{:ok,
 %Mojito.Response{
   body: "{\"message\":\"service 1234 not found\"}\n",
   complete: true,
   headers: [
     {"api-version", "1.40"},
     {"content-type", "application/json"},
     {"docker-experimental", "false"},
     {"ostype", "linux"},
     {"server", "Docker/19.03.12 (linux)"},
     {"date", "Tue, 12 Oct 2021 08:47:16 GMT"},
     {"content-length", "37"}
   ],
   size: 37,
   status_code: 404
 }}
iex(5)> "1234" |> Service.get() |> Client.query("https://192.168.99.102:2376", cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem", certfile: "/usr/src/app/certs/local-vm-1/cert.pem", keyfile: "/usr/src/app/certs/local-vm-1/key.pem")
%Mojito.Request{
  body: "",
  headers: [],
  method: :get,
  opts: [
    transport_opts: [
      server_name_indication: :disable,
      verify: :verify_peer,
      cacertfile: "/usr/src/app/certs/local-vm-1/ca.pem",
      certfile: "/usr/src/app/certs/local-vm-1/cert.pem",
      keyfile: "/usr/src/app/certs/local-vm-1/key.pem"
    ]
  ],
  url: "https://192.168.99.102:2376/services/1234"
}
{:error,
 %Mojito.Error{
   message: nil,
   reason: %Mint.TransportError{reason: :econnrefused}
 }}

Mojito leaking messages to caller

We have a GenServer that is polling a webservice with Mojito. Constantly the genserver process is receiving messages like:

{:mojito_response, ref, payload}

We ignore them like so:

def handle_info({:mojito_response, _ref, message}, state) do
  # Not sure why is Mojito sending it here.
  {:noreply, state}
end

But I think this is a bug. Why is the caller process receiving these messages in the first place? I think these messages come back after Mojito already returned a timeout error, but not sure.

Connect timeout vs Timeout

I'm having a hard time wrapping my head around the timeout config setting; does the timeout setting cater for connect timeout as well as timeout(waiting to receive data after connecting to the server). I've seen error in production relating to timeout, the upstream node claims the request was not received.

If the connect timeout is different from the timeout, how does the connect timeout affects the overall timeout of the request, I assume connection timeout + timeout will be the total time for a particular request. Is there a default connect timeout setting in mojito?

transport_opts: [timeout: connect_timeout] - will it cater for connect timeout.

Also note that I took a tcpdump and notice that the upstream server is actually sending the response and the response time shouldn't generate a timeout by mojito. I've disable pooling, my timeout in config is 5000 for timeout and 3000 for connect_timeout

timeout =  String.to_integer( Application.get_env(:bonus, :hss_timeout))
connect_timeout = String.to_integer(Application.get_env(:bonus, :connect_timeout))

{_ignore, response} = 
      Application.get_env(:bonus, :sim_check_hss)
      |> Mojito.post([{"Content-Type", "text/xml; charset=UTF-8"}, {"Content-Length", "#{String.length(request)}"}], request, timeout: timeout, pool: false, transport_opts: [timeout: connect_timeout])
      Logger.info("HSS response [#{msisdn}:" <> inspect(Map.get(response, :body) || Map.get(response, :reason)) <> "]")
  
response

Thank you.

Add lifecycle hooks

Add pre- and post-request hooks that can be used to implement e.g., request metrics.

Header Capitalization

I ran into an issue today where the server I'm connecting to was returning an error indicating that a mandatory custom header parameter was missing. After hours of troubleshooting, we discovered that the server couldn't read the request header mainly due to it casing. Is there a way to prevent to force Mojito not to downcase your request headers. It been an awesome library and will be hard for me to turn to a different library.
Kindly assist pls

Feature: fail fast

Hi gamache, very nice project, I am looking for something almost exactly like this, mint + poolboy + simplicity, though I will also need a fail fast mechanism preferably a (sophisticated configurable) circuit breaker.
Do you have any thoughts on that or are you maybe even planning a feature like that?
I could put Mojito behind a circuit breaker of course, that may be even better, just interested in your thoughts.
For me it would be a killer feature.

Memory leak when downloading files

Simply downloading some large files (tested with a 17MB file) is enough to cause the Beam's memory to grow and not be reclaimed after a garbage collection:

iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 

This may be caused by how poolboy copies data between processes.

Originally reported on the ElixirForum:
https://elixirforum.com/t/memory-leak-binaries-that-only-recon-bin-leak-1-helps-with/35804

FunctionClauseError with 304 Response

When playing around with etag and modified_since headers I found something weird.

req_headers = [{"If-Modified-Since", "Wed, 22 May 2019 07:43:40 GMT"}]

Mojito.get("https://robynthinks.wordpress.com/feed/", req_headers)

results in:

{:error, %Mojito.Error{message: nil, reason: :closed}}

[error] GenServer #PID<0.596.0> terminating
** (FunctionClauseError) no function clause matching in Mojito.ConnServer.apply_resp/2
    (mojito) lib/mojito/conn_server.ex:144: Mojito.ConnServer.apply_resp(%{conn: %Mojito.Conn{conn: %Mint.HTTP2{buffer: "", client_settings: %{enable_push: true, max_concurrent_streams: 100, max_frame_size: 16384}, client_settings_queue: {[], []}, decode_table: %Mint.HTTP2.HPACK.Table{entries: [], length: 0, max_table_size: 4096, size: 0}, encode_table: %Mint.HTTP2.HPACK.Table{entries: [{"If-Modified-Since", "Wed, 22 May 2019 07:43:40 GMT"}], length: 1, max_table_size: 4096, size: 78}, headers_being_processed: nil, hostname: "robynthinks.wordpress.com", next_stream_id: 5, open_client_stream_count: 0, open_server_stream_count: 0, ping_queue: {[], []}, port: 443, private: %{}, ref_to_stream_id: %{}, scheme: "https", server_settings: %{enable_push: true, initial_window_size: 65536, max_concurrent_streams: 128, max_frame_size: 16777215, max_header_list_size: :infinity}, socket: {:sslsocket, {:gen_tcp, #Port<0.95>, :tls_connection, :undefined}, [#PID<0.606.0>, #PID<0.605.0>]}, state: :open, streams: %{}, transport: Mint.Core.Transport.SSL, window_size: 2147483647}, hostname: "robynthinks.wordpress.com", port: 443, protocol: :https}, hostname: "robynthinks.wordpress.com", port: 443, protocol: "https", reply_tos: %{#Reference<0.2892813219.510394372.50541> => #PID<0.460.0>}, responses: %{#Reference<0.2892813219.510394372.50541> => %Mojito.Response{body: [], complete: false, headers: [], status_code: nil}}}, {:error, #Reference<0.2892813219.510394372.50541>, %Mint.HTTPError{module: Mint.HTTP2, reason: {:server_closed_request, :protocol_error}}})
    (mojito) lib/mojito/conn_server.ex:141: Mojito.ConnServer.apply_resps/2
    (mojito) lib/mojito/conn_server.ex:114: Mojito.ConnServer.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:ssl, {:sslsocket, {:gen_tcp, #Port<0.95>, :tls_connection, :undefined}, [#PID<0.606.0>, #PID<0.605.0>]}, <<0, 0, 4, 3, 0, 0, 0, 0, 3, 0, 0, 0, 1>>}
State: %{conn: %Mojito.Conn{conn: %Mint.HTTP2{buffer: "", client_settings: %{enable_push: true, max_concurrent_streams: 100, max_frame_size: 16384}, client_settings_queue: {[], []}, decode_table: %Mint.HTTP2.HPACK.Table{entries: [], length: 0, max_table_size: 4096, size: 0}, encode_table: %Mint.HTTP2.HPACK.Table{entries: [{"If-Modified-Since", "Wed, 22 May 2019 07:43:40 GMT"}], length: 1, max_table_size: 4096, size: 78}, headers_being_processed: nil, hostname: "robynthinks.wordpress.com", next_stream_id: 5, open_client_stream_count: 1, open_server_stream_count: 0, ping_queue: {[], []}, port: 443, private: %{}, ref_to_stream_id: %{#Reference<0.2892813219.510394372.50541> => 3}, scheme: "https", server_settings: %{enable_push: true, initial_window_size: 65536, max_concurrent_streams: 128, max_frame_size: 16777215, max_header_list_size: :infinity}, socket: {:sslsocket, {:gen_tcp, #Port<0.95>, :tls_connection, :undefined}, [#PID<0.606.0>, #PID<0.605.0>]}, state: :open, streams: %{3 => %{id: 3, ref: #Reference<0.2892813219.510394372.50541>, state: :half_closed_local, window_size: 65536}}, transport: Mint.Core.Transport.SSL, window_size: 2147483647}, hostname: "robynthinks.wordpress.com", port: 443, protocol: :https}, hostname: "robynthinks.wordpress.com", port: 443, protocol: "https", reply_tos: %{#Reference<0.2892813219.510394372.50541> => #PID<0.460.0>}, responses: %{#Reference<0.2892813219.510394372.50541> => %Mojito.Response{body: [], complete: false, headers: [], status_code: nil}}}

---

When doing the same using curl

curl -I --http2 https://robynthinks.wordpress.com/feed/ --header "If-Modified-Since: Wed, 22 May 2019 07:43:40 GMT"

I get this response:

HTTP/2 304 
server: nginx
date: Thu, 23 May 2019 19:44:29 GMT
vary: Accept-Encoding
last-modified: Wed, 22 May 2019 07:43:40 GMT
x-nc: HIT dfw 93
x-ac: 5.ams _dfw 
strict-transport-security: max-age=15552000

Mojito requests return the wrong response after a timeout

It seems that if Mojito times out in the client when making a request, but the request eventually succeeds, the next request will return the previous response.

The test case here:

• Makes a request with a timeout of 1ms, knowing that the request will take 10ms
• Waits 100ms for the request to complete behind the scenes
• Makes another request that should succeed

Result: The "Alice" request receives the "wait" response

    it "handles requests after a timeout" do
      assert({:error, %{reason: :timeout}} = get("/wait?d=10", timeout: 1))
      Process.sleep(100)
      assert({:ok, %{body: "Hello Alice!"}} = get("?name=Alice"))
    end

...............................

  1) test local server tests handles requests after a timeout (MojitoTest)
     test/mojito_test.exs:252
     match (=) failed
     code:  assert {:ok, %{body: "Hello Alice!"}} = get("?name=Alice")
     right: {:ok,
             %Mojito.Response{
               body: "ok",
               complete: true,
               headers: [
                 {"server", "Cowboy"},
                 {"date", "Wed, 21 Aug 2019 00:19:44 GMT"},
                 {"content-length", "2"},
                 {"cache-control", "max-age=0, private, must-revalidate"}
               ],
               status_code: 200
             }}
     stacktrace:
       test/mojito_test.exs:256: (test)

............................

Finished in 2.3 seconds
11 doctests, 49 tests, 1 failure

Randomized with seed 149333

Turning off the debug logger

Can the debug logger be turned off? The responses are flooding my console

exceeds_window_size from Mint.HTTP2 when using Mojito.post

{:error, %Mojito.Error{message: nil, reason:
  %Mint.HTTPError{module: Mint.HTTP2,
    reason: {:exceeds_window_size, :request, 65536}}}}

I am seeing this bubble up occasionally from Mojito.post (when POSTing to Wordpress); it looks like Mojito needs to renegotiate the window size after initial setup of the connection, or when this error is returned, and then send chunked data over the stream.

• https://github.com/elixir-mint/mint/blob/v1.0.0/lib/mint/http2.ex#L288-L291
• https://hexdocs.pm/mint/Mint.HTTP2.html#get_window_size/2-http-2-flow-control

Header non-string values

I'm using mojito as a Http client for ExAws.

defmodule MyApp.S3.HttpClient do
  @behaviour ExAws.Request.HttpClient
  def request(method, url, body, headers, http_opts \\ []) do
    Mojito.request(method, url, headers, body, http_opts)
  end
end

When trying to download a file, I'm getting:

error] GenServer #PID<0.637.0> terminating
** (FunctionClauseError) no function clause matching in Mint.HTTP1.Request."-validate_header_value!/2-lc$^0/1-0-"/1

Last message (from #PID<0.478.0>): {:request, %Mojito.Request{body: "", headers: [{"x-amz-date", "20191213T171338Z"}, {"content-length", 0}], method: :head, opts: [], url: "https://s3.eu-central-1.amazonaws.com/bucket/file"}, #PID<0.478.0>, #Reference<0.938540534.2258632705.207122>}
State: %{conn: nil, hostname: nil, port: nil, protocol: nil, reply_tos: %{}, response_refs: %{}, responses: %{}}
Client #PID<0.478.0> is alive

This is happening because of this part {"content-length", 0}, if I preprocess the headers and convert the values to strings, then it works.

Should mojito be more permissive, and auto-convert the values? Will gladly do a PR.

Does Mojito support a way to specify between HTTP/1 or HTTP/2?

I am hitting a bug in my code that looks something like this:

{:error, %Mojito.Error{message: nil, reason: %Mint.HTTPError{module: Mint.HTTP2, reason: {:exceeds_window_size, :connection, 65535}}}}

Upon some inspection, it looks like it's a HTTP/2 specific thing, so I went digging to see if there's a way for me to specify to use HTTP/2 instead. I can't really find anything in the Mojito doc, but I managed to find out that Mint does support it over here, so just curious if Mojito can support that?

Handling empty responses

Hi, I've noticed that Mojito tries to decompress a response even when it's empty, which will result in this error:

** (ErlangError) Erlang error: :data_error
    :zlib.inflateEnd_nif(#Reference<0.3283516080.844496899.227436>)
    :zlib.gunzip/1
    (mojito 0.7.9) lib/mojito.ex:7: Mojito.maybe_decompress/2

When the status code is 200, I think the expected behavior would be returning an empty response.

Feature request: async requests

Perhaps its just not a design goal for Mojito but it would be great if there was an async: <pid> option for the request functions. This way the request doesn't block and a future message to the pid can capture the response.

Convenience functions for GET, POST, etc.

In addition to Mojito.request/5, we should provide convenience functions for head/3, get/3, post/4, put/4, patch/4, delete/3, and options/3. People like those.

Doesn't handle Connection:Close Responses

The single request pool doesn't handle responses with Connection:close.

https://github.com/appcues/mojito/blob/master/lib/mojito/request/single.ex#L39

There is no handling of tcp_closed message so the Mint state machine never receives the close indicator so never knows the response has been fully received. Similar, issue with TLS as well.

task mode: unknown registry: Mojito.Pool.Poolboy.Registry

just simple code
Mojito.request(method: :get, url: "http://127.0.0.1" )
work fine in "mix run".

but when use in Mix.Task , it got error.
Compiling 1 file (.ex)
** (ArgumentError) unknown registry: Mojito.Pool.Poolboy.Registry
(elixir) lib/registry.ex:1243: Registry.key_info!/1
(elixir) lib/registry.ex:568: Registry.lookup/2
lib/mojito/pool/poolboy.ex:65: Mojito.Pool.Poolboy.get_pools/1
lib/mojito/pool/poolboy.ex:49: Mojito.Pool.Poolboy.get_pool/1
lib/mojito/pool/poolboy.ex:28: Mojito.Pool.Poolboy.request/1
lib/httpc.ex:4: Httpc.get/1
lib/task/test.ex:26: Mix.Tasks.Web.geturl/1
(mix) lib/mix/task.ex:331: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
(elixir) lib/code.ex:813: Code.require_file/2

Mojito's problem ? or poolboy's ?

timeout: :infinity fails.

just in case we want to use mojito to download a really big file.

Invalid URL call

Hello,
I was using Mint library for test, and trying to move on here because it seems more simple to use. Thanks for your work.

This is my Mint code:

{:ok, conn} = Mint.HTTP.connect(:http, "localhost", 8080)
    {:ok, conn, request_ref} =
      Mint.HTTP.request(conn, "GET", "/v1/api/user/001", [
        {"content-type", "application/json"}
      ], "")

    # read and parse the response
    receive do
      message ->
        {:ok, conn, responses} = Mint.HTTP.stream(conn, message)

this was working well, but:

api_endpoint = "http://localhost:8080/v1/api/user/001"

case Mojito.request(:get, url: api_endpoint) do
  {:ok, response} -> Logger.info "Okay: #{response}"
  {:error, reason} -> IO.puts(reason)
end

this returns:

** (Protocol.UndefinedError) protocol String.Chars not implemented for %Mojito.Error{message: "invalid URL", reason: %FunctionClauseError{args: nil, arity: 1, clauses: nil, function: :parse, kind: nil, module: URI}} of type Mojito.Error (a struct)
    (elixir) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir) lib/string/chars.ex:22: String.Chars.to_string/1
    (elixir) lib/io.ex:654: IO.puts/2
    (mix) lib/mix/task.ex:331: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2

Could you give me a guide if I'm doing something wrong?

Thanks.

Mojito returns `{:error, %Mojito.Error{message: nil, reason: :timeout}}` while HTTPoison or cURL works

Mojito returns {:error, %Mojito.Error{message: nil, reason: :timeout}} intermittently. It's rare, but it happens in our production system and it will be impossible to keep using Mojito with this problem.

I used this to reproduce. It takes sometime, but eventually you will see it:

  defmodule Bug do
    def reproduce(amount \\ 10, http_client \\ Mojito) do
      Enum.each(1..amount, fn i ->
        IO.write("#{i}/#{amount}\t\r")
        http_client |> request() |> handle_response()
      end)
    end

    defp request(http_client) do
      http_client.get("https://jsonplaceholder.typicode.com/todos/1")
    end

    defp handle_response({:ok, %{status_code: 200}}), do: :ok

    defp handle_response(response) do
      IO.inspect(response, label: :error)
      raise(RuntimeError)
    end
  end

  Bug.reproduce(10000)

It never happens on Bug.reproduce(10000, HTTPoison).
I'm on 0.7.7 and my config is default.

Mint with pool: false leaks HTTP connections like a sieve

Search for close in Single (https://github.com/appcues/mojito/blob/master/lib/mojito/request/single.ex). There are no calls to Mint.HTTP.close. If the socket is not closed it is leaked until the process dies. see: https://hexdocs.pm/mint/Mint.HTTP.html#close/1

Pools should handle connections to multiple hosts

The current implementation of connection pooling in Mojito.Pool is a naive Poolboy setup, with the limitation that the pool should only be used to connect to a single protocol+host+port. Otherwise, the connection checked out of the pool is likely to be a connection to a different server, and we drop the existing connection and open another.

Consider a two-level pooling system, where a pool exists for each protocol+host+port we want to make requests to, and the process registry directs each request to its correct pool.

(Thanks to @ericmj for talking this through on Slack)

Follow redirect

Support option to follow_redirect and return the response at the final location.

Fragment handling in URI

At the moment, an existing fragment of the requested URI is passed down to mint. If I understand it correctly, this should not be the case. I checked with hackney, httpie and other tools and they all omit the fragment.

Clients are not supposed to send URI fragments to servers when they retrieve a document, […]
https://en.wikipedia.org/wiki/Fragment_identifier

Proxy Support

Is there currently a way to use the proxy functionality from Mint with Mojito, or does that still need to be implemented?

I'm not seeing any references to proxy support at the moment.

Document that by default you will receive tcp messages in the process that calls `Mojito.request`

From a slack discussion with @gamache

gamache [9 hours ago]
That's correct, the caller process will receive messages of the form {:tcp, _, _} and {:ssl, _, _} and handle them transparently.

gamache [9 hours ago]
Other messages will be left in the mailbox, which should keep most callers out of trouble, but if other TCP/SSL messages were expected, Mojito will gobble them up. In that case, wrap the one-off request in a Task or use Mojito.Pool to make requests.

I think this should be called out on the docs at https://hexdocs.pm/mojito/0.2.1/Mojito.html#request/1 and https://hexdocs.pm/mojito/0.2.1/Mojito.html#request/5

And the module doc could probably be a little more explicit as well.

More friendly API to deal with errors

I tested your library and notice that I can´t parse error response easily, because the library doesn't handle/parse it to a common contract.

Example of errors doing requests:

%Mojito.Error{message: "invalid URL"}

%Mojito.Error{reason: %Mint.TransportError{reason: :nxdomain}}}

A common contract could be like this:

%Mojito.Error{reason: :invalid_url, message: "invalid URL"}

%Mojito.Error{reason: :nx_domain, details: %Mint.TransportError{reason: :nxdomain}}

%Mojito.Error{reason: :some_other_error}

Explicitly test both HTTP1 and HTTP2

We should test both protocols separately rather than letting Cowboy decide.

Discussion: Mojito should not use the application environment

Hi @gamache!

Just a quick topic for discussion. From the README, it seems Mojito is using the application environment / config files to configure its pools. The problem with this design is that it is not composable.

For example, imagine I want to write a library called "foobar" that needs to use Mojito. Now if someone wants to use my "foobar" library in their application, they need to explicitly configure Mojito inside their app, because it is not possible (by design) for one library to configure the environment of another library in behalf of an application.

You could argue that the application may most likely to configure the pool anyway, which is a good point, but that may not always be the case and the current design always forces me to. Similarly, as the author of "foobar", I may not want to leak to my users that I am actually using Mojito, as it is an implementation detail, and currently it seems I don't have the option.

Possible solution: how to configure pools

One option to configure pools is to have users explicitly start pools in their app:

[
  {Mojito.Pool, name: MyApp.Pool, ...}
}

And now when doing the request you pass which pool to use: Mojito.get(MyApp.Pool, ...). This gives users complete control when the pools are started and configured.

You may still want to provide a built-in, no-host pool, when Mojito is started, which is configured via the application environment, but it should be absolutely clear that the config of this pool will be shared across all libraries and apps.

PS: Note I am writing this report based on the README, so if any of it is inaccurate, apologies and close this issue. :)

Mojito possibly leaking {:tcp_closed, port}

I ran into a strange error today whiles working with Mojito. I've a GenServer that invokes an http endpoint via Mojito. Initially it was crashing with undefined function clause for handle_info, after adding a general function clause, {:tcp_closed, port} was being sent to the GenServer. It's quite surprising since I've Mojito in production with no such message. I've hackney in my codebase as well but the GenServer generating this message uses only Mojito. Could it be that Mojito is sending the above message to the GenServer.

on error sends me a message as well as return value

My understanding is that all I need to do to handle errors is look at the result, e.g.

case Mojito.get(url, headers) do
    {:ok, %Mojito.Response{status_code: 200, body: body}} ->
      {:ok, Jason.decode!(body)}

    {:ok, %Mojito.Response{status_code: status_code}} ->
      {:error, "Unexpected response #{status_code}"}

    {:error, error} ->
      {:error, error}
 end

But I find on certain types of errors, I also have to implement the appropriate handle_info call.

e.g. I got the error:

{:error, %Mojito.Error{message: nil, reason: :timeout}}

But my genserver died also, about 0.2 seconds later.

** (FunctionClauseError) no function clause matching in PenguinNodes.Life360.Circles.handle_info/2
    (penguin_nodes 0.1.0) lib/penguin_nodes/life360/circles.ex:79: PenguinNodes.Life360.Circles.handle_info({:mojito_response, #Reference<0.1958781291.3315859457.252057>, {:ok, %Mojito.Response{...}], size: 328, status_code: 200}}}, %PenguinNodes.Nodes.NodeModule.State{...}})

What seems strange is that this looks like a valid response. So maybe the problem is that mojito timed out, and then we got a response and mojito is confused by this response after it already timed out?

Am I expected to receive messages from mojito too (if so this should be documented in README.md)? Or is this message a bug?

Out of the box support for Mox

Thoughts on implementing Mox support similar to what was does in HTTPoison?

https://github.com/edgurgel/httpoison/pull/330/files

I can make the effort on the PR if @gamache is open to this.

Receiving `{:error, %Mint.TransportError{reason: :timeout}}`

Hello,

When we make a call with mojito that results in {:error, %Mojito.Error{message: nil, reason: :checkout_timeout}}, we receive a message like this: {#Reference<0.981418250.3438280705.26871>, {:error, %Mint.TransportError{reason: :timeout}}} after a while.

It is not a big deal since we can handle those message and ignore them. But shouldn't mojito and mint have the same timeout ?

Passing in a nil URL returns a hard-to-debug error

Calling `Mojito.request(:get, nil) will return the following error:

{:error,
 %Mojito.Error{
   message: nil,
   reason: %FunctionClauseError{
     args: nil,
     arity: 2,
     clauses: nil,
     function: :from_string,
     kind: nil,
     module: Fuzzyurl
   }
 }}

I somewhat expected Dialyzer to pick this up, but it didn't at least for my project. It would be better in my opinion to add a guard clause preventing this somewhat common user behavior. Doing so would generate an easy to understand error message. I will open a PR to consider for this issue

mix release breaks with 0.7.4

Hey everyone,

I've noticed that all our mix releases break with the new 0.7.4 version with the following error:

** (Mix) Duplicated modules: 
mint_shims specified in mint and mojito

This happens as mix somehow creates a link to the mint directory:

ubuntu@machine:~/test1234$ ls -la deps/mojito/
total 44
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov  3 11:27 .
drwxrwxr-x 6 ubuntu ubuntu 4096 Nov  3 11:27 ..
-rw-rw-r-- 1 ubuntu ubuntu 4395 Nov  3 11:27 CHANGELOG.md
-rw-rw-r-- 1 ubuntu ubuntu    0 Nov  3 11:27 .fetch
-rw-r--r-- 1 ubuntu ubuntu  128 Nov  3 11:27 .formatter.exs
-rw-rw-r-- 1 ubuntu ubuntu  270 Nov  3 11:27 .hex
-rw-rw-r-- 1 ubuntu ubuntu 1525 Nov  3 11:27 hex_metadata.config
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov  3 11:27 lib
-rw-rw-r-- 1 ubuntu ubuntu 1415 Nov  3 11:27 mix.exs
-rw-rw-r-- 1 ubuntu ubuntu 6924 Nov  3 11:27 README.md
lrwxrwxrwx 1 ubuntu ubuntu   11 Nov  3 11:27 src -> ../mint/src

Compared to 0.7.3:

ubuntu@machine:~/test1234$ ls -la deps/mojito/
total 44
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov  3 11:17 .
drwxrwxr-x 6 ubuntu ubuntu 4096 Nov  3 11:17 ..
-rw-rw-r-- 1 ubuntu ubuntu 4171 Nov  3 11:17 CHANGELOG.md
-rw-rw-r-- 1 ubuntu ubuntu    0 Nov  3 11:17 .fetch
-rw-r--r-- 1 ubuntu ubuntu  128 Nov  3 11:17 .formatter.exs
-rw-rw-r-- 1 ubuntu ubuntu  270 Nov  3 11:17 .hex
-rw-rw-r-- 1 ubuntu ubuntu 1515 Nov  3 11:17 hex_metadata.config
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov  3 11:17 lib
-rw-rw-r-- 1 ubuntu ubuntu 1415 Nov  3 11:17 mix.exs
-rw-rw-r-- 1 ubuntu ubuntu 6924 Nov  3 11:17 README.md

To reproduce:

$ mix new test1234
$ cd test1234
# add {:mojito, "0.7.4"} to mix.exs
$ mix deps.get
$ mix release

Mojito fails to send if the query string has %7B in it

I will dig more into why this happens, but here's my minimal repro:

It doesn't make any outgoing request, but fails with :invalid_request_target

iex(2)> Mojito.request(:get, "http://localhost:4000/?r=%7B")

18:28:55.479 [debug] Mojito.ConnServer #PID<0.361.0>: get http://localhost:4000/?r=%7B

18:28:55.481 [debug] Mojito.ConnServer #PID<0.361.0>: cleaning up
{:error,
%Mojito.Error{
message: {:error,
%Mint.HTTP1{
buffer: "",
host: "localhost",
private: %{},
request: nil,
requests: {[], []},
socket: #Port<0.13>,
state: :open,
transport: Mint.Core.Transport.TCP
}, :invalid_request_target},
reason: :unknown
}}