mamantoha/crest

HTTP and REST client for Crystal

crest's Issues

Possibility to "download by streaming"

Hi, I looked around inside the source but I didn't find an obvious way to achieve the same result of this code:

require "http/client"

HTTP::Client.get("http://example.org") do |response|
  File.write("example.com.html", response.body_io)
end

I think that It may be an interesting feature to develop and one little feature that could make Crest even more awesome.

For any kind of info about the context in which I would personally love to have this feature don't hesitate to ask me, I'll be more than happy to talk about it!

Cheers! 🍻

Streaming responses

Something like https://github.com/rest-client/rest-client#streaming-responses

Digest Access Authentication

JSON params not accepting Int64 values

 response = Crest.post(url, {
   "name" => "hello",
   "time" => Time.utc.to_unix
 }, json: true)

The code above will not compile with the following message

Showing last frame. Use --error-trace for full trace.

In lib/crest/src/crest/params_encoder.cr:105:16

 105 | memo << {processed_key, v}
            ^-
Error: no overload matches 'Array(Tuple(String, Bool | File | Int32 | String | Symbol | Nil))#<<' with type Tuple(String, Int32 | Int64 | String)

Overloads are:
 - Array(T)#<<(value : T)
 

Integer values in Crystal JSON are Int64. Currently, I worked around this issue by editing Crest's source to include Int64 in the ParamsValue type.

alias ParamsValue = Bool | Int32 | String | Symbol | Nil | File? | Int64 <--- this one

Error: can't find file 'log'

Hello there here is the following issue I get when trying to run my specs:

In lib/crest/src/crest/logger.cr:1:1

 1 | require "log"
     ^
Error: can't find file 'log'

Return expression from response block

Feature request: When doing something like this:

Crest.get("http://example.com") do |resp|
    resp.body_io
end

Would it be possible to have the method return the expression from the block? break/next don't seem to be respected here, and after looking at the source I think it's because this version of the method always just returns nil?

Or am I missing something?

Bug: Arrays in JSON not working

Hello,
I have a request I am trying to send with the library. The request takes as a body a JSON that can have an array object as follows:

{:name => name, :options => options, :users => [{:u => 1}]

The library keeps throwing the following error:

no overload matches 'Array(Tuple(String, Bool | Float32 | Float64 | IO | Int32 | Int64 | String | Symbol | Nil))#<<' with type Tuple(String, Array(Tuple(String, Int32)))

Technical information:

• Crystal 1.5.1
• Crest 1.3.1
• Archlinux

Thanks in advance

Error handling and exceptions

URL params

Extract the query parameters and append them to the url

Add built in connection pooling

It would be nice, for my use case at least, if crest had the ability to maintain a pool of internal clients, and gave the user the ability to decide how large that pool was. Something like ysbaddaden/pool could be used underneath.

Crest v0.10.0 is buggy (?)

shards-info with crest v0.10.1 randomly crash on Heroku about 1-2 times per hour 💩

Proxy support

Can't connect to an HTTPS url that requires an SSL certificate

trying to connect to an https url with:

Crest.get("https://url", read_timeout: 5.second)

but getting this error:

Unhandled exception: SSL_connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (OpenSSL::SSL::Error)

Is there a way to ignore the cert like in Curl with -k or something?

Support passing IO to request

Currently, below code raises exception:

Crest.post(
  "http://httpbin.org/post",
  File.open("avatar.png"),
  headers: {"Content-Type" => "image/png"}
)

Exception:

In src/crest/request.cr:125:7

 125 | generate_form_data!(form) if form
       ^------------------
Error: no overload matches 'Crest::Request#generate_form_data!' with type File

Overloads are:
 - Crest::Request#generate_form_data!(form : Hash)
 - Crest::Request#generate_form_data!(form : String)

It seems to be trivial to support it, see code in https://github.com/cyangle/crest/tree/support_io_as_form_data

@mamantoha

why use my code for logger?

i checked logger.cr and common_logger.cr both are 100% copy my halite code, why do that?

Allow to filter logs by pattern.

Something like

logger.filter(/(api_key=)(\w+)/, "\1[SECRET]")

handle_errors is ignored for redirect errors

Crest.get(url, max_redirects: 0, handle_errors: false) do |response|
end

against a response of 302 will always raise a "Found" exception. The usecase is to have a streaming response, but with an ability to detect it in case the site returns a 302 with an invalid response.

JSON request

Allow to don't raise exceptions but return the response

Upload file without extension results in exception

Got below error when try to upload a file without extension.

Unhandled exception: Missing MIME type for extension "" (KeyError)
  from /usr/share/crystal/src/mime.cr:158:33 in 'from_extension'
  from /usr/share/crystal/src/mime.cr:192:5 in 'from_filename'
  from lib/crest/src/crest/forms/data_form.cr:39:7 in 'add_field'
  from lib/crest/src/crest/forms/data_form.cr:20:11 in 'generate'
  from lib/crest/src/crest/form.cr:9:7 in 'generate'
  from lib/crest/src/crest/request.cr:303:14 in 'generate_form_data!'
  from lib/crest/src/crest/request.cr:125:7 in 'initialize:params:headers:form:logging:handle_errors:params_encoder'
  from lib/crest/src/crest/request.cr:159:5 in 'new:params:headers:form:logging:handle_errors:params_encoder'

I think it's reasonable to set a default mime type application/octet-stream when you can not get it from file extension.

Also, it would be nice to be able to upload IO::Memory instead of only File which is possible because HTTP::FormData::Builder#file accepts any instance of IO.

@mamantoha

Repeating request on failure

The application that I'm working on submits an authentication request and receives a token and then uses that token to submit a request for data. Sometimes, the auth succeeds but the main request times out. I want to trigger a repeat of the main request at that point.

I can catch the error and repeat the request, but that seems to cause an issue with the subsequent processes. Is there a recommended way to repeat on failure?

Thanks!
Marc

Multipart in Resource

Logging

Add custom exceptions for each status code

Response with 404 code should raise Crest::NotFound etc

Keepalive support?

If it has, how do we use it?

How to post more complex JSON data

Hey! 👋
Thanks for this great shard! I'm currently using it to consume an API that requires a JSON schema, that contains arrays and numbers. If I understand correctly one can use params that are converted to JSON automatically. However params have the type Hash(String, String) so we cannot use different types for values.

According to the documentation there is the possibility to pass a raw string as a second argument to Crest.post. However, this throws a compiler error for me:

Error: no overload matches 'Crest.post' with types String, String

Was this behavior changed recently and is there an alternative to use?

Implement real example with cookies, redirect, etc

Add before_request callback

Cookies

Basic Auth on redirects

Resource Nesting

site = Crest::Resource.new('http://example.com')
site['posts/1/comments'].post 'Good article.', :content_type => 'text/plain'

Allow to pass a block to the Crest method, this block will then be called with the Response/Resource

Issue with Crest::Utils.flatten_params

The documentation says that this code should be correct:

Crest::Utils.flatten_params({:key1 => {:arr => ["1", "2", "3"]}, :key2 => "123"})
# => {"key1[arr][]" => "1", "key1[arr][]" => "2", "key1[arr][]" => "3", "key2" => "123"}

Actual result is {"key1[arr][]" => "3", "key2" => "123"}

Unable to connect to host due to OpenSSL Issue

When attempting to contact a JSON API over HTTPS I get an OpenSSL Error. I'm able to access the website in Safari without an issue so I know it's not a server issue. Could the developers please advise on this?

System information:
macOS 10.12.6 (16G1918)
ASDF Version Manager
Crystal 0.27.2
Crest installs as dependency from github: mamantoha/crest

Example code:

require "crest"
require "json"

# TODO: Write documentation for `CrystalProxmox`
module CrystalProxmox
  VERSION = "0.1.0"

  class Proxmox
    @auth_params : Hash(String, String)

    def initialize(pve_cluster : String, node : String, username : String, password : String, realm : String)
      @pve_cluster = pve_cluster # https://your-proxmox-server.local:8006/api2/json/
      @node = node               # node
      @username = username       # root
      @password = password       # password
      @realm = realm             # pve
      @connection_status = "error"
      @site = Crest::Resource.new(@pve_cluster)
      @auth_params = create_ticket()
    end

    def get(path, args = Hash.new)
      @site.get(path, args)
    end

    def post(path, args = Hash.new)
      @site.post(path, args)
    end

    def put(path, args = Hash.new)
      @site.put(path, args)
    end

    def delete(path, args = Hash.new)
      @site.delete(path, args)
    end

    def show_ticket
      @auth_params["cookie"]
    end

    def create_ticket : Hash(String, String)
      response = @site["access/ticket"].post(
        headers: {
          "Content-Type" => "application/json",
        },
        params: {
          "username" => @username,
          "password" => @password,
          "realm"    => @realm,
        }
      )
      extract_ticket(response)
    end

    def extract_ticket(response : Object) : Hash(String, String)
      data = JSON.parse(response.body)
      ticket = data["data"]["ticket"]
      csrf_prevention_token = data["data"]["CSRFPreventionToken"]
      token = "PVEAuthCookie=" + ticket.as_s.gsub(/:/, "%3A").gsub(/=/, "%3D")
      @connection_status = "connected"
      return {
        "CSRFPreventionToken" => csrf_prevention_token.as_s,
        "cookie"              => token,
      }
    end
  end
end

my_proxmox_host = CrystalProxmox::Proxmox.new("https://hypervisor-cust-01.bhs-ca.ulayer.net:8006/api2/json/", "hypervisor-cust-01", "username", "password", "pve")
puts my_proxmox_host.show_ticket

Log output:

Nathaniels-MBP% crystal run src/CrystalProxmox.cr
Unhandled exception: SSL_connect: Unexpected EOF (OpenSSL::SSL::Error)
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/openssl/ssl/socket.cr:34:9 in 'initialize'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/openssl/ssl/socket.cr:3:5 in 'new:context:sync_close:hostname'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/http/client.cr:680:5 in 'socket'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/http/client.cr:572:5 in 'send_request'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/http/client.cr:511:5 in 'exec_internal_single'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/http/client.cr:498:5 in 'exec_internal'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/http/client.cr:494:5 in 'exec'
  from lib/crest/src/crest/request.cr:177:7 in 'execute'
  from lib/crest/src/crest/request.cr:79:7 in 'execute:method:form:url:params:headers:tls:user:password:p_addr:p_port:p_user:p_pass:logging:logger:handle_errors:http_client'
  from lib/crest/src/crest/resource.cr:137:7 in 'execute_request'
  from lib/crest/src/crest/resource.cr:92:5 in 'post:headers:params'
  from src/CrystalProxmox.cr:43:7 in 'create_ticket'
  from src/CrystalProxmox.cr:19:7 in 'initialize'
  from src/CrystalProxmox.cr:11:5 in 'new'
  from src/CrystalProxmox.cr:70:1 in '__crystal_main'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/crystal/main.cr:97:5 in 'main_user_code'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/crystal/main.cr:86:7 in 'main'
  from /Users/nathanielsuchy/.asdf/installs/crystal/0.27.2/src/crystal/main.cr:106:3 in 'main'

Error in Crest.method_name with different payload

This code cause an error:

Crest.post("http://localhost/add-files", payload: {"files[file.txt]" => file, "type" => "txt"})
Crest.post("http://localhost/delete-file", payload: {"file" => "file.txt"})

in /usr/lib/crystal/primitives.cr:175: type must be Array(File | String), not (Array(File | String) | Array(String))

Follow redirects

Publish a new release?

@mamantoha Could you publish a new release with the latest changes? Thanks.

Basic Auth

Bug: Request headers get merged into Response

On Crystal 1.7.0, crest 1.3.7:

require "crest"

r = Crest::Resource.new(
  "http://httpbin.org",
  headers: { "Foo" => "bar" },
)

response = r.get("/get")
p response.headers

Output:

{"Foo" => "bar", "Date" => "Fri, 13 Jan 2023 23:37:49 GMT", "Content-Type" => "application/json", "Content-Length" => "330", "Connection" => "keep-alive", "Server" => "gunicorn/19.9.0", "Access-Control-Allow-Origin" => "*", "Access-Control-Allow-Credentials" => "true"}

logo proposal

Hi @mamantoha I want to design a logo for crest. If there is your an idea, please tell me.

Add method to convert Request object to curl command