nahi/httpclient

'httpclient' gives something like the functionality of libwww-perl (LWP) in Ruby.

httpclient - HTTP accessing library.

Copyright (C) 2000-2015 NAKAMURA, Hiroshi [email protected].

'httpclient' gives something like the functionality of libwww-perl (LWP) in Ruby. 'httpclient' formerly known as 'http-access2'.

See HTTPClient for documentation.

Features

• methods like GET/HEAD/POST/* via HTTP/1.1.
• HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
• asynchronous HTTP request, streaming HTTP request.
• debug mode CLI.
• by contrast with net/http in standard distribution;
  • Cookies support
  • MT-safe
  • streaming POST (POST with File/IO)
  • Digest auth
  • Negotiate/NTLM auth for WWW-Authenticate (requires net/ntlm module; rubyntlm gem)
  • NTLM auth for Proxy-Authenticate (requires 'win32/sspi' module; rubysspi gem)
  • extensible with filter interface
  • you don't have to care HTTP/1.1 persistent connection (httpclient cares instead of you)
• Not supported now
  • Cache
  • Rather advanced HTTP/1.1 usage such as Range, deflate, etc. (of course you can set it in header by yourself)

httpclient command

Usage: 1) httpclient get https://www.google.co.jp/?q=ruby
Usage: 2) httpclient

For 1) it issues a GET request to the given URI and shows the wiredump and the parsed result. For 2) it invokes irb shell with the binding that has a HTTPClient as 'self'. You can call HTTPClient instance methods like;

get "https://www.google.co.jp/", :q => :ruby

Author

• Name:: Hiroshi Nakamura
• E-mail:: [email protected]
• Project web site:: http://github.com/nahi/httpclient

License

This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby's license; either the dual license version in 2003, or any later version.

httpclient/session.rb is based on http-access.rb in http-access/0.0.4. Some part of it is copyrighted by Maebashi-san who made and published http-access/0.0.4. http-access/0.0.4 did not include license notice but when I asked Maebashi-san he agreed that I can redistribute it under the same terms of Ruby. Many thanks to Maebashi-san.

Install

You can install httpclient via rubygems: gem install httpclient

Usage

See HTTPClient for documentation. You can also check sample/howto.rb how to use APIs.

Bug report or Feature request

Please file a ticket at the project web site.

1. find a similar ticket from https://github.com/nahi/httpclient/issues
2. create a new ticket by clicking 'Create Issue' button.
3. you can use github features such as pull-request if you like.

Changes

See ChangeLog

httpclient's Issues

Update cacert.p7s

Comparing string to float

http://dev.ctor.org/http-access2/ticket/231

Problem with non-ascii characters

Hi,

It seems like there's a bug in HttpClient for MRI 1.9, but it seems to work fine in 1.8.7. It seems like the encoding isn't set correctly and hence the operations that works on doesn't work properly as it should. It works fine though if the encoding is forced with force_encoding on the content returned.

I guess that the encoding should be pulled from Content-Type header if available.

Examples, code and gems used can be found at http://gist.github.com/649839

Thanks to apeiros @ #ruby-lang@freenode for finding this.

//Victor

set_request_header clobbers 'Host' header setting

If you provide a Host header entry, set_request_header will ignore it and assume that the hostname in the URI is the correct one. This is normally a good assumption, except when you are accessing a virtualhost without going through DNS.

This simple change fixes it:

def set_request_header
  return if @dumped
  @dumped = true
  keep_alive = Message.keep_alive_enabled?(@http_version)
  if !keep_alive and @request_method != 'CONNECT'
    set('Connection', 'close')
  end
  if @chunked
    set('Transfer-Encoding', 'chunked')
  elsif @body_size and (keep_alive or @body_size != 0)
    set('Content-Length', @body_size.to_s)
  end
  if @http_version >= 1.1
    if (h = get('Host')).length > 0
      host = h.flatten.last
    else
      host = @request_uri.host
    end
    if @request_uri.port == @request_uri.default_port
      # GFE/1.3 dislikes default port number (returns 404)
      set('Host', "#{host}")
    else
      set('Host', "#{host}:#{@request_uri.port}")
    end
  end
end

Opened SSL files not being closed properly

http://dev.ctor.org/http-access2/ticket/220

httpclient-2.1.5.2 fails tests

I get several test failures when trying to run the tests that come with httpclient:

1. Error:
   test_find_cookie(TestCookieManager):
   NoMethodError: undefined method use=' for nil:NilClass ./test/test_cookie.rb:239:intest_find_cookie'

2. Failure:
   test_cookies(TestHTTPClient) [./test/test_httpclient.rb:797]:
   <"http://rubyforge.org/account/login.php\tsession_ser\tLjEwMy45Ni40Ni0q%2A-fa0537de8cc31\t2000000000\t.rubyforge.org\t/\t13\nhttp://rubyforge.org/account/login.php\tfoo\tbar\t1924902000\trubyforge.org\t/login.php\t1\n"> expected to be =~
   </http://rubyforge.org/account/login.php foo bar 1924873200 rubyforge.org /login.php 1/>.

3. Failure:
   test_response(TestHTTPClient) [./test/test_httpclient.rb:964]:
   <["",
   "Content-Length: 8",
   "Content-Type: text/plain",
   "Last-Modified: Fri, 31 Dec 1999 15:00:00 GMT",
   "Status: 200 OK",
   "response"]> expected but was
   <["",
   "Content-Length: 8",
   "Content-Type: text/plain",
   "Last-Modified: Fri, 31 Dec 1999 23:00:00 GMT",
   "Status: 200 OK",
   "response"]>.

129 tests, 525 assertions, 2 failures, 1 errors
rake aborted!
Command failed with status (1): [/usr/bin/ruby18 -w -I"lib:lib" "/usr/lib64...]

When retrying a failed persistent connection, httpclient should use a fresh connection

When retrying a failed persistent connection, httpclient should use a fresh connection, not an already-used connection, because it may very well be that the already used-connection will also fail. This may happen for example if httpclient has established multiple connections to the same server and

1. the server has been restarted in the meantime or
2. the traffic volume has become low such that the server's persistent connection timeout limit triggers. Unfortunately, in low traffic sitations, timeouts are triggerd typically not only for one of the persistent connections, but for all of these.

Suggestion to implement transparent gzip decompression.

Hi,

I think it would be great if httpclient support gzip decompression because it is
widely used on many websites and reduces network traffics dramatically.
So I made a small patch to support gzip decompression on httpclient.

https://gist.github.com/702845

This patch adds a "transparent_gzip_decompression" attribute to HTTPClient. It is set to false by default for backward compatibility. With setting true to this attribute:

• "Accept-Encoding: gzip,deflate" will be added to every request headers.
• Decompress the content body automatically if the response header has "Content-Encoding: gzip OR x-gzip OR deflate".

Test cases for this change is also included in the patch.

Regards,

Wrong HTTP version in headers with Qt4 applications

If you create a Qt application before making a call with httpclient, the sent header looks like this:

POST /global/v3/BFGlobalService HTTP/1,1

The version is 1,1 instead of 1.1. It seems httpclient stores the http version like a double, and the Qt library overrides the method to_s from Float depending on the locale configuration of the user:

wotan@rodia:~$ irb

irb(main):001:0> require 'Qt4'

=> true

irb(main):002:0> a = Qt::Application.new(ARGV)

=> #<Qt::Application:0x7fc2800a8b98 objectName="irb">

irb(main):003:0> b = 1.1

=> 1,1

"cacerts failed loading" on 2.1.6 and centos 4

I haven't been able to track down why quite yet but the verification step of the p7s seems to be failing. Unfortunately ruby's openssl setup doesn't provide anything helpful in the error msg.

Also, this seems to work on my mac.

Digest authentication fails when the uri contains a query

Digest authentication fails when the uri contains a query, such as /users/search?email=[email protected] .

We have traced this back to the DigestAuth#calc_cred where only uri.path is used in variable a_2 and uri.query is discarded.

We are not 100% sure that this is not valid behavior, but it fails when authenticating against a Rails app. When the query is included, it works.

post_content: returning full header

This is about the HTTPClient#follow_redirect / HTTPClient#post_content methods.

It seems it is not standard browser behaviour to follow redirects returned for POST requests, and post the same message to the new location. However, for what I use this library for, it is useful. The post_content method reads at revision 280:

def post_content(uri, body = nil, extheader = {}, &block)
  follow_redirect(:post, uri, nil, body, extheader, &block).content
end

The follow_redirect method is marked as private.

Since the post_content only returns the content, the content-type, and other header entries are not available.

An easy solution would be to add a method that does the same as post_content, but returns the return value of follow_redirect ?

def post_with_redirect(uri, body = nil, extheader = {}, &block)
  follow_redirect(:post, uri, nil, body, extheader, &block)
end

URL with /com after the host causes unexpected 404 error

http://dev.ctor.org/http-access2/ticket/234

If you have a url thats formatted as such "http://host.com/com/" it seems to cause an error. If you attempt to manually navigate to the URL everything is fine.

# Sample code, the url is the first one i found while googling .com/com

require 'HTTPClient'
client = HTTPClient.new
url = "http://www.microsoft.com/com/default.mspx"
client.get_content(url)

# error

c:/dev/ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/HTTPClient.rb:843:in `follow_redirect': unexpected response: # (HTTPClient::BadResponseError)
    from c:/dev/ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/HTTPClient.rb:519:in `get_content'
    from test.rb:9
http://www.microsoft.com/com/default.mspx

doesn't support multiline server headers

same as net/http.rb http://redmine.ruby-lang.org/issues/show/1796#change-6587

For WebAgent::Cookie test code.

http://dev.ctor.org/http-access2/ticket/230

cacerts loading failed

Suddenly we have started getting 'cacerts loading failed' on initializing HTTPClient, i.e. HTTPClient.new

I am using Leopard and it is doing with both version 2.1.5.2 and 2.1.6 of httpclient.

However, 2.1.5.2 is fine on the Debian distro.

[PATCH] https-over-proxy fails with IIS (possible RFC violation on both?)

Hi.

I'm having trouble accessing https/IIS server over web proxy, and it seems there's some weak RFC violation on both sides. I'm using version 2.1.5.2 on Debian sid.

The problem is that

• httpclient/ruby sends out absolute URI for https URL in request-line, even when the connection is tunneled over CONNECT via proxy.
• IIS (sometimes?) returns 404 when request is given in absolute URI and the scheme is "https".

I noticed this issue when I moved my https-based automation tool (to automate web-UI on IIS) to off-site, and had to use web proxy. Once it went over proxy, every request returned 404.

IMHO, this interop problem seems to be a corner case of RFC. RFC2616 (HTTP/1.1) states as follows:

   ==== 5.1.2 Request-URI ===
   The absoluteURI form is REQUIRED when the request is being
   made to a proxy.

BUT, above line lacks some consideraion of https/CONNECT case. Going down some lines, there is another statement:

   ==== RFC2616: 5.1.2 Request-URI ===
   The most common form of Request-URI is that used to identify a
   resource on an origin server or gateway. In this case the absolute
   path of the URI MUST be transmitted (see section 3.2.1, abs_path) as
   the Request-URI, and the network location of the URI (authority) MUST
   be transmitted in a Host header field.

And, looking into RFC2817, CONNECT is guranteed to establish tunnel to origin server/gateway.

  === RFC2817: 5.3 Establishing a Tunnel with CONNECT ===
  Any successful (2xx) response to a CONNECT request indicates that the
  proxy has established a connection to the requested host and port,
  and has switched to tunneling the current connection to that server
  connection.
  
  It may be the case that the proxy itself can only reach the requested
  origin server through another proxy.  In this case, the first proxy
  SHOULD make a CONNECT request of that next proxy, requesting a tunnel
  to the authority.  A proxy MUST NOT respond with any 2xx status code
  unless it has either a direct or tunnel connection established to the
  authority.
  
  An origin server which receives a CONNECT request for itself MAY
  respond with a 2xx status code to indicate that a connection is
  established.

This means client should handle CONNECT-ed tunnel as same as a connection to origin server/gateway. So user-agent must generate request-line with absolute-path, not absolute-URI.

Looking at RFC numbers, https/CONNECT (RFC2817) was probably not defined good enough when RFC2616 was standardized, leaving this corner case.

Also, it seems IIS should also be at blame as RFC2616 states

  === RFC2616: 5.1.2 Request-URI ===
  To allow for transition to absoluteURIs in all requests in future
  versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI
  form in requests, even though HTTP/1.1 clients will only generate
  them in requests to proxies.

But since fixing httpclient/ruby is so much easier, I'm submitting this ticket as well as a patch. It's only an one-line fix.

  diff --git a/lib/httpclient/http.rb b/lib/httpclient/http.rb
  index 0b24071..65aa46e 100644
  --- a/lib/httpclient/http.rb
  +++ b/lib/httpclient/http.rb
  @@ -316,7 +316,7 @@ module HTTP

         def request_line
           path = create_query_uri()
  -        if @request_via_proxy
  +        if @request_via_proxy && @request_uri.scheme != "https"
             path = "#{ @request_uri.scheme }://#{ @request_uri.host }:#{ @requ\
est
_uri.port }#{ path }"
           end
           "#{ @request_method } #{ path } HTTP/#{ @http_version }#{ CRLF }"

I'm not sure if whitespace is preserved appropriately, but it should be easy to do the same fix manually.

Hope this gets accepted for next release.

httpclient 2.1.5.2 under JRuby 1.4.0 consumes 100% CPU in idle mode

2 thread dumps show the following:

"Thread-7" daemon prio=10 tid=0x00002aacc8490400 nid=0x3d3c runnable [0x000000004264d000..0x000000004264ea20]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.notify(Native Method)
at org.jruby.libraries.ThreadLibrary$Mutex.unlock(ThreadLibrary.java:156)
- locked <0x00002aaaccbb4ed8> (a org.jruby.libraries.ThreadLibrary$Mutex)
at org.jruby.libraries.ThreadLibrary$Mutex.synchronize(ThreadLibrary.java:169)
at org.jruby.libraries.ThreadLibrary$Mutex$i_method_0_0$RUBYINVOKER$synchronize.call(org/jruby/libraries/ThreadLibrary$Mutex$i_method_0_0$RUBYINVOKER$synchronize.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:116)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:133)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.method__4$RUBY$raise(timeout.rb:40)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutInvokermethod__4$RUBY$raiseFixed1.call(timeout#raise)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutInvokermethod__4$RUBY$raiseFixed1.call(timeout#raise)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.block_4$RUBY$block(timeout.rb:102)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutBlockCallback$block_4$RUBY$block__xx1.call(Unknown Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:105)
at org.jruby.runtime.Block.yield(Block.java:194)
at org.jruby.RubyArray.eachCommon(RubyArray.java:1635)
at org.jruby.RubyArray.each(RubyArray.java:1642)
at org.jruby.RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.call(org/jruby/RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:116)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:133)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.rescue_2$RUBY$__rescue___12(timeout.rb:100)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.block_2$RUBY$__block(timeout.rb:87)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutBlockCallback$block_2$RUBY$__block__xx1.call(Unknown Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:125)
at org.jruby.runtime.BlockBody.call(BlockBody.java:72)
at org.jruby.runtime.BlockBody.call(BlockBody.java:78)
at org.jruby.runtime.Block.call(Block.java:89)
at org.jruby.RubyProc.call(RubyProc.java:221)
at org.jruby.RubyProc.call(RubyProc.java:204)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:94)
at org.jruby.internal.runtime.FutureThread$1.run(FutureThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

"Thread-7" daemon prio=10 tid=0x00002aacc8490400 nid=0x3d3c runnable [0x000000004264d000..0x000000004264ea20]
java.lang.Thread.State: RUNNABLE
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutInvokermethod__4$RUBY$raiseFixed1.call(timeout#raise)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutInvokermethod__4$RUBY$raiseFixed1.call(timeout#raise)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.block_4$RUBY$block(timeout.rb:102)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutBlockCallback$block_4$RUBY$block__xx1.call(Unknown Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:105)
at org.jruby.runtime.Block.yield(Block.java:194)
at org.jruby.RubyArray.eachCommon(RubyArray.java:1635)
at org.jruby.RubyArray.each(RubyArray.java:1642)
at org.jruby.RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.call(org/jruby/RubyArray$i_method_0_0$RUBYFRAMEDINVOKER$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:116)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:133)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.rescue_2$RUBY$__rescue___12(timeout.rb:100)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeout.block_2$RUBY$__block(timeout.rb:87)
at usr.local.lib.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.httpclient_minus_2_dot_1_dot_5_dot_2.lib.httpclient.timeoutBlockCallback$block_2$RUBY$__block__xx1.call(Unknown Source)
at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:125)
at org.jruby.runtime.BlockBody.call(BlockBody.java:72)
at org.jruby.runtime.BlockBody.call(BlockBody.java:78)
at org.jruby.runtime.Block.call(Block.java:89)
at org.jruby.RubyProc.call(RubyProc.java:221)
at org.jruby.RubyProc.call(RubyProc.java:204)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:94)
at org.jruby.internal.runtime.FutureThread$1.run(FutureThread.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

It looks like for some reason @pool is not empty for the period in the pool, period.@thread == nil. Thus, Peroid.raise does nothing but @pool is still not empty.

SSLConfig is very hard to set up with the right certs etc.

http://dev.ctor.org/http-access2/ticket/207

Possible data corruption problem in asynchronous methods

http://dev.ctor.org/http-access2/ticket/228

Possible hang-up problem in asynchronous methods

Hi,

It seems that asynchronous methods (get_async, post_async, etc.) does not raises an exception. Therefore errors cannot be handled.

How to reproduce the problem:

clnt = HTTPClient.new
conn = clnt.get_async( "http://non-existing-host/" )
res = conn.pop

I expected clnt.get_async or conn.pop raises an exception due to "host not found." But neither clnt.get_async nor conn.pop raises any exception, and conn.pop just blocks forever.

As far as I can see, the problem is in do_request_async method. What do_request_async does is:

1. Create a thread.
2. In the thread, create and send a HTTP request, then read response headers.
3. conn.push( res )

When an error occurred in 2), it just kills the thread created in 1), and 3) will never be executed so that invoking conn.pop blocks forever.

IMO, conn.pop should raise an exception if the thread is killed by exception. It allows users to handle an error in asynchronous methods like this:

clnt = HTTPClient.new
conn = clnt.get_async( "http://non-existing-host/" )
begin
  res = conn.pop
rescue
  abort $!
end

Here is a patch.

https://gist.github.com/714955

Regards,

SNI support

call SSLSocket#hosname=(hostname)

Post_content 302 redirect fails

I'm trying to implement login to Google Voice:

http://gist.github.com/506736

and redirection ends up in non-https page. Similar procedures in C# and Python work.

What might be wrong?

Cookie store set world-readable by default

http://dev.ctor.org/http-access2/ticket/198

"get" method gets blocked in "readpartial" when receiving a 304 with no Content-Length

http://dev.ctor.org/http-access2/ticket/224

Errors with read_body_chunked

The code is at https://github.com/nahi/httpclient/blob/master/lib/httpclient/session.rb#L906

Sometimes @socket.gets(RS) returns nil, so I'm getting a nil error because .hex is being called on nil.

I've been able to "solve" it by setting @chunk_length to 0 if len is nil, but this feels like a huge hack.

Please let me know what info I should provide for good debugging.

httpclient-2.1.6.1

ruby 1.8.7 (2010-04-19 patchlevel 253) [i686-darwin10.4.0], MBARI 0x6770, Ruby Enterprise Edition 2010.02

A way to track the progress of an async file upload and to cancel an async file upload

From an user: (Thanks!)

I've incorporated httpclient into one of my projects, and needed to add
a few features:

- A way to track the progress of an async file upload
- A way to cancel an async file upload
- A few more extensions for the default MIME handler

Connection#cancel cancels an ongoing upload, and Connection#written can
be used to ask the connection object how much data has been transferred
so far, in bytes.

I have gzipped and attached the output of 'svn diff'.

use *args for set that doesn't need args

taryneast@d180675

add gssapi auth support

zenchild@ad6e9ef

Redirect after POST req fails

http://dev.ctor.org/http-access2/ticket/208

Relative Location on https redirect w/ patch

I have an instance where a https request does a redirect to a relative path and it was not behaving nicely with httpclient. Since it was a relative link the 'newuri' being generated on line 571 of httpclient.rb was an instance of URI::Generic and would subsequently fail on the next line with:
"NoMethodError: undefined method `downcase' for nil:NilClass". The below link is a diff of my fix against the current tree. Have a look. If you don't want to implement it that way I can work around it for now.

http://gist.github.com/236661

Cheers,

Dan Wanek

Encoding of the response body and Ruby1.9

Hi, in Ruby1.9 when httpclient gets a response wth body, that body is encoded as US-ASCII even if the HTTP response contains:

Content-Type: lalala/lololo;charset=UTF-8

This forces me to convert the got body to UTF-8 by running:
body.force-encoding(:'UTF-8')

It would be great if httpclient would, by itself, encode the retrieved body in the appropiate endoding, which could be (ir orden or preference):

1. The value of "charset" param ni Content-Type header.
2. A default encoding set by the user when configuring the httpclient instance (or as a new parametner in 'request' method).

2.1.5.2 ssl_config.rb Segmentation Fault

Try to run Google AdWords API example (from adwords4r 19.0.0) examples/v200909/get_related_keywords.rb.

Got following error:
/ruby/gems/gems/httpclient-2.1.5.2/lib/httpclient/ssl_config.rb:348: [BUG] Segmentation fault
ruby 1.8.6 (2007-09-23) [i686-linux]

Aborted

My OS is ReadHat Enterprise Linux AS release 4 (Linux 2.6.9-42.ELswp)

My related gems are:

adwords4r (19.0.0)
httpclient (2.1.5.2)
rubygems-update (1.3.7, 1.3.6)
soap4r (1.5.8)
sources (0.0.1)

'qop' paramter must not be enclosed between doble quotation (")

httpclient generates "Authorization" header like this:

Authorization: Digest username="alice", realm="domain.org",
  nonce="MTI2MTE2NTY1MyBlMGIyMWE5NDRmNmZhZTgyMjRiZWU0NDU1MTg4Y2M3Yw==",
  uri="/pres-rules/users/sip:[email protected]/pres-rules",
  cnonce="MDAxMjYxMTY1NjUzOjIzZjI0MzVmMGRhZmU2M2Q5MGY2NDEzMTY4YjIzNzlm",
  nc=00000002, qop="auth", response="87b00e1898f635f03cdc3e3df4ebf8d7",
  algorithm="MD5", opaque="34437adb80c38b197974f5105052609a"

Note that 'qop' is:

qop="auth"

This is wrong according to RFC 2617 as in "Authorization" 'qop' value must not be enclosed between ":

RFC 2617 - Section 3.2.2 The Authorization Request Header

   credentials      = "Digest" digest-response
   digest-response  = 1#( username | realm | nonce | digest-uri
                   | response | [ algorithm ] | [cnonce] |
                   [opaque] | [message-qop] |
                       [nonce-count]  | [auth-param] )

   username         = "username" "=" username-value
   username-value   = quoted-string
   digest-uri       = "uri" "=" digest-uri-value
   digest-uri-value = request-uri   ; As specified by HTTP/1.1
   message-qop      = "qop" "=" qop-value
   cnonce           = "cnonce" "=" cnonce-value
   cnonce-value     = nonce-value
   nonce-count      = "nc" "=" nc-value
   nc-value         = 8LHEX
   response         = "response" "=" request-digest
   request-digest = <"> 32LHEX <">
   LHEX             =  "0" | "1" | "2" | "3" |
                       "4" | "5" | "6" | "7" |
                       "8" | "9" | "a" | "b" |
                       "c" | "d" | "e" | "f"

Note that 'message-qop' is:

message-qop      = "qop" "=" qop-value

and 'qop-value' is defined in section 3.2.1 as:

qop-value         = "auth" | "auth-int" | token

so it must appear without quotation.

However "WWW-Authenticate" header sent by the server contains a 'qop' parameter that must be enclosed between doble quotation as it allows varioes values separated by space:

3.2.1 The WWW-Authenticate Response Header

  challenge        =  "Digest" digest-challenge

  digest-challenge  = 1#( realm | [ domain ] | nonce |
                      [ opaque ] |[ stale ] | [ algorithm ] |
                      [ qop-options ] | [auth-param] )


  domain            = "domain" "=" <"> URI ( 1*SP URI ) <">
  URI               = absoluteURI | abs_path
  nonce             = "nonce" "=" nonce-value
  nonce-value       = quoted-string
  opaque            = "opaque" "=" quoted-string
  stale             = "stale" "=" ( "true" | "false" )
  algorithm         = "algorithm" "=" ( "MD5" | "MD5-sess" |
                       token )
  qop-options       = "qop" "=" <"> 1#qop-value <">
  qop-value         = "auth" | "auth-int" | token

Note that 'qop-options' is:

qop-options = "qop" "=" <"> 1#qop-value <">

But as said above, in client's request "Authorization" header 'qop' value must not be enclosed between ".

This patch fixes it: http://gist.github.com/259724

Digest auth results in exception

http://dev.ctor.org/http-access2/ticket/229

ssl_config.rb: 'require openssl' should be changed to 'require openssl.rb'

http://dev.ctor.org/http-access2/ticket/213

basic auth with POST-request

http://dev.ctor.org/http-access2/ticket/232

Need to set hostname

Hi!
I use httpclient to test my REST service. One of tests checks whether links are generated with address presented in HOST field of request.
To do that I need to set manualy host property and I get really angry when library overwrites my setting.
So I propose to change in file http.rb line 353:
if @http_version >= 1.1
to
if @http_version >= 1.1 and get('Host').empty?

I would love to see it in next update.

test against ruby/1.9 before 2.1.6 release

Posts not timing out

http://dev.ctor.org/http-access2/ticket/233

"rake" fails "no such file to load -- httpclient/util"

So "rake gem" cannot be executed.

OpenSSL::SSL::SSLError

if i make a "get" request on a ssl secured URL using HTTPClient, i get an
OpenSSL::SSL::SSLError saying:
certificate verify failed

this is the code:
def my_function
require 'httpclient'

'# create HTTP Object
my_url = "https://my_url.com/path/init?my_param=1&my_second_param=http://localhost/"
my_url = HTTPClient.new()

'# set Cookie

my_url.set_cookie_store('cookie.dat')

'# clear anchor-cert and load it localy:

my_url.ssl_config.clear_cert_store()
my_url.ssl_config.set_trust_ca('public/cert/GlobalSignRootCA.crt') # has no effect to the Error
'#my_url.ssl_config.set_trust_ca('public/cert/cacert_sha1.p7s') # also no effect
'#my_url.ssl_config.set_trust_ca('public/cert/cacert_1.p7s') # also no effect

'# testing Cert:

puts my_url.ssl_config.cert_store.to_s # seems to be ok
puts my_url.ssl_config.verify_callback() # is always nil

'# fire the get request

puts my_url.get_content(my_url_url) # this does not work
'#puts my_url.get_content('http://www.google.com/') # this would work :)

end

Is this a coding failure, or might this be a failure in HTTPClient?

httpclient uses String#size instead of String#bytesize so sets a wrong value in Ruby 1.9

http://dev.ctor.org/http-access2/ticket/235

httpclient uses "size" to calculate the body size in PUT requests, so in Ruby 1.9 it sets a wrong value (it should use "bytesize").

httpclient-2.1.5.2/lib/httpclient/http.rb:

      # Initialize this instance as a response.
      def init_response(body = nil)
        @body = body
        if @body.respond_to?(:size)
          @size = @body.size
        else
          @size = nil
        end
      end

If the body contains a "ñ" or "€" (or any other symbol which takes more than one byte in UTF-8) then "size" counts 1 while "bytesize" would count the real number of required bytes for such symbol).

A workaround could be something as:

if RUBY_VERSION < 1.9
  class String
    alias bytezise size
  end
end

And then use "bytesize" into the code, so it would work for 1.8 and 1.9.

Resource Leak: If httpclient establishes two connections to the same server in parallel, one of these connections will be leaked.

That is, these connections will stay in CLOSE_WAIT state forever (after the server side closes the connection) and httpclient will never close these connections. If this case (two connections to the same server) happen repeatedly, there is a resource like (e.g. a file descriptor leak) and httpclient will exhaust all available file descriptors and then fail miserably.

[RDOC] Public Class methods new(*args)

(from http://dev.ctor.org/http-access2/ticket/238)

In Public Class methods new(*args) description for the HTTPClient class,

HTTPClient.new(:agent_name = 'MyAgent/?0.1')

should be

HTTPClient.new(:agent_name => 'MyAgent/?0.1')

Not only "Negotiate" but also "NTLM"

I heard that IIS/6 + MicrosoftSharePointTeamServices uses "NTLM".
Both should be accepted

OAuth support (Accessing protected resources based on given access token)

http://dev.ctor.org/http-access2/ticket/227

Posting files causes timeout

http://dev.ctor.org/http-access2/ticket/226

test against jruby/1.5 before 2.1.6 release

HTTPClient::TimeoutError should subclass ::TimeoutError

Here's the scenario. Older code wrapped HTTPClient with a ::Timeout.timeout, but HTTPClient is handing it's own timeouts:

def my_method
  Timeout.timeout(120) do
    some_deep_method
  end
rescue ::Timeout => err
  take_evasive_action
end
def some_deep_method
  result = HTTPClient....
  process result
end

If HTTPClient's internal timeout mechanism raises a HTTPClient::ReceiveTimeoutError before the outer ::Timeout.timeout(), ReceiveTimeoutError is thrown, which is not a subclass of ::TimeoutError, thus take_evasive_action is never called and the exception is never caught.

The set of exceptions thrown by an API are a part of the API --- ReceiveTimeoutError is not a ::TimeoutError, thus the API was changed.

The fix is:

# httpclient.rb
...
require 'timeout'
...
class HTTPClient
  ...
  class TimeoutError < ::TimeoutError ; end
  ...
end

This fix should be backwards- and forwards- compatible.