cmullaparthi / ibrowse Goto Github PK
View Code? Open in Web Editor NEWErlang HTTP client
License: Other
Erlang HTTP client
License: Other
When I try to build ibrowse from commande line, it just works. When I try to automated it with a script, the %IBROWSE_VSN% is not resolved.
Why don't you use rebar?
It is random and rare error. I've seen it in logs only several times on random web-pages and cannot reproduce it by hands.
But i think, it is fixable using following error info (look at "Reason for termination" function to_ascii/1 arguments at the end):
=ERROR REPORT==== 23-May-2012::16:02:09 ===
** Generic server <0.323.0> terminating
** Last message in was {tcp,#Port<0.7054>,
<<72,84,84,80,47,49,46,49,32,50,48,48,32,79,75,13,
10,83,101,114,118,101,114,58,32,110,103,105,110,
120,13,10,68,97,116,101,58,32,87,101,100,44,32,
50,51,32,77,97,121,32,50,48,49,50,32,49,50,58,
48,50,58,48,57,32,71,77,84,13,10,67,111,110,116,
101,110,116,45,84,121,112,101,58,32,116,101,120,
116,47,104,116,109,108,59,32,99,104,97,114,115,
101,116,61,119,105,110,100,111,119,115,45,49,50,
53,49,13,10,84,114,97,110,115,102,101,114,45,69,
110,99,111,100,105,110,103,58,32,99,104,117,110,
107,101,100,13,10,67,111,110,110,101,99,116,105,
111,110,58,32,107,101,101,112,45,97,108,105,118,
101,13,10,75,101,101,112,45,65,108,105,118,101,
58,32,116,105,109,101,111,117,116,61,50,48,13,
10,67,97,99,104,101,45,67,111,110,116,114,111,
108,58,32,109,97,120,45,97,103,101,61,49,50,48,
44,32,112,117,98,108,105,99,13,10,69,120,112,
105,114,101,115,58,32,87,101,100,44,32,50,51,32,
77,97,121,32,50,48,49,50,32,49,50,58,48,52,58,
48,57,32,71,77,84,13,10,67,111,110,116,101,110,
116,45,69,110,99,111,100,105,110,103,58,32,103,
122,105,112,13,10,13,10,31,139,8,0,0,0,0,0,0,3,
3,0,0,0,0,0,0,0,0,0>>}
** When Server state == {state,"newsru.com",80,20000,undefined,false,
undefined,[],false,#Port<0.7054>,false,[],
{[{request,
{url,
"http://newsru.com/sport/24apr2012/bonus.html",
"newsru.com",80,undefined,undefined,
"/sport/24apr2012/bonus.html",http,
hostname},
head,
[{response_format,binary}],
{<0.174.0>,#Ref<0.0.3.130630>},
undefined,false,[],
{1337,774529,505993},
1048576,false,undefined,undefined,false,
binary,#Ref<0.0.3.130631>}],
[{request,
{url,"http://newsru.com","newsru.com",80,
undefined,undefined,"/",http,hostname},
head,
[{response_format,binary}],
{<0.161.0>,#Ref<0.0.3.126284>},
undefined,false,[],
{1337,774511,393636},
1048576,false,undefined,undefined,false,
binary,#Ref<0.0.3.126358>}]},
{request,
{url,"http://newsru.com","newsru.com",80,
undefined,undefined,"/",http,hostname},
head,
[{response_format,binary}],
{<0.161.0>,#Ref<0.0.3.126284>},
undefined,false,[],
{1337,774511,393636},
1048576,false,undefined,undefined,false,
binary,#Ref<0.0.3.126358>},
get_body,"200",<<>>,0,0,
[{"Server","nginx"},
{"Date","Wed, 23 May 2012 12:01:51 GMT"},
{"Content-Type",
"text/html; charset=windows-1251"},
{"Transfer-Encoding","chunked"},
{"Connection","keep-alive"},
{"Keep-Alive","timeout=20"},
{"Expires","Wed, 23 May 2012 12:02:21 GMT"},
{"Cache-Control","max-age=30"},
{"Serv","ng1"},
{"Content-Encoding","gzip"}],
undefined,undefined,false,undefined,false,chunked,
chunk_start,
<<31,139,8,0,0,0,0,0,0,3,3,0,0,0,0,0,0,0,0,0>>,
undefined,false,61473,2,undefined}
** Reason for termination ==
** {function_clause,[{ibrowse_http_client,to_ascii,"K"},
{ibrowse_http_client,hexlist_to_integer,3},
{ibrowse_http_client,parse_11_response,2},
{ibrowse_http_client,handle_sock_data,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
Version: latest git (2012-05-15).
Req Method: GET
Req URL: http://newsru.com/sport/24apr2012/bonus.html
Req Headers: [{"Accept-Encoding", "gzip"}]
Req Params: [{response_format, binary}]
What do you think about it? Buggy server or problems with chunked encoding?
Hi Chandru,
The following patch (I polished it a bit to reduce code size) fixes issues (ibrowse hanging or timing out) when streaming chunked encoded responses:
http://github.com/fdmanana/ibrowse/commit/e9ce3f9e999cc10bb971091b8fee18edfd106f58
Let me know if it's fine for you.
cheers
I think ibrowse is choking because news.ycombinator.com lacks a content length (pg uses his own custom non-complaint web server for news.yc).
I tried to get it working for a few hours, but ended up using httpc:request instead.
Try for yourself: ibrowse:send_req("http://news.ycombinator.com/", [], get). Using v2.2.0 just returns {error, retry_later}.
Hi,
I encountered today an error when doing an HTTPS POST Request and I don't really know why.
The following error is printed:
{'EXIT',
{{function_clause,
[{ibrowse_http_client,
'-make_request/8-lc$^0/1-0-',
[{"Host",
["example.com",
58,
"443"]},
#Fun<ibrowse_http_client.1.70135144>]},
{ibrowse_http_client,
'-make_request/8-lc$^0/1-0-',
2},
{ibrowse_http_client,
make_request,
8},
{ibrowse_http_client,
send_req_1,
8},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]},
{gen_server,
call,
[<0.543.0>,
{send_req,
{{url,
"https://example.com:443/someresource",
"example.com",
443,
undefined,
undefined,
"/someresource",
https,
hostname},
[{"Content-Type",
"application/json;charset=UTF-8"},
{"Authorization",
"SomeAuthorization"},
{"User-Agent",
"SomeUserAgent"},
{"Accept",
"application/json;charset=UTF-8"}|
{"Host",
["example.com",
58,
"443"]}],
post,
<<"{\"badge\":1}">>,
[],
30000}},
30000]}}}
It would be nice if someone could shed some light on this.
Of course ibrowse:start()
and ssl:start()
has been called.
Code is the following:
handle_cast({example_api, UserId, Count}, State) ->
Url = lists:flatten(io_lib:format("~s/~s", [State#state.url, UserId])),
Headers = State#state.headers,
Body = lists:flatten(mochijson:encode({struct, [{badge, Count}]})),
?DEBUG("Body: ~p", [Body]),
?DEBUG("Headers: ~p", [Headers]),
?DEBUG("Url: ~p", [Url]),
case ibrowse:send_req(Url, Headers, post, Body) of
{ok, Status, ResponseHeaders, ResponseBody} ->
case Status of
"202" ->
{noreply, State};
"400" ->
?ERROR_MSG("Got a 400 BAD Request.~nResponseHeaders: ~p~nResponseBody: ~p",
[ResponseHeaders, ResponseBody]),
{noreply, State}
end;
{error, Reason} ->
?ERROR_MSG("Error, reason: ~p", [Reason]),
{noreply, State}
end.
After compiling with ERL_COMPILER_OPTIONS=bin_opt_info set for binary use warnings, this was printed.
It might be good to fix these.
src/ibrowse_http_client.erl:974: Warning: NOT OPTIMIZED: called function chunk_request_body/3 does not begin with a suitable binary matching instruction
src/ibrowse_http_client.erl:999: Warning: variable 'State' is unused
src/ibrowse_http_client.erl:1494: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1497: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1509: Warning: INFO: using a matched out sub binary will prevent delayed sub binary optimization
src/ibrowse_http_client.erl:1509: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1513: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1516: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1524: Warning: OPTIMIZED: creation of sub binary delayed
src/ibrowse_http_client.erl:1530: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1544: Warning: INFO: using a matched out sub binary will prevent delayed sub binary optimization
src/ibrowse_http_client.erl:1544: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1548: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1559: Warning: OPTIMIZED: creation of sub binary delayed
src/ibrowse_http_client.erl:1634: Warning: OPTIMIZED: creation of sub binary delayed
I noticed that ibrowse was polluting the virtualmachine with atoms when passed bogus URLs.
The functions ibrowse_lib:parse_url calls list_to_atom to get the URL protocol but if it's not a well formed URL it will pollute the node with useless atoms.
I see 2 possible solutions:
Which means it isn't found in the standard search path which looks for things in include. If possible can this file be moved into the include directory.
If I try to make and https request from behind a proxy I always get {error,conn_failed}. The same request over http succeeds.
If I try to set ssl_options like this {ssl_options, [{ssl_imp, new}]} I instead get the exception
=ERROR REPORT==== 20-Apr-2010::14:44:57 ===
SSL: hello: ./ssl_record.erl:292:Fatal error: record_overflow
{error,conn_failed}
Is it supported to use SSL from behind a proxy?
I am seeing errors in my log like
** Generic server my_server terminating
** Last message in was timeout
** When Server state == ...
** Reason for termination ==
** {timeout,
{gen_server,call,
[ibrowse,
{get_lb_pid,
{url,
"http://192.168.1.103:80/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
"192.168.1.103",80,undefined,undefined,
"/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
http,ipv4_address}}]}}
and
** Reason for termination ==
** {timeout,
{gen_server,call,
[<0.118.0>,
{spawn_connection,
{url,
"http://192.168.1.103:80/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
"192.168.1.103",80,undefined,undefined,
"/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
http,ipv4_address},
10,10,
{[],false}}]}}
after calling ibrowse:send_req/5
with timeout infinity
.
Hi,
I am working on one project, which need show the download speed during a interval repeatly. I know i can use {stream_to, self()} to handle the write file part. but it is not efficient engouth that data passed as message and write to file.
So is it possible to add one option which is used to get the download speed infomation when use save_response_to_file for send_req/send_req_direct?
Thank you.
I've been using ibrowse_lib to parse urls and have found that the behaviour is inconsistent on bad input.
1> ibrowse_lib:parse_url("http://domain:not_a_port").
** exception error: bad argument
in function list_to_integer/1
called as list_to_integer("not_a_port")
in call from ibrowse_lib:parse_url/4
in call from ibrowse_lib:parse_url/1
2> ibrowse_lib:parse_url("http://domain:80:").
** exception error: bad argument
in function list_to_integer/1
called as list_to_integer("80:")
in call from ibrowse_lib:parse_url/4
in call from ibrowse_lib:parse_url/1
3> ibrowse_lib:parse_url("http:/domain:80:").
{invalid_uri_2,get_protocol,
{url,"http:/domain:80:",undefined,undefined,undefined,
undefined,undefined,undefined,undefined},
":08:niamod/:ptth"}
As I see it the function should either always crash, or always return some sort of error message. Not chose randomly depending on the reason the input is wrong.
websocket is new and coming, it would be fantastic to see websocket support in ibrowse.
Hi,
I've been experiencing a weird bug where some of the ibrowse connections die and are never restarted.... It usually take about 24 hours at 2500 QPS to kill all of the 100 connections (max_sessions).
Here is what my request looks like:
Body = "{\"name\": \"Bob\"}",
Headers = [{"Connection", "Keep-Alive"},
{"Content-Type", "application/json"}],
Options = [{connect_timeout, ?CONNECTION_TIMEOUT},
{max_sessions, 100},
{max_pipeline_size, 1},
{stream_to, self()}],
ibrowse:send_req("69.90.7.1:9950", Headers, post, Body, Options, 50)
If you check the status everything looks good....
ibrowse:show_dest_status().
Server:port | ETS | Num conns | LB Pid
================================================================================
69.90.7.1:9950 | 28699 | 100 | <0.658.0>
ok
But if you check netstat, you'll see that there's only 76 sockets open and not 100!
netstat -n | grep 69.90.7.1:9950 | grep ESTABLISHED | wc -l
76
Now If you dig into the LB process, everything looks good...
process_info(list_to_pid("<0.658.0>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.7547.2582>,<0.9962.3775>,<0.30613.3784>,
<0.32157.3784>,<0.1168.3785>,<0.1521.3785>,<0.1522.3785>,
<0.1525.3785>,<0.1176.3785>,<0.32166.3784>,<0.32170.3784>,
<0.1166.3785>,<0.32167.3784>,<0.32162.3784>,<0.32163.3784>,
<0.32158.3784>,<0.30624.3784>,<0.32154.3784>,<0.32155.3784>,
<0.30625.3784>,<0.30622.3784>|...]},
{dictionary,[{my_trace_flag,false},
{ibrowse_trace_token,["LB: ","69.90.7.1",58,"9950"]},
{'$ancestors',[ibrowse,ibrowse_sup,<0.288.0>]},
{'$initial_call',{ibrowse_lb,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.287.0>},
{total_heap_size,3571},
{heap_size,2584},
{stack_size,9},
{reductions,1530637035},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,16917}]},
{suspending,[]}]
But when you check some of it's children... you can see that there's no link to a #Port!
process_info(list_to_pid("<0.7547.2582>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.658.0>]},
{dictionary,[{my_trace_flag,false},
{ibrowse_trace_token,["69.90.7.1",58,"9950"]},
{'$ancestors',[<0.658.0>,ibrowse,ibrowse_sup,<0.288.0>]},
{'$initial_call',{ibrowse_http_client,init,1}}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.287.0>},
{total_heap_size,233},
{heap_size,233},
{stack_size,9},
{reductions,42},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]
Any ideas?
Hi Chandru,
I stumbled into this when browsing the source on github:
tcp_error comes with 2 parameters: {tcp_error, Socket, Reason}.
handle_info({tcp_error, _Sock}, State) in ibrowse_http_client probably won't match it.
I suspect it might leave dead Pids in ETS tables.
BR,
Ferenc
Hello!
While packaging your application for Fedora, I stumbled over one issue - we didn't come in agreement about correct license tag for ibrowse - is it available under either BSD or LGPLv2+ license or both licenses' terms must be applied simultaneously (e.g. BSD and LGPLv2+)?
See this link for the details:
https://bugzilla.redhat.com/show_bug.cgi?id=581279#c5
This greatly helps in packaging ibrowse. GitHub automatically creates download links for each tag and there is no need to dig into sources to find exact commit for particular version.
cc: @nivertech, @oribrost
$ make
sed -e s^%IBROWSE_VSN%^1.5.6^ ibrowse.app.src > ../ebin/ibrowse.app
/bin/sh: ibrowse.app.src: not found
/bin/sh: %IBROWSE_VSN%: not found
/bin/sh: 1.5.6: not found
make: *** [../ebin/ibrowse.app] Error 1
sed: command garbled: s
Is it possible to add debug_info in the list of compiler options?
diff --git a/src/Makefile b/src/Makefile index 145c0a2..9c1bf63 100644 --- a/src/Makefile +++ b/src/Makefile @@ -13,7 +13,7 @@ INCLUDE_DIRS = -I./ ERLC ?= erlc ERLC_EMULATOR ?= erl -boot start_clean -COMPILER_OPTIONS = -W +warn_unused_vars +nowarn_shadow_vars +warn_unused_import +COMPILER_OPTIONS = -W +debug_info +warn_unused_vars +nowarn_shadow_vars +warn_unused_import
Also, you may want to run dialyzer with the -Wunmatched_returns option and possibly correct these warnings. I can help, if needed.
Kostis
is there any options for urls redirection 302 "Moved Temporally".. thank's
Seems a little odd to have a single compilation warning.
deps/ibrowse/src/ibrowse_http_client.erl:999: Warning: variable 'State' is unused
[ibrowse ((v4.0.1))]$ git log -n 1
commit 9b8b042
Author: Chandrashekhar Mullaparthi [email protected]
Date: Tue Aug 7 07:05:37 2012 +0100
Fixed issue #67 with eyes wide open
Hi Chandru,
The following patch adds support for adding chunked transfer-encoding when streaming requests with a user supplied function:
http://github.com/fdmanana/ibrowse/commit/6a6852c8b59de79a52c2c00d57909c006a7e635a
After recent update, parse_url() became unable to properly parse ipv6 as hostname:
1> ibrowse_lib:parse_url("http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html").
{error,invalid_uri}
2> ibrowse_lib:parse_url("http://[1080:0:0:0:8:800:200C:417A]/index.html").
{error,invalid_uri}
3> ibrowse_lib:parse_url("http://[::192.9.5.5]/ipng").
{error,invalid_uri}
Example URLs is taken from http://www.ietf.org/rfc/rfc2732.txt
Regression is not reproduceable before this commit: 26d34d5
Hi Chandru.
I found out that, with ibrowse 2.0.x, if I use a worker for several request and the first request had an activity timeout X specified in the options, the timer is never cleared, causing a subsequent request (after X milliseconds) to fail.
Patch at:
http://github.com/fdmanana/ibrowse/commit/072d4f865fb77609e6cd75639a36264fbfb19933
Cheers
There was a regression in save_response_to_file option in v2.1.3. Instead of returning a response in the non-2xx case it throws a badmatch. This exact request works fine in 2.1.2.
7> ibrowse:send_req("http://www.google.com/tsdfasd", [], get, [], [{save_response_to_file, "response"}], 10000).
{error,{'EXIT',{{{badmatch,{error,badarg}},
[{ibrowse_http_client,handle_response,2},
{ibrowse_http_client,parse_11_response,2},
{ibrowse_http_client,handle_sock_data,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
[<0.47.0>,
{send_req,{{url,"http://www.google.com/tsdfasd",
"www.google.com",80,undefined,undefined,"/tsdfasd",http},
[],get,<<>>,
[{save_response_to_file,"response"}],
10000}},
10000]}}}}
8>
=ERROR REPORT==== 31-Jan-2011::10:35:49 ===
** Generic server <0.47.0> terminating
** Last message in was {tcp,#Port<0.706>,
<<"=0 cellspacing=0><tr><td bgcolor=\"#3366cc\"><img alt=\"\" width=1 height=4></td></tr></table>\n</body></html>\n">>}
** When Server state == {state,"www.google.com",80,10000,undefined,false,
undefined,[],false,#Port<0.706>,false,[],
{[{request,
{url,"http://www.google.com/tsdfasd",
"www.google.com",80,undefined,undefined,
"/tsdfasd",http},
get,
[{save_response_to_file,"response"}],
{<0.33.0>,#Ref<0.0.0.73>},
undefined,false,[],
{1296,488149,148952},
1048576,"response",undefined,undefined,
false,list}],
[]},
{request,
{url,"http://www.google.com/tsdfasd",
"www.google.com",80,undefined,undefined,
"/tsdfasd",http},
get,
[{save_response_to_file,"response"}],
{<0.33.0>,#Ref<0.0.0.73>},
undefined,false,[],
{1296,488149,148952},
1048576,"response",undefined,undefined,false,
list},
get_body,"404",
<<"\n\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>404 Not Found</title>\n<style><!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--></style>\n<script><!--\nvar rc=404;\n//-->\n</script>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<table border=0 cellpadding=2 cellspacing=0 width=100%><tr><td rowspan=3 width=1% nowrap>\n<b><font face=times color=#0039b6 size=10>G</font><font face=times color=#c41200 size=10>o</font><font face=times color=#f3c518 size=10>o</font><font face=times color=#0039b6 size=10>g</font><font face=times color=#30a72f size=10>l</font><font face=times color=#c41200 size=10>e</font> </b>\n<td> </td></tr>\n<tr><td bgcolor=\"#3366cc\"><font face=arial,sans-serif color=\"#ffffff\"><b>Error</b></td></tr>\n<tr><td> </td></tr></table>\n<blockquote>\n<H1>Not Found</H1>\nThe requested URL <code>/tsdfasd</code> was not found on this server.\n\n<p>\n</blockquote>\n<table width=100% cellpadding">>,
1255,0,
[{"Content-Type","text/html; charset=UTF-8"},
{"X-Content-Type-Options","nosniff"},
{"Date","Mon, 31 Jan 2011 15:35:49 GMT"},
{"Server","sffe"},
{"Content-Length","1361"},
{"X-XSS-Protection","1; mode=block"}],
undefined,undefined,false,#Ref<0.0.0.77>,1361,
false,undefined,undefined,<<>>,undefined,false,
32787,1,undefined}
** Reason for termination ==
** {{badmatch,{error,badarg}},
[{ibrowse_http_client,handle_response,2},
{ibrowse_http_client,parse_11_response,2},
{ibrowse_http_client,handle_sock_data,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
I tried requesting
ibrowse:send_req("http://www.google.com", [], get, [], [{save_response_to_file, "/tmp/foo.txt"}]).
and expected a file /tmp/foo.txt but instead the response body was returned. {save_response_to_file, true} worked just fine.
Self explanatory.
In ibrowse_http_client:make_request/7 there's a call to ibrowse_lib:get_value(content_length, Headers, false) and ibrowse_lib:get_value(transfer_encoding, Headers, false). get_value does not attempt to check if the header exists as a string which ends up causing things like having multiple Content-Length headers set.
http://github.com/cmullaparthi/ibrowse/blob/master/src/ibrowse_http_client.erl#L806
I haven't checked the details but there are more than a few dialyzer warnings that might be of interest.
$ dialyzer --plt ~/.dialyzer_plt.R14B03 -Wunmatched_returns -r ./
ibrowse_http_client.erl:191: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:294: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:318: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:331: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:348: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:570: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:573: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:576: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:692: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:775: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:783: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:1374: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:79: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:80: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:81: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:145: Expression produces a value of type 'ok' | integer(), but this value is unmatched
ibrowse_test.erl:189: Expression produces a value of type 'ok' | {'error',atom()}, but this value is unmatched
ibrowse_test.erl:233: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:234: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:235: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:237: Expression produces a value of type 'ignore' | {'error',} | {'ok',pid()}, but this value is unmatched
ibrowse_test.erl:379: Expression produces a value of type 'ok' | {'error','unknown_req_id'}, but this value is unmatched
ibrowse_test.erl:386: Expression produces a value of type 'ok' | {'error','unknown_req_id'}, but this value is unmatched
Unknown functions:
ibrowse_test_server:start_server/2
done in 0m24.47s
done (warnings were emitted)
Following error reported by Sergey Samokhi.
I found that when calling the following function:
test() ->
ibrowse:start(),
ibrowse:send_req("http://google.com/", [], get),
ibrowse:stop().
ibrowse:stop() causes an error
=ERROR REPORT==== 31-Jan-2012::01:34:41 ===
** Generic server <0.160.0> terminating
** Last message in was stop
** When Server state == {state,<0.158.0>,36883,"google.com",80,10,10,1,
undefined}
** Reason for termination ==
** {function_clause,[{ibrowse_lb,'-handle_call/3-fun-0-',
[{<0.161.0>,0,0},[]],
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
{ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
{ets,foldl,3,[{file,"ets.erl"},{line,187}]},
{ibrowse_lb,handle_call,3,
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,578}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}
=ERROR REPORT==== 31-Jan-2012::01:34:41 ===
** Generic server ibrowse terminating
** Last message in was stop
** When Server state == {state,false}
** Reason for termination ==
** {{case_clause,
{'EXIT',
{{function_clause,
[{ibrowse_lb,'-handle_call/3-fun-0-',
[{<0.161.0>,0,0},[]],
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
{ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
{ets,foldl,3,[{file,"ets.erl"},{line,187}]},
{ibrowse_lb,handle_call,3,
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,578}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]},
{gen_server,call,[<0.160.0>,stop]}}}},
[{ibrowse_lb,stop,1,[{file,"src/ibrowse_lb.erl"},{line,93}]},
{ibrowse,'-handle_call/3-fun-0-',2,[{file,"src/ibrowse.erl"},{line,820}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
{ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
{ets,foldl,3,[{file,"ets.erl"},{line,187}]},
{ibrowse,handle_call,3,[{file,"src/ibrowse.erl"},{line,819}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,578}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
{'EXIT',
{{{case_clause,
{'EXIT',
{{function_clause,
[{ibrowse_lb,'-handle_call/3-fun-0-',
[{<0.161.0>,0,0},[]],
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
{ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
{ets,foldl,3,[{file,"ets.erl"},{line,187}]},
{ibrowse_lb,handle_call,3,
[{file,"src/ibrowse_lb.erl"},{line,115}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,578}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]},
{gen_server,call,[<0.160.0>,stop]}}}},
[{ibrowse_lb,stop,1,[{file,"src/ibrowse_lb.erl"},{line,93}]},
{ibrowse,'-handle_call/3-fun-0-',2,
[{file,"src/ibrowse.erl"},{line,820}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
{ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
{ets,foldl,3,[{file,"ets.erl"},{line,187}]},
{ibrowse,handle_call,3,
[{file,"src/ibrowse.erl"},{line,819}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,578}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]},
{gen_server,call,[ibrowse,stop]}}}
Hello,
I have tried to run a make test
on the master and I get this error:
[nrdufour 01:40 PM] ~/tmp/ibrowse $ make test
./rebar compile
==> ibrowse (compile)
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_test.erl
Compiled src/ibrowse_app.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse.erl
Compiled src/ibrowse_http_client.erl
./rebar eunit
==> ibrowse (eunit)
======================== EUnit ========================
module 'ibrowse'
module 'ibrowse_app'
module 'ibrowse_http_client'
module 'ibrowse_lb'
module 'ibrowse_lib'
ibrowse_lib_tests: parse_urls_test_ (module 'ibrowse_lib_tests')...*failed*
::{assertMatch_failed,[{module,ibrowse_lib_tests},
{line,127},
{expression,"ibrowse_lib : parse_url ( \"http://[:1080:0:0:0:8:800:200C:417A:]:6000/foo?q=bar\" )"},
{pattern,"{ error , { invalid_ipv6_address , \":1080:0:0:0:8:800:200C:417A:\" } }"},
{value,{error,invalid_uri}}]}
[done in 0.003 s]
module 'ibrowse_sup'
ibrowse_test: ue_test (module 'ibrowse_test')...[0.004 s] ok
module 'ibrowse_test_server'
=======================================================
Failed: 1. Skipped: 0. Passed: 1.
ERROR: One or more eunit tests failed.
make: *** [test] Error 1
Any idea?
I think infinity
is the more expected value.
Hi Chandru,
Could you add a target to run the tests?
http://github.com/fdmanana/ibrowse/compare/makefile_tests
I think this is fine and useful.
cheers
Hi Chandrashekhar,
First of all, thanks for ibrowse. A very useful, and well done, library.
We use it in CouchDB, and I was about to apply the following patch to the ibrowse version shipped with CouchDB:
http://github.com/fdmanana/ibrowse/commit/1bdc1185c67c3845969996ad1b3824be9b4afa8d
Do you think you can apply it as well to standard ibrowse? We plan to bump our version to the latest soon, so it would be great to have our version as close to yours as possible.
Thanks, and keep up the good work.
Hi there,
I think I found two more bugs...
find_best_connection(Tid, Max_pipe) ->
case ets:first(Tid) of
{Cur_sz, Pid} when Cur_sz < Max_pipe ->
ets:delete(Tid, {Cur_sz, Pid}),
ets:insert(Tid, {{Cur_sz + 1, Pid}, []}),
{ok, Pid};
X ->
error_logger:error_report([Tid, Max_pipe, X]),
{error, retry_later}
end.
Now that I have some visibility I can see that I get {error, retry_later} because of the gard (Cur_sz < Max_pipe). That's ok and expected...
=ERROR REPORT==== 19-May-2011::16:29:13 ===
{ibrowse,retry_later}
=ERROR REPORT==== 19-May-2011::16:29:13 ===
28699
1
1: <0.469.0>
But if I check the state of the process, I can see that it's idle and that the cur_pipeline_size = 0!
io:format("~p~n", [sys:get_status(list_to_pid("<0.469.0>"))]).
{status,<0.469.0>,
{module,gen_server},
[[{my_trace_flag,false},
{ibrowse_trace_token,["69.90.7.149",58,"9950"]},
{'$ancestors',[<0.463.0>,ibrowse,ibrowse_sup,<0.288.0>]},
{http_prot_vsn,"HTTP/1.1"},
{conn_close,"keep-alive"},
{'$initial_call',{ibrowse_http_client,init,1}}],
running,<0.463.0>,[],
[{header,"Status for generic server <0.469.0>"},
{data,[{"Status",running},
{"Parent",<0.463.0>},
{"Logged events",[]}]},
{data,[{"State",
{state,"69.90.7.149",9950,10,#Ref<0.0.33.237942>,false,
undefined,[],false,#Port<0.4923>,false,[],
{[],[]},
undefined,idle,undefined,<<>>,0,0,[],undefined,
undefined,false,#Ref<0.0.33.237748>,undefined,false,
undefined,undefined,<<>>,undefined,false,28699,0,
{1305,836006,880134}}}]}]]}
That means that the ets state != process state...
Any ideas?
vsn.mk != ibrowse.app
When I call
ibrowse:send_req(Url, [], get, [],
[{save_response_to_file,
"/mware/working_dir/media/video/7eb8971f-0527-4c6c-af93-11aa0afb0c1a.avi"},
{socket_options,[{keepalive,true}]},
{inactivity_timeout,180000},
{connect_timeout,180000}])
and the request times out, it returns {error,req_timedout}
. But then the process which called send_req
also receives a message {#Ref<0.0.0.821>,{error,req_timedout}}
, which is not expected, since there is no {stream_to, self()}
option.
Hi Chandru. The commit message explains it all:
Found this issue when trying to send a chunk with size > 65535 bytes.
Replaced the implementation of dec2hex with a call to io_lib:format/2. Also we don't need the width anywhere in current ibrowse's code (and chunk headers don't apply restrictions to the size's width).
cheers
ibrowse never timesout on a open connection if the server never closes the connection.
This will cause an application that connects to several different domains to eventually exhaust all file descriptors.
I suggest the use of some sort of argument to explicitly pass a timeout in seconds or infinity to a send_req.
The connection should be closed after the timeout has expired.
After starting a download with
25/Apr/2011 12:59:24.091 Debug: ibrowse options: [{save_response_to_file,
{append,
"/mware/working_dir/media/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi"}},
{socket_options,[{keepalive,true}]},
{inactivity_timeout,180000},
{connect_timeout,180000}]
(and infinity
timeout), there is more than 3 minutes of inactivity but the connection isn't broken.
{'EXIT',
{{function_clause,
[{ibrowse_http_client,'-parse_response/2-lc$^0/1-0-',
[<<>>],
[{file,"src/ibrowse_http_client.erl"},
{line,1012}]},
{ibrowse_http_client,parse_response,2,
[{file,"src/ibrowse_http_client.erl"},
{line,1012}]},
{ibrowse_http_client,handle_sock_data,2,
[{file,"src/ibrowse_http_client.erl"},{line,285}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,597}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]},
{gen_server,call,
[<4646.13865.4>,
{send_req,
{{url,
"http://foohost:8010/foopath",
"foohost",8010,undefined,
undefined,
"/foopath",
http,hostname},
[],head,<<>>,
[{max_sessions,100},
{max_pipeline_size,1000},
{response_format,list}],
30000}},
30000]}}}}
ibrowse never seems to clean up the gen_servers it creates, making it difficult to use for crawling. There could be an option to limit the maximum number of load balancers / clients it creates.
Please, read the comment of the following commit:
http://github.com/fdmanana/ibrowse/commit/8b963892acbdb7d9e42ef6f59ff76717c93a0742
cheers
It would be nice to have Oauth 2. support in Erlang. Is ibowse good place for that?
https://github.com/tim/erlang-oauth/issues/4#comment_1218849
Regards
Matteo
http://www.redaelli.org/matteo/
cc: @oribrost , @nivertech
{vsn, git} is not being correctly replaced when used as a rebar dependency eg
{deps, [{ibrowse, "2.2.*", {git, "git://github.com/cmullaparthi/ibrowse", "HEAD"}}]}
fails with
Uncaught error in rebar_core: {'EXIT',
{badarg,
[{re,run,[git,"2.2.0",[{capture,none}]]},
{rebar_deps,is_app_available,3},
{rebar_deps,find_deps,3},
{rebar_deps,preprocess,2},
{rebar_core,acc_modules,5},
{rebar_core,process_dir,4},
{rebar_core,process_commands,1},
{rebar,main,1}]}}
make: *** [deps] Error 1
Does ibrowse support ssl certificate alternative names?
If I run
ibrowse:send_req("https://github.com", [], get).
it fails with
{error,{conn_failed,{error,esslconnect}}}
httpc:request("https://github.com").
on the other hand works fine (but is it checking the cert in any way???)
I'm not sure if the issue is alternative names, but it looks like it. Here's the discussion:
https://github.com/blog/738-sidejack-prevention-phase-2-ssl-everywhere#comment-9002
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.