bitly/simplehttp

a family of libraries and daemons for building scalable web infrastructure

simplehttp's Issues

ps_to_http should take a verbose param to enable/disable request logging

n/t

simplehttp closecb for async handlers

because we don't currently explicitly set evhttp_connection_set_closecb there is a bug where asynchronous handlers could dereference invalid request or connection objects when the remote hangs up before the request was finished.

simplehttp could facilitate cleanly handling both success and failure cases.

simpleleveldb /fwmatch /range_match /mget only support json format

they should also support a simple csv like txt format

when installing module gcc complains 'undefined reference to `current_base`' in simplehttp.c

I'm using libevent 2.0.12.
After "cd simplehttp ; make && make install", I switched to simplequeue to make install, but gcc began to complain, saying "../simplehttp/libsimplehttp.a(simplehttp.o): In function simplehttp_init': /home/lamusicoscos/Base/bitly-simplehttp-10406dc/simplehttp/simplehttp.c:152: undefined reference tocurrent_base'
collect2: ld returned 1 exit status make: *** [simplequeue] Error 1".
But being with libevent 1.4, everything's OK. I'm just wondering why it happened and how can I make it work with libevent 2.0.12+.

ps_to_http results in "ERROR: request failed"

Hi all,
I've been trying to follow the tutorial found here: http://micha.gd/realtimestream/ .
I am using libevent-1.4.14 . I get to the point in the tutorial where I issue this command:

%> ps_to_http --destination-get-url="http://127.0.0.1:8081/put?data=%s" --pubsub-url="http://127.0.0.1:8080/sub?multipart=0"

It works fine for a few minutes [pub/ sub/ requests ] but the it dies with this error:

ERROR: request failed

Any ideas what I might be doing wrong?

pubsubclient (ps_to_http, ps_to_file) should support basic auth

add basic auth support to pubsubclient and add command line options to ps_to_http and ps_to_file

Mac compatability in makefile

changed install -D to use the more compatable -d

Untitled

simpleleveldb leak in fwmatch_cb()

from valgrind, during the included simpleleveldb tests:

==25024== 24 bytes in 1 blocks are indirectly lost in loss record 4 of 22
==25024==    at 0x4C21430: calloc (vg_replace_malloc.c:418)
==25024==    by 0x504341F: array_list_new (arraylist.c:31)
==25024==    by 0x50439B4: json_object_new_array (json_object.c:491)
==25024==    by 0x404FCE: fwmatch_cb (simpleleveldb.c:463)
==25024==    by 0x40606E: generic_request_handler (simplehttp.c:132) 
==25024==    by 0x4E33F21: evhttp_get_body (http.c:1592)
==25024==    by 0x4E2D6F7: event_base_loop (event.c:395)
==25024==    by 0x40665A: simplehttp_run (simplehttp.c:300)
==25024==    by 0x406685: simplehttp_main (simplehttp.c:308)
==25024==    by 0x40453C: main (simpleleveldb.c:922)

HTTP 1.1 connection leaks

according to http.c:1886 in libevent source we should be able to set outbound 'Connection: close' headers in order to circumvent the HTTP 1.1 related leaks we encounter.

simpleleveldb json format transitions

json format responses of some simpleleveldb endpoints is not the way we'd prefer. We'd like to identify what currently uses the current structure and transition them to a new structure that puts all response data somewhere in the value of the "data" member of the root, and include useful keys and metadata.

undefined reference to 'json_object_new_string'

got a err when compile simpleleveldb on ubuntu 12.04

cc -I. -I../simplehttp/.. -I/usr/local/include -I/usr/local/lib//include -Wall -g -O2 -MM -MT simpleleveldb -MF simpleleveldb.deps simpleleveldb.c str_list_set.c
cc -I. -I../simplehttp/.. -I/usr/local/include -I/usr/local/lib//include -Wall -g -O2 -o simpleleveldb simpleleveldb.c str_list_set.c -L. -L../simplehttp -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib//lib -levent -ljson -lsimplehttp -lleveldb -lm -lstdc++ -lsnappy -lpthread
/tmp/cccCgPcZ.o: In function finalize_request': /temp/project/simplehttp/simpleleveldb/simpleleveldb.c:103: undefined reference tojson_object_new_string'

pubsub leaves many file descriptors open

I have a pubsub instance not doing much and using 28135 connections, 573 of which are TCP and 27496 of which are (according to lsof):

pubsub 3392 root *462u sock 0,5 0t0 36281617 can't identify protocol

where the 462 is incrementing.

I think this is related to the HTTP/1.1 issue; when the HTTP/1.1 client goes away, that fd is never cleaned up. #44

Update pysimplehttp

pysimplehttp has been updated and BaseReader has been added in

queuereader lib should handle heartbeat

queuereader lib should handle heartbeat messages and not pass them to the callback

simplehttp async connection pool freeing

simplehttp async connection pool resources are never all freed, even if there is no connection activity for a long time. This can contribute to memory fragmentation and pin large heap regions, so even if the application using simplehttp async connections naturally frees all its other resources, it still has an RSS near its maximum.

Maybe async connection pool resources should be freed if all connections they are related to are done and gone for more than a few seconds.

simpleleveldb /mput could write some records then return error

maybe it should check all input before writing to the db?
maybe it should return an indication of how many records it wrote successfully?

queuereader lib takes an errorcb and never calls it

what the title says

simpleleveldb does not correctly state Content-Type

http = httpclient.HTTPClient()
url = "http://localhost:13731/get"
params = urllib.urlencode({'key':'f00'})
url += '?' + params
resp = http.fetch(url)
print resp.headers.get('Content-Type', "No content type!") # prints: text/html; charset=ISO-8859-1

Should be application/json or text/javascript

ps_to_file , segfault on directory not existing or improper permissions

Currently ps_to_file will segfault if it does not have write permission on the file specified or if that directory does not exist.

test

test

merge ps_to_http and ps_to_file to ps_reader

simplequeue page alloc

Update simplequeue to use page based allocation. This will allow linux to reclaim free'd memory regardless of heap position.

Add mput call to simplequeue

Add a multiput call (mput) to simplequeue. Allows posting multiple items at once to the queue separated by either a default separator of newline or a user specified separator. The separator can be one or more characters.