jbittel / httpry Goto Github PK

HTTP logging and information retrieval tool

Home Page: dumpsterventures.com/jason/httpry

C 48.37% Perl 50.41% CSS 0.78% Shell 0.45%

httpry's Introduction

                        _     _   _
                       | |   | | | |
                       | |__ | |_| |_ _ __  _ __ _   _
                       | '_ \| __| __| '_ \| '__| | | |
                       | | | | |_| |_| |_) | |  | |_| |
                       |_| |_|\__|\__| .__/|_|   \__, |
                                     | |          __/ |
                                     |_|         |___/


                  HTTP logging and information retrieval tool
                                 version 0.1.8

         Copyright (c) 2005-2014 Jason Bittel <[email protected]>

                For further information about the program, see:
                   http://dumpsterventures.com/jason/httpry

       For modification and redistribution information, see COPYING file


--{ ABOUT }--

httpry is a tool designed for displaying and logging HTTP traffic. It is not
intended to perform analysis itself, but instead to capture, parse and/or
log the traffic for later analysis. It can be run in real-time displaying
the live traffic on the wire, or as a daemon process that logs to an output
file. It is written to be as lightweight and flexible as possible, so that
it can be easily adaptable to different applications. It does not display
the raw HTTP data transferred, but instead focuses on parsing and displaying
the request/response line along with associated header fields.

"How is this tool useful?" you may ask. Here's just a few ideas:

 > See what users on your network are browsing online
 > Check for proper server configuration (or improper, as the case may be)
 > Research patterns in HTTP usage
 > Watch for dangerous downloaded files
 > Verify the enforcement of HTTP policy on your network
 > Extract HTTP statistics out of saved capture files
 > It's just plain fun to watch in realtime

In addition to the core program, there are several Perl scripts included
for processing httpry log files. They should be useful for a number of
generic situations, and can serve as a useful starting point for your own
log parsing toolset. More information about these scripts can be found in
the doc/perl-tools file.


--{ INSTALLATION }--

httpry should compile on almost any *nix based OS with a relatively recent
version of libpcap (specifically tested against 1.1.1 and newer). To compile
and install, run these commands in the base httpry directory:

 $ make
 # make install

which compiles the program and copies the binary and man page to their
appropriate locations. You can run the binary from the compilation directory
if you don't want to install it. To uninstall the program, run:

 # make uninstall

from the installation directory, or manually delete the executable and man
page.


--{ USAGE }--

Running httpry with no options will cause it to listen on the first network
device and output to the console with some sane defaults. The -h switch will
print out an abbreviated description of the available options to change the
defaults. This section describes these options in greater detail.

httpry [ -dFhpqs ] [ -b file ] [ -f format ] [ -i device ] [ -l threshold ]
       [ -m methods ] [ -n count ] [ -o file ] [ -P file ] [ -r file ]
       [ -S bytes ] [ -t seconds ] [ -u user ] [ 'expression' ]

-b file
Write all processed HTTP packets to a binary pcap dump file. Useful for
further analysis of logged data.

-d
Run the program as a daemon process. All program status output will be sent
to syslog. A pid file is created for the process in /var/run/httpry.pid by
default. Requires an output file specified with -o.

-f format
Provide a comma-delimited string specifying the parsed HTTP data to output.
See the doc/format-string file for further information regarding available
options and syntax.

-F
Disable all output buffering. This may be helpful when piping httpry output
into another program.

-h
Display a brief summary of these options.

-i device
Specify an ethernet interface for the program to listen on. If not specified,
the program will poll the system for a list of interfaces and select the
first one found.

-l threshold
Specify a requests per second rate threshold value when running in rate
statistics mode (-s). Only hosts with a rps value greater than or equal to
this number will be displayed. Defaults to 1.

-m methods
Provide a comma-delimited string that specifies the request methods to parse.
The program defaults to parsing all of the standard RFC2616 method strings if
this option is not set. See the doc/method-string file for more information.

-n count
Parse this number of HTTP packets and then exit. Defaults to 0, which means
loop forever.

-o file
Specify an output file for writing parsed packet data.

-p
Do not put the NIC in promiscuous mode on startup. Note that the NIC could
already be in that mode for another reason.

-P file
Specify a path and filename for creating the PID file in daemon mode.

-q
Suppress non-critical output (startup banner, statistics, etc.).

-r file
Provide an input capture file to read from instead of performing
a live capture. This option does not require root privileges.

-s
Run httpry in an HTTP request per second display mode. This periodically
displays the rate per active host and total rate at a specified interval.

-S
Specify a number of bytes to skip in the ethernet header. This allows for
custom header offsets to be accounted for.

-t seconds
Specify the host statistics display interval in seconds when running in
rate statistics mode (-s). Defaults to 5 seconds.

-u user
Specify an alternate user to take ownership of the process and any output
files. You will need root privileges to do this; it will switch to the new
user after initialization.

'expression'
Specify a bpf-style capture filter, overriding the default. Here are a few
basic examples, starting with the default filter:

 'tcp port 80 or 8080'
 'tcp dst port 80'
 'tcp dst port 80 and src host 192.168.1.1'

These filters will capture all web traffic both directions on two common
ports, capture only requests made to port 80, and capture requests to port
80 by a particular host, respectively. See 'man tcpdump' for further
information on the syntax and available primitives.


--{ KNOWN ISSUES }--

It is worth noting that httpry is rather naive when it comes to parsing HTTP
packets. It does not perform any reordering or reassembly of packets and
simply searches the start of each packet for HTTP data and ignores the packet
if it does not find valid data. HTTP packets that are fragmented within the
request/response line will be parsed to the end of the packet and any header
data present in subsequent packets will not be parsed.

httpry's People

Contributors

Stargazers

Watchers

Forkers

vicfast jrm57 rzr shailcoolboy zclfly mikebd andy737 wsxiaoys ashwin31 live4code open-party jinguodong mysterier cofyc elain jingchunzhang webvul zhanglei shawnqi kissthink cvexva danilogiacomi g711ab zhulinu inn1983 noscripter arkaitzj cainiaocome cdlz deeper-think chickenlove kisbuddy hellyhe hqren kutogroup gloryliu ecore2018 tsoderstrom stmmcu qwivan rbadamsjr jijicanyu jabend tigerqiu712 jiafenggit suyy mforsetti znsstudio fanyunfeng iranbodi gokulvanan forging2012 leiwenbin hdyangxuan uther518 wangyx0055 wilvk qt-pay fooltux walkerfly snirad 2d0g skymysky andyprasetya codecopy tobeornottoer erdoukki tools-env dan4in fangrn zjjcorbin azridev super-l erwin-faceit dna2fork r-matous emersonx regud louygan songdonghui kacim62bobo sachiotomita gondsandeep jeffyoungl itsanni

httpry's Issues

Can httpry intercept http requests

Is it possible to intercept an http request using httpry and respond with a previously saved response?

I want a way to get integration tests running quicker and less dependent on external services. Basically I want to run tests once, record all the responses of external http requests and then the next time the tests are run just return these saved responses instead of going off to the network.

Something similar to https://github.com/myronmarston/vcr but at a lower level?

No incoming requests captured

running httpry -p -i eth0 on a host but cannot capture incoming requests, only the outgoing requests can be captured.
incoming requests are from another host in the same LAN. using tcpdump can capture the requests.
please tell me how to give further information about this problem.

Able to see requests on only one IP in the network

.. the IP on which the httpry was installed, Not able to see from other devices connected.

Requests greater than MTU fail to parse

I know that the proper fix is to implement stream-based packet capture, but a simple truncation method for requests greater than the packet MTU would be much better than simply ignoring the request. Can this be added?

make failing

running make fails on:
~/programs/httpry-0.1.8 ⌚ 12:14:21
$ make
gcc -Wall -O3 -funroll-loops -I/usr/include/pcap -I/usr/local/include/pcap -o httpry httpry.c format.c methods.c utility.c rate.c -lpcap -lm -pthread
httpry.c:14:18: fatal error: pcap.h: No such file or directory
compilation terminated.
Makefile:21: recipe for target 'httpry' failed
make: *** [httpry] Error 1

running linux mint:4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
using the downloaded zip, same error on the git version though

Timeout to pcap_open_live() shouldn't be zero

In prepare_capture(), pcap_open_live() is called with the third argument, which is the timeout, being 0.

The pcap man page says:

   read timeout
          If, when capturing,  packets  are  delivered  as  soon  as  they
          arrive,  the  application capturing the packets will be woken up
          for each packet as it arrives, and might have  to  make  one  or
          more calls to the operating system to fetch each packet.

          If,  instead,  packets are not delivered as soon as they arrive,
          but are delivered after a short delay (called a "read timeout"),
          more  than  one packet can be accumulated before the packets are
          delivered, so that a single wakeup would be  done  for  multiple
          packets,  and  each  set  of  calls made to the operating system
          would supply multiple packets,  rather  than  a  single  packet.
          This reduces the per-packet CPU overhead if packets are arriving
          at a high rate, increasing the number of packets per second that
          can be captured.

          The  read  timeout is required so that an application won't wait
          for the operating system's capture  buffer  to  fill  up  before
          packets are delivered; if packets are arriving slowly, that wait
          could take an arbitrarily long period of time.

          Not all platforms support a  read  timeout;  on  platforms  that
          don't,  the read timeout is ignored.  A zero value for the time-
          out, on platforms that support a read timeout, will cause a read
          to wait forever to allow enough packets to arrive, with no time-
          out.

This means that on several platforms, including all the BSDs and OS X, httpry will not see any packets until enough packets arrive to fill BPF's buffer.

A non-zero value should be specified here; the value should probably be between 1 and 1000.

Is there anyway to allow me to show all the HTTP headers or body?

With the -f format option,I could select which header to output,but what should I do if I want to show all the headers send and receive?

ppp support

Any chance you could support listening on a ppp interface?

Daemon run not outputing result to file for HTTP request per second display mode

Hello,

I have installed httpry-0.1.8-1.el7.x86_64.rpm on CentOS and i am running httpry in the following way:

sudo httpry -d -s -i eth0 'tcp dst port 3000' -o pry.txt

I want to have HTTP request per second display mode in that pry.txt output.

It creates the file with the following content:

[centos@nifi1-qa ~]$ cat pry
# httpry version 0.1.8
# Fields: host

but, unfortunately, no other output like:
"```
2018-09-07 11:25:59 127.0.0.1:3000 466 rps
2018-09-07 11:25:59 totals 468.40 rps

10g EndaceDAG Data Capture Card support

Hello,

I love httpry for copper interfaces. I recently purchased a 10g EndaceDAG Data Capture Card and attempted to make httpry look at the dag0 interface without success (error below). I'm able to get this to work with bro and tcpdump without issue. Any ideas on how I can get httpry to recognize this device like bro/tcpdump does? I suspect it lies somewhere in the libpcap module but thats outside my capabilities at the moment. Looking to see if anyone else has ideas or has experienced a similar issue. Redhat 6.5 OS.

"httpry version 0.1.8 -- HTTP logging and information retrieval tool
Copyright (c) 2005-2014 Jason Bittel [email protected]
Error: Cannot open live capture on 'dag0': dag0: No such device exists (SIOCGIFHWADDR: No such device)
Caught SIGINT, shutting down..."

"tcpdump -i dag0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on dag0, link-type EN10MB (Ethernet), capture size 262144 bytes
...
73 packets captured
73 packets received by filter
0 packets dropped by kernel"

Card reference:
http://www.endace.com/products/endacedag-data-capture-cards/features/

Thanks!

Doesn't work on OS X

Should this work on OS X? tcpdump gives me tons of output but httpry never shows anything at all... but does give a package count when quitting such as:

95 packets received, 0 packets dropped, 0 http packets parsed