hgn/captcp

A open source program for TCP analysis of PCAP files

captcp's Issues

How to extract contents of a TCP stream like "-e" parameter of tcptrace?

I've read the doc again and again but found nothing. Then I looked into the source code and seems that this feature is not provided?

animation module fails with KeyError: 'twh-delay'

./captcp.py animation test.pcap -f 13.1 --output-dir foo
# captcp 2010-2013 Hagen Paul Pfeifer (c)             
# http://research.protocollabs.com/captcp/            
# connection: 13 (local flow: 1, remote flow: 2)      
Traceback (most recent call last):                    
  File "./captcp.py", line 5210, in <module>          
    sys.exit(captcp.run())                            
  File "./captcp.py", line 5191, in run               
    pcap_parser.run()                                 
  File "./captcp.py", line 562, in run                
    self.callback(dt, packet.data)                    
  File "./captcp.py", line 4360, in process_packet    
    self.process_local_side(ts, packet, tpi)          
  File "./captcp.py", line 4302, in process_local_side                   
    (td, len(packet) / 2, self.rtt_data["twh-delay"] / self.acceleration))
KeyError: 'twh-delay'                                         

Error title in Spacing mode

When I was generating spacing graph, all two lines are named "TX" in spacing.gpi
The 2nd Line should be named "RX"
Should be fixed in file 'captcp / data / templates / spacing.gpi' Line 28

Error in extracting pcap wiles generated by WiFi 802.11ac using ns-3

I am using captcp to extract tcp throughput for point to point protocol links (PPP) from pcap files. Now using same captcp, I am not able to extract WiFi 802.11ac pcap files which are of the type IEEE_11_RADIO (802.11 plus radiotap header).
I have tried to add a line into captcp.py as:
dpkt.pcap.DLT_IEEE802_11_RADIO: dpkt.ethernet.Ethernet
But still it is generating error and I do not understand what I need to do.
Please find attached out.pcap files.
Please help me in this issue.
Thanks!!

• Anagha
  out.pcap.zip

About 802.11 pcap files

Hello!

Thank you for implementing the captcp.

I have a small question.
I am not sure whether bug or not.

When I was running on captcp about 802.11 pcap files, captcp had a error following as:
captcp 2010-2013 Hagen Paul Pfeifer and others (c)
http://research.protocollabs.com/captcp/
Packet link type not know (105)! Interpret at Ethernet now - but be carefull!
Traceback (most recent call last):
File "captcp.py", line 5225, in
sys.exit(captcp.run())
File "captcp.py", line 5196, in run
pcap_parser.run()
File "captcp.py", line 574, in run
packet = self.decode(pkt)
File "C:\Python27\lib\site-packages\dpkt\dpkt.py", line 75, in init
raise NeedData
dpkt.dpkt.NeedData

As I know the dpkt in python, dpkt have supported 802.11 (i.e. dpkt.ieee80211.IEEE80211). But, PcapParser class in source does not support 802.11.

why does not captcp support 802.11?
please consider the 802.11 pcap.

Thank you in advance.

Captcp fails to analize pcap file. Error: TemplateMod instance has no attribute 'logger'

Recent version of captcp fails to analyze pcap file.
Version 1.7 works fine so possibly this issue was introduced by recent commit c18793c
"Consolidate logging.getLogger() calls"

$ captcp throughput -i -o test ./test.pcap 
# captcp 2010-2013 Hagen Paul Pfeifer and others (c)
# http://research.protocollabs.com/captcp/
Traceback (most recent call last):
  File "/usr/bin/captcp", line 5220, in 
    sys.exit(captcp.run())
  File "/usr/bin/captcp", line 5181, in run
    classinstance.initialize()
  File "/usr/bin/captcp", line 2702, in initialize
    self.create_gnuplot_environment()
  File "/usr/bin/captcp", line 2652, in create_gnuplot_environment
    tmpl = string.Template(TemplateMod().get_content_by_name("throughput"))
  File "/usr/bin/captcp", line 1014, in __init__
    self.init_db()
  File "/usr/bin/captcp", line 1041, in init_db
    self.logger.debug("initialize local template database")
AttributeError: TemplateMod instance has no attribute 'logger'

Pcap file is a capture of few SSH packets from the network.
Captcp v1.7 analyzes same capture successfully.

Fix can be:

diff --git a/captcp.py b/captcp.py
index d3bf33f..930d69e 100755
--- a/captcp.py
+++ b/captcp.py
@@ -1011,6 +1011,7 @@ class TemplateMod(Mod):
 
 
     def __init__(self):
+       Mod.__init__(self)
         self.init_db()
 

Big white space areas above and below graphs

There is quite a big amount of white space above and below of plotted graphs. Especially when embedding in other documents this is inconvenient. Following some additions to the default makefile to automatically crop graph files:

GNUPLOT_FILES = $(wildcard *.gpi)
PNG_OBJ = $(patsubst %.gpi,%.png,  $(GNUPLOT_FILES))
PDF_OBJ = $(patsubst %.gpi,%.pdf,  $(GNUPLOT_FILES))

->TEMPFILE := $(shell mktemp)

%.pdf: %.eps
@echo "conversion in pdf format"

->@ps2epsi $< TEMPFILE
->@epstopdf --outfile=$*.pdf TEMPFILE
->@rm TEMPFILE
@echo "end"

Daniel

no Graphs with FreeBSD 8.3

I'm struggeling with the graphical part. I'm able to parse and analyze capture files, but generating PDF/PNG doesn't work. All packages are installed except texlive-latex-extra, which is not availabe under BSD.

OS:
[root@psrISS001 ~]# uname -a
FreeBSD psrISS001 8.3-RELEASE-p11 FreeBSD 8.3-RELEASE-p11 #0: Tue Sep 10 00:06:55 UTC 2013 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64

[root@psrISS001 /data/HSI-10G/captcp]# captcp timesequence -i -o timesequence -f 1.1 data.pcap
Traceback (most recent call last):
File "/usr/bin/captcp", line 4932, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 4893, in run
classinstance.initialize()
File "/usr/bin/captcp", line 1275, in initialize
self.parse_local_options()
File "/usr/bin/captcp", line 1515, in parse_local_options
parser.add_option( "-g", "--gnuplot-options", dest="gnuplotoptions", default=none,
NameError: global name 'none' is not defined

any help appreciated !

Regards
Daniel

Error parsing pcap file

$ captcp statistic 500M.pcap
# captcp 2010-2013 Hagen Paul Pfeifer and others (c)
# http://research.protocollabs.com/captcp/
Traceback (most recent call last):
  File "/usr/bin/captcp", line 6054, in <module>
    sys.exit(captcp.run())
  File "/usr/bin/captcp", line 6040, in run
    ret = classinstance.process_final()
  File "/usr/bin/captcp", line 4993, in process_final
    self.process_final_data()
  File "/usr/bin/captcp", line 4987, in process_final_data
    self.rexmt_final(connection.sc1)
  File "/usr/bin/captcp", line 4671, in rexmt_final
    sc.user_data["tl-ps-avg"]    = "%.2f" % numpy.mean(sc.user_data["_tl_pkt_sizes"])
AttributeError: 'NoneType' object has no attribute 'mean'

PPP Accounting/Statistics Bugs

captcp is Ethernet centric. Some values like link layer overhead is always calculated with a Ethernet assumption. Since bb00a87 and PPP support this is not true.

There is no easy workaround because the API as based on a Ethernet assumption. Solution: shift packet data argument to packet and not packet.data and provide real link layer access. This needs modification to all modules!

sack problem

Hi,
when read the source of captcp.py, sack start is left edge, and stop is right edge, also in memory. but the function linear_sackblocks_array() exchange start and stop.

throughput.data is empty

I have ONE TCP flow of 64kb download a pcap file. When I run the following command, I get throughput.data but it is empty! Please correct me if I have missed anything.

captcp throughput test.pcap --init -o temp/

Let me know if you need more info.

Time-Sequence Module plots - Make png

Hello people. When I use the module " TIMESECUENCE " I keep everything in a folder, that folder and income to do "make all" or "make png" displays on screen " compillation of time- sequence.gpi " and stays down waiting indefinitely . I check the time- code sequence.gpi and me is because the lines :

1. load " data- arrow.data "
2. load " data- arrow- retrans.data "
3. load " data- arrow- sack.data "
4. load " data- arrow- push.data "
5. load " data- arrow- ece.data "
6. load " data- arrow- cwr.data "

If I comment that the scrip works, but do not think all that well as prints 3 equal lines .

Someone happens the same? What could we be doing?
Thanks , I hope your help please

Comands:
mkdir time
captcp timesequence -i --zero -e -o time -f 1.1 eth1G2.pcap
cd time
make all

python 3

Please consider migrating the application to python 3: python 2 is EOL from January 1st 2020 on.

AttributeError: 'int' object has no attribute 'seconds'

Hi,

I´m getting an error while trying to graph the throughput with captcp on actual Kubuntu 13.04 64bit. I would fix this mysef if i could write python but I can´t so here is the output for you:

floh1111@floh1111-VirtualBox:~/Rechnernetze/ns-allinone-3.16/ns-3.16$ captcp throughput -s 0.1 -i -f 45.2 -o throughput-graph/ meine_aufgabe-5-2.pcap 
# captcp 2010-2013 Hagen Paul Pfeifer (c)
# Packet link type not know (9)! Interpret at Ethernet now - but be carefull!
# Packet link type not know (9)! Interpret at Ethernet now - but be carefull!
Traceback (most recent call last):
  File "/usr/bin/captcp", line 4345, in <module>
    sys.exit(captcp.run())
  File "/usr/bin/captcp", line 4331, in run
    ret = classinstance.process_final()
  File "/usr/bin/captcp", line 2711, in process_final
    timediff =  Utils.ts_tofloat(self.end_time - self.start_time)
  File "/usr/bin/captcp", line 455, in ts_tofloat
    return float(ts.seconds) + ts.microseconds / 1E6 + ts.days * 86400 
AttributeError: 'int' object has no attribute 'seconds'

I´m running a plain installation in an virtual machine but I think I installed all needed packages: sudo apt-get install python-dpkt python-numpy make gnuplot texlive-latex-extra mupdf

Regards
Floh1111

Captcp error wtih ns3

Hi
I´m getting an error while trying to graph the throughput with captcp on actual ubuntu 12.04
the pcap file generate using ns3 simulator and i would like to graph the output with captcp

root@salma:~# captcp statistic Amrr2.pcap
captcp 2010-2013 Hagen Paul Pfeifer (c)
Traceback (most recent call last):
File "/usr/bin/captcp", line 4364, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 4332, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "/usr/bin/captcp", line 518, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 105, in init
self.dloff = dltoff[self.__fh.linktype]
KeyError: 105

please could you tell me why ?

MemoryError on sequencegraph

Guessing this is not a captcp problem, but wondering if you have some tips? Trying to run this on OSX.

$> python captcp.py sequencegraph -v debug -i 1 -l 192.168.0.199 ~/Desktop/archive.pcap

# captcp 2010,2011 Hagen Paul Pfeifer (c)
# python: 2.7.1 [releaselevel: final, serial: 0]
# pcap file: /Users/igrigorik/Desktop/archive.pcap
# visualization limited to the following connections: ['1']
# call pre_process_packet [1/4]
# call pre_process_final [2/4]
# now draw 974 packets
# call process_packet [3/4]
# call pre_process_final [4/4]
Traceback (most recent call last):
  File "captcp.py", line 3753, in <module>
    sys.exit(captcp.run())
  File "captcp.py", line 3740, in run
    ret = classinstance.process_final()
  File "captcp.py", line 1848, in process_final
    self.cr.show_page()
MemoryError

Cairo issue / bug? pcap file: http://dl.dropbox.com/u/2132185/archive.pcap

Failed to anaylize small pcap file

some pcap files captcp failes to parse data.
For example, attached small pcap file with ~300 packets of flow control in rate of
30 packets per second. Running captcp on that file cause following backtrace:

$ captcp throughput  --verbose=debug -o vm2 ./vm2.pcap
# captcp 2010-2013 Hagen Paul Pfeifer (c)
# http://research.protocollabs.com/captcp/
# python: 2.7.5 [releaselevel: final, serial: 0]
# pcap file: ./vm2.pcap
# call pre_process_packet [1/4]
# call pre_process_final [2/4]
# call process_packet [3/4]
# skip processing step
# call pre_process_final [4/4]
Traceback (most recent call last):
  File "/usr/bin/captcp", line 5210, in 
    sys.exit(captcp.run())
  File "/usr/bin/captcp", line 5196, in run
    ret = classinstance.process_final()
  File "/usr/bin/captcp", line 2820, in process_final
    timediff =  Utils.ts_tofloat(self.end_time - self.start_time)
  File "/usr/bin/captcp", line 457, in ts_tofloat
    return float(ts.seconds) + ts.microseconds / 1E6 + ts.days * 86400
AttributeError: 'int' object has no attribute 'seconds'
$>

Here is the trace from pcap:

01:58:21.282251 MPCP, Opcode Pause, length 46
01:58:21.312596 MPCP, Opcode Pause, length 46
01:58:21.342912 MPCP, Opcode Pause, length 46
01:58:21.373227 MPCP, Opcode Pause, length 46
.....
01:58:31.225154 MPCP, Opcode Pause, length 46
01:58:31.255468 MPCP, Opcode Pause, length 46

Connection hash collisions

Hi,

I discovered that the way captcp hashes/identifies flows in order to associate them with connections is flawed.

Connections and subflows are identified by a kind of unique identifier/hash, uid computed in TcpConn.__init__ beginning at line 2273:

l = [ord(a) ^ ord(b) for a,b in zip(self.sipnum, self.dipnum)]

self.uid = "%s:%s:%s" % (
               str(self.ipversion),
               str(l),
               str(long(self.sport) + long(self.dport)))

This identifier is used in various places as an index/key into hashtables (well, dicts :) ) containing connections, and is notably used for associating subflows (uni-directional) with the connection (bi-directional) they are part of. For this purpose, the hash needs to be the same for any given port-pair between the same two addresses, regardless of source/destination ordering. I suppose this is why they are added together, since this is a commutative operation.

The problem is that uid is used as a direct key into tables, without checking for hash collisions. The above function is obviously not a perfect hash function, and will produce collisions if you are unlucky enough to have two connections with port pairs that add up to the same thing, which corrupts the data for those connections.

I am proposing a patch to rectify this by concatenating the (hex string representations of the) ordered port pair together instead of summation. This preserves the commutativity, but ensures we will not get collisions for the port pair part of the identifier. Note that the hash function overall still isn't perfect, you could get collisions between the IP address pairs, but that is perhaps slightly less likely.

Best wishes,
Kristian

PS. If you wonder how I came across this, I managed to trigger this bug by starting a bunch of connections to sequentially increasing port numbers almost simultaneously under FreeBSD. When FreeBSD has to generate a lot of port numbers in a very short time, it temporarily switches ephemeral port selection strategy to simple sequential incrementation, see: https://www.cymru.com/jtk/misc/ephemeralports.html

Sometimes a pair of connect() calls would get inversed and I ended up with port pairs such as

• 10000 - 5000 (sum=15000)
• 10001 - 5001 (sum=15002)
• 10002 - 5003 (sum=15005)
• 10003 - 5002 (sum=15005)
• 10004 - 5004 (sum=15008)
• ...

Replacing pypcap with something more usable?

The pypcap module seems to be abandoned, and it is nearly impossible to install it on Fedora. What about moving to something more widespread and maintained?

timesequence -t option not working

The -t option for timesequence doesn't appear to be working.

captcp timesequence -o timesequence/ -z -f 1.1 -i -t 4:6 *.pcap

The resulting pdf seems to ignore the -t option and includes the entire time sequence (10 seconds in my case).

AttributeError: TemplateContainer instance has no attribute 'name'

When running captcp with --init for creating gnuplots, I've encountered an error that is raised due to the TemplateContainer instance not having an attribute name. I've traced this issue to the init_db() method in TemplateMod. There is possibly an erroneous if-statement creating attributes name and type on TemplateContainers. I tried adding name and type always (empty strings if neither of the conditions are true) and this seemed to fix the problem.

I'm using python version 2.7.1, and the commit I'm currently on is 0eb9fdf (which I think is HEAD).

== OUTPUT ==
$ captcp inflight -v debug -f 1.1 send.pcap -o test -i

captcp 2010,2011 Hagen Paul Pfeifer (c)

python: 2.7.1 [releaselevel: final, serial: 0]

pcap file: send.pcap

connection: 1 (data flow: 1, ACK flow: 2)

initialize local template database

Traceback (most recent call last):
File "/usr/bin/captcp", line 3749, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 3711, in run
classinstance.pre_initialize()
File "/usr/bin/captcp", line 2417, in pre_initialize
self.create_gnuplot_environment()
File "/usr/bin/captcp", line 2471, in create_gnuplot_environment
fd.write("%s" % (TemplateMod().get_content_by_name("inflight")))
File "/usr/bin/captcp", line 844, in get_content_by_name
if i.name == name:
AttributeError: TemplateContainer instance has no attribute 'name'

== Possible fix? ==
Adding the following on line 880
else
tc.name = ""
tc.type = ""

UDP Throughput calculation

I need to calculate the throughput over a trace contains only UDP packets.
I try the program first but it terminated therefore I check the code and I find that:
In throughput module class, the program check if the packet is part of TCP flow in line (2805). TCP or UDP has no effect in the remain of the code.

can you help me?

UnboundLocalError: local variable 'title' referenced before assignment

Hi,
When I run the following cmd: (captcp 1.7)

   captcp throughput -o out -i tcp-large-transfer-0-0.pcap

I got this error:

# captcp 2010-2013 Hagen Paul Pfeifer and others (c)
# http://research.protocollabs.com/captcp/
Traceback (most recent call last):
  File "/usr/bin/captcp", line 5269, in <module>
    sys.exit(captcp.run())
  File "/usr/bin/captcp", line 5228, in run
    classinstance.initialize()
  File "/usr/bin/captcp", line 2710, in initialize
    self.create_gnuplot_environment()
  File "/usr/bin/captcp", line 2663, in create_gnuplot_environment
    gpi_cmd = tmpl.substitute(TITLE=title,
UnboundLocalError: local variable 'title' referenced before assignment

Printing one data file with throughput multiple flows

Actually, if I have 3 flows and want to plot their throughput, I need to run 3 times captcp, using each time a different -f parameter .

What about adding a range capability for the -f parameter ? The output .dat file will be something like

Time Throughput flow 1 Throughput flow 2
0.0 0.0 0.0
1.0 100.0 85.0
2.0 120.0 95.0

.....

Thank you

Spurious Retransmissions Module

View merge request comment for detailed info.

Issues:

• Module is not very efficient
• Might miss Spur. Retr. atm due to Linux echo timestamp behaviour

Dumps:
all-dumpfiles.zip

Cmds:
./captcp.py spurious-retransmissions -f 1.1 -m summary ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m retransmissions ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m spurious ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m list ~/megadump.pcap

Time-Sequence Module plots - Arrows outside of t-range

When using the t-range of the timesequence module, the arrows in plots are not bounded by the gnuplot y-range and thus be drawn in the whole plot outside of the axes.

time-sequence.pdf

Attribute error while parsing pcap file

File can be found at https://goo.gl/AWATmN
$captcp statistic file.pcap
Traceback (most recent call last):
File "/usr/bin/captcp", line 5868, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 5854, in run
ret = classinstance.process_final()
File "/usr/bin/captcp", line 4807, in process_final
self.process_final_data()
File "/usr/bin/captcp", line 4801, in process_final_data
self.rexmt_final(connection.sc1)
File "/usr/bin/captcp", line 4485, in rexmt_final
sc.user_data["tl-ps-avg"] = "%.2f" % numpy.mean(sc.user_data["_tl_pkt_sizes"])
AttributeError: 'NoneType' object has no attribute 'mean'

Option to ignore corrupt packets

Feature request for an option to be able to ignore packets that are corrupt/mangled instead of erroring out like so:

root@host ~/t/g/captcp# ./captcp.py throughput --output-dir /root/traffic-stats/ /media/storage/3-20.pcap
captcp 2010-2013 Hagen Paul Pfeifer and others (c)
http://research.protocollabs.com/captcp/
Traceback (most recent call last):
File "./captcp.py", line 5298, in
sys.exit(captcp.run())
File "./captcp.py", line 5266, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "./captcp.py", line 534, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 103, in init
raise ValueError, 'invalid tcpdump header'
ValueError: invalid tcpdump header

Need to check file format

python captcp.py statistic file.txt

captcp 2010-2013 Hagen Paul Pfeifer and others (c)

http://research.protocollabs.com/captcp/

Traceback (most recent call last):
File "captcp.py", line 5301, in
sys.exit(captcp.run())
File "captcp.py", line 5269, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "captcp.py", line 534, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 97, in init
self.fh = FileHdr(buf)
File "/usr/lib/pymodules/python2.7/dpkt/dpkt.py", line 75, in __init
raise NeedData
dpkt.dpkt.NeedData

It can be checked with "FileHdr" class in the dpkt module.